Trying to debug a broken pipe on

tap-postgres target-snowflake

. Looks like the issue is with write_schema_message() from the postgres tap, however the stream discovery catalog does succeed so a bit confused.

@lucas_w Hmm, that's odd. The issue may actually be the target failing and closing its stdin pipe, which is the tap's stdout pipe and results in a write error there.

Yeah, that would make me think it’s a snowflake configuration issue, but I’m not seeing it there

Not unless you have some Singer messages to feed it! So what we could do is run

meltano invoke tap-postgres > tap-postgres.jsonl

, followed by

cat tap-postgres.jsonl | meltano invoke target-snowflake

That makes sense, will test it now. Thank you

@lucas_w Hmm, so

target-snowflake

isn't raising any errors or anything, it's just doesn't appear to be doing anything at all?

Yeah there’s no stack trace

meltano) lucaswiley@lucas-wiley pg-snowflake % meltano invoke tap-postgres > tap-postgres.jsonl                                      
INFO Selected streams: ['userdata-public-RedactedTableName'] 
INFO No currently_syncing found
INFO Beginning sync of stream(userdata-public-RedactedTableName) with sync method(full)
INFO Stream userdata-public-RedactedTableName is using full_table replication
INFO Current Server Encoding: UTF8
INFO Current Client Encoding: UTF8
INFO hstore is UNavailable
INFO Beginning new Full Table replication 1594050550627
INFO select SELECT  "***redactedtableinfo"
                                      FROM "public"."RedactedTableName***"
                                     ORDER BY xmin::text ASC with itersize 20000
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 2035, "tags": {}}
(meltano) lucaswiley@lucas-wiley pg-snowflake % cat tap-postgres.jsonl | meltano invoke target-snowflake
(meltano) lucaswiley@lucas-wiley pg-snowflake % meltano invoke target-snowflake

@lucas_w How comfortable are you finding the

target-snowflake

source in

.meltano/loaders/target-snowflake/venv/lib/python3.7/site-packages/target_snowflake

and adding some print-based debugging of your own? 🙂

Comfortable enough 🙂 thanks for your help

@lucas_w Please let me know what you find! There is probably still more Meltano can do to help in situations like this. https://gitlab.com/meltano/meltano/-/issues/2116 would be a start, for example, but even the "it's probably the target that failed, not the tap" hunch would be something Meltano could be more clear about

Will do! Thanks

Which is consistent with

target-snowflake

's silent failing and just closing the stdin pipe, of course

Not yet, still poking through the target file

We could put a

breakpoint()

in there and step through until it goes off the expected track

Yeah, I have a set_trace() but it hasn’t tripped yet 🤷‍♂️

Note that you can use their target-snowflake with Meltano as well 😉

Sure

@lucas_w Got Zoom? https://gitlab.zoom.us/my/gitlab.douwemaan

Yes, Jumping on

@lucas_w I've created https://gitlab.com/meltano/target-snowflake/-/issues/27, but simply bumping

asn1crypto

up to

1.3.0

may not be a full solution, since https://gitlab.com/meltano/target-snowflake/-/merge_requests/28 suggests that it's incompatible with the current version of

snowflake-connector-python

we use.

Great, I’ll give it a look. Thanks for your help

I've also created https://gitlab.com/meltano/meltano/-/issues/2134 (Consider recommending DataMill or PipelineWise target-snowflake over our own)