Does anyone have a yaml with a working target-sno...
# troubleshootingj
jaye_howell
12/15/2022, 7:25 PMDoes anyone have a yaml with a working target-snowflake destination? Still having issues so if someone is loading data to snowflake would love some help
v
visch
12/15/2022, 7:47 PMhttps://github.com/meltano/squared/blob/main/data/load/loaders.meltano.yml#L30 meltano uses one for their own data project
p
pat_nadolny
12/15/2022, 7:50 PM@jaye_howell yeah the link Derek shared is a production meltano project using snowflake. What issues are you running into?
j
jaye_howell
12/15/2022, 8:02 PMi was trying to use the meltano variant, not the transferwise. Fomr the looks it seems transferwise requires
1) an S3 bucket or some other external stage in snowflake
2) it converts the extract output to CSV for loading to snowflake. The extracts (unless it escapes things) will have issues as a CSV formated file.
jaye_howell
12/15/2022, 8:03 PMI was hoping the meltano was using json or some other format to load
p
pat_nadolny
12/15/2022, 8:17 PM@jaye_howell no for the transferwise variant you dont need to create an S3 bucket/external stage. The target uses an internal stage by default but has an advanced feature that allows you to use an external stage if you want to.
pat_nadolny
12/15/2022, 8:17 PMI dont use an external stage
pat_nadolny
12/15/2022, 8:19 PMfor point 2 I dont know for sure but I think it does handle all escaping properly. The stage/file format piece is an implementation detail in my mind, like the user shouldnt really have to be concerned with it. I've sent a solid variety of data through it and never had an issue with it being CSV in the intermediate loading area
pat_nadolny
12/15/2022, 8:21 PMAlso the meltano variant thats listed is basically deprecated soon to be replaced by https://github.com/MeltanoLabs/target-snowflake although thats still being developed. See #C04BG3978NT for chatter around that
j
jaye_howell
12/15/2022, 8:23 PMok working on the transferwise variant load now so will see how that goes
p
pat_nadolny
12/15/2022, 8:28 PMAlso in case you missed it, dont forget to create the file format object https://hub.meltano.com/loaders/target-snowflake#dependencies
j
jaye_howell
12/15/2022, 8:54 PMi see it says cant connect to snowflake backend
p
pat_nadolny
12/15/2022, 8:56 PMTry removing the
.<http://snowflakecomputing.com|snowflakecomputing.com>j
jaye_howell
12/15/2022, 9:01 PMyea i just did and seem to work...was comparing it to the yaml above
p
pat_nadolny
12/15/2022, 9:03 PMyou're seeing data in snowflake?!? seems like its working then
j
jaye_howell
12/15/2022, 9:03 PMyes
jaye_howell
12/15/2022, 9:05 PMthank you @pat_nadolny @christoph @thomas_briggs For your help...now with POC done. Need to look at how changed data is handled, how to add columns if needed and other things
p
pat_nadolny
12/15/2022, 9:07 PMSince youre using the transferwise variant you can check out their schema change docs https://transferwise.github.io/pipelinewise/user_guide/schema_changes.html. I thought they were pretty good
c
christoph
12/15/2022, 9:09 PMNeed to look at how changed data is handledPreemptively posting this link ... 😁 https://stackoverflow.com/q/71669574/6056177