We’re considering moving to Snowflake. I’m wonderi...
# getting-startedm
matt_cooley
02/16/2022, 11:18 PMWe’re considering moving to Snowflake. I’m wondering if anyone knows which is easier/cheaper:
• Target in S3 (csv, jsonl, etc) and ingest with snowpipe
• Target Snowflake directly using a warehouse
Not new to Meltano, but a total noob when it comes to Snowflake. (also, let me know if there’s a better channel to ask this…I thought this one seemed appropriate)
a
aaronsteers
02/16/2022, 11:51 PMHi, @matt_cooley! These are not necessarily mutually exclusive - and you may want to have certain pipelines use one or another approach. With Snowpipe, you'll generally get lower cost for near-realtime use cases. However, for once-per-day batches and/or relatively low volume streams, your cost might be lower with a once-per-day or twice-per-day batch window.
aaronsteers
02/16/2022, 11:53 PMAnother important consideration is if you need to adapt to new column additions. For tables where you expect columns to be added upstream (esp. something like Salesforce which is highly customizable), then you're likely going to want to take advantage of the automatic table maintenance and automatic column addition that you'll get from target-snowflake.
aaronsteers
02/16/2022, 11:53 PMIs this helpful?
aaronsteers
02/16/2022, 11:54 PMIn a past life, we started with everything going through target-snowflake, and then added snowpipe just for a couple one-off tables which (1) benefited from near-realtime data and (2) weren't a liability in terms of new columns getting added/modified over the project lifecycle.
m
matt_cooley
02/17/2022, 12:41 AMThat makes sense! We’re not super worried about “real-time” data. Mostly want to make sure we can minimize the up-time of the snowflake warehouses. Sounds like meltano should work well for in general
s
shagility__agiledata.io_
02/19/2022, 9:16 PMUse Google BigQuery instead, use Meltano to drop the files into GCS and then ingest into BigQuery (ingestion is free)