Left snowflake warehouse on 4XL during trial. Move...
# randomv
visch
11/09/2022, 8:13 PMLeft snowflake warehouse on 4XL during trial. Move to prod, ran some queries against the Metadata on Views of the snowflake database (took 1 hr to run), $400 bill. I now understand why they make so much money. What a joke
I'd assume I'm not the only one
visch
11/09/2022, 8:15 PMyes I downsized my compute wh down to small.
Crazy you need to know ahead of time what compute wh cluster you need. I'm sure the blame is all on me, but this isn't "easy" to me
visch
11/09/2022, 8:20 PMi'm the idiot
t
thomas_briggs
11/09/2022, 8:22 PMYeah we had ended up with a bill from AWS from testing Redshift that was ummm... a bit larger than that 😮 So I feel your pain.
s
Sven Balnojan
11/09/2022, 8:28 PM^^ Love it. So compute & storage are becoming almost free, except when you put them together and call it a database 😄
j
jacob_matson
11/09/2022, 8:47 PMOuch! We had some migrations where we were keeping old stuff running and then moved everything to a new instance and paid 2x for a bit. That was not fun.
a
aaronsteers
11/09/2022, 9:41 PMOMG - sorry to hear this @visch! You are definitely not the only one. This is a story as old as Snowflake itself.
aaronsteers
11/09/2022, 9:42 PMMost orgs almost immediately pre-create warehouses and restrict access to X-SMALL only except for approved use cases.
aaronsteers
11/09/2022, 9:45 PMTo anyone who think Meltano and the open source data stack just competes with Fivetran and friends, imagine moving all your Snowflake CI and ETL workloads to something like Meltano+DuckDB and publishing just the result to Snowflake. All the ease-of-use of Snowflake for querying, but without Snowflake's compute markup for your daily workloads. 😄 🤫
j
jacob_matson
11/09/2022, 9:45 PMworking on it
a
aaronsteers
11/09/2022, 9:45 PMOh yeah, @jacob_matson - was thinking of you when I wrote that 👆 😄
j
jacob_matson
11/09/2022, 9:46 PMi'm eyeing that dagster util as well.
jacob_matson
11/09/2022, 9:46 PMI think with a bit of work it will run in a container (already working) but then also spin up additional containers as required
jacob_matson
11/09/2022, 9:47 PMMDS in a box, in a container etc etc
v
visch
11/10/2022, 1:01 AMDoes sound nice for some of this stuff! I'm trying to pull metadata information out of snowflake so no good for me but that sounds much better than running python in a warehouse and having the bill rack up
a
aaron_phethean
11/10/2022, 7:20 AMSorry to hear that @visch - feels like a right of passage for everyone using Snowflake. We had a similar experience too.
On a brighter note. Spoke to a couple of guys working on a snowflake cost model, aim is to get a query cost: https://hub.getdbt.com/get-select/dbt_snowflake_monitoring/latest/
One good tip was to configure the auto suspend to 1 minute. By default it's 10 minutes - so for your 1 meta data query it will be idle for 9 minutes…. Jokes
j
jacob_matson
11/10/2022, 3:04 PMlots of heat right now in snowflake cost management: capital one slingshot, select.dev, and a few more
m
max_de_rooij
11/11/2022, 7:46 AM@aaronsteers That sounds like a workflow that could really solve my current case. Any idea where I can learn more?
j
jacob_matson
11/11/2022, 4:33 PM@max_de_rooij mdsinabox.com for what’s working today or DM me and I can show you what’s next
a
aaronsteers
11/11/2022, 4:42 PM@max_de_rooij - I'd definitely start with the mds-in-a-box example from @jacob_matson. Then, as a final step, there are a few options you'd have to push the result back to Snowflake or similar. Does this help?
a
aaron_phethean
11/11/2022, 5:55 PMIt is certainly an interesting model. We are using Postgres in this way, and pushing the result back to MongoDB for a client! We could definitely be using DuckDB (perhaps to parquet) to do the same and of course, the results could be simply copied anywhere.
m
max_de_rooij
11/12/2022, 7:56 AMThanks @aaronsteers and @jacob_matson. I reserved some time coming week to implement a simple variant. Will DM you Jacob 🙂