<@U06CE9BN604> Thoughts on handling json schema co...
# announcementsn
nevin_morgan
08/03/2020, 8:10 PM@douwe_maan Thoughts on handling json schema combinations when targeting postgres, or do you have a strong recommendation for a different data warehouse?
d
douwe_maan
08/04/2020, 3:54 PM@nevin_morgan You're referring to columns that can hold different types, per the discovered JSON schema?
In general, Snowflake is more optimized for data warehouse tasks than Postgres, which is "just" an RDBMS, but I don't know how best to deal with such columns in either. The specific target you'll use (different versions of target-postgres and target-snowflake exist) will also could also make a difference.
n
nevin_morgan
08/04/2020, 4:46 PMspecifically the anyof / oneof combos choke with https://github.com/meltano/target-postgres.git
nevin_morgan
08/04/2020, 4:47 PM@douwe_maan I have been considering snowflake, but I can't figure out for the life of me the cost of it vs something like bigquery or redshift
d
douwe_maan
08/04/2020, 4:47 PM@nevin_morgan An
anyOfdouwe_maan
08/04/2020, 4:47 PMOtherwise, you may have more luck with https://github.com/transferwise/pipelinewise-target-postgres
n
nevin_morgan
08/04/2020, 4:50 PMwould you happen to have any insight on how to figure out the cost of snowflake? I was only looking at postgres because it can be run on my fleet of servers in house relatively easily
nevin_morgan
08/04/2020, 4:51 PMthat pricing page makes no sense lol
j
julian_knight
08/04/2020, 4:52 PMIn the process of that fix, I considered the possibility of target-postgres treating any type that didn't map to a normal postgres type to a json column. It would be a bit of a pain to query, but you'd be able to support situations like
{"type": ["string", "number"]}d
douwe_maan
08/04/2020, 5:08 PM@nevin_morgan Lol, they have an entire Pricing Guide that requires you to register 🤦♂️ https://www.snowflake.com/pricing-page-registration-page/
I don't have experience with Snowflake pricing myself though; I'd suggest asking @adam_roderick since you're already working together, I'm sure he'll be able to advise you better on what will be most appropriate in your situation 🙂
n
nevin_morgan
08/04/2020, 5:12 PM👍 thanks for that info. I will see if Adam has any insight. I just started a trial to see how fast I burn the credits they give me lol. Meltano has really helped me get the data flowing, but I need to figure out the nexus of all of this which is the warehouse.
n
nehil_jain
08/04/2020, 10:28 PMHi @nevin_morgan I use Snowflake a lot. I can help you figure out pricing
n
nevin_morgan
08/04/2020, 10:29 PMI am in a bit of a unique position. I would need snowflake to come in under the cost of my in house compute resources, or be so much better that it justifies the cost which has been the rub
n
nehil_jain
08/04/2020, 10:29 PMWhat is your cloud provider? what are is the biggest source of data for your setup? how many users/bi tools will query snowflake?
nehil_jain
08/04/2020, 10:30 PMSnowflake pricing is a combination of storage + compute. In my opinion, storage is very very cheap. It's all about the compute. I think its similar for big query as well. Your storage comes down to storing raw data on gcs vs s3 vs azure blob store for both so dont worry about that
n
nevin_morgan
08/04/2020, 10:31 PMI really doubt it will cost me less than 400 a month for 6 or 8 clients pulling data from GA, adwords, bing, facebook, and criteo, plus an eventual snowplow setup
n
nehil_jain
08/04/2020, 10:31 PMcompute cost = credits consumed x cost per credit. cost per credit you can assume to be 2 CAD.
nehil_jain
08/04/2020, 10:32 PMif you are going to move to snowplow data I dont think postgres will cut if for you
nehil_jain
08/04/2020, 10:32 PMunless you are okay with migrating in the middle of the project which is usually not a great idea
nehil_jain
08/04/2020, 10:33 PMyou can definitely bring the snowflake cost to be under 400 for initial months but eventually you might exceed 400 as more and more data is being crunched
n
nevin_morgan
08/04/2020, 10:34 PMthat depends on the sort of hardware you have to throw at a problem. I am eyeballing 20 cores / 40 threads of spare capacity with another 48 cores / 96 threads that aren't online yet. with that much compute on tap I can probably get away with https://www.citusdata.com
nevin_morgan
08/04/2020, 10:36 PMit's what makes snowflake a tough sell for me. I have an in house rack full of idle servers, so snowflake needs to show value for me that once I did figure out the math on costs earlier today hasn't quite made sense.
n
nehil_jain
08/04/2020, 10:37 PMmmm, if have db experts and resources for maintaining a performant postgresql citusdata does look interesting but still looks like a OLTP to me
n
nevin_morgan
08/04/2020, 10:41 PMthat has been my exact concern. My ideal scenario is to find an open source OLAP platform I can run on these machines.
nevin_morgan
08/04/2020, 10:42 PMsnowflake would be my ideal solution, but I am a bit hamstrung by a shoestring budget for outside resources courtesy of good old Covid
n
nehil_jain
08/04/2020, 10:45 PMits all pay per use for snowflake and big query but if you still dont like it. then you can try setting up druid or presto