We’d all be better off—well, all of us except ELT vendors—if more apps did what Segment does, and wrote data straight to the warehouse themselves.

SaaS vendors could charge for this service, opening a new revenue stream that’s considerably easier to build than an embedded BI tool. If customers are already paying usage fees to Fivetran to sync data to their warehouse, the sale is easy: Pay us instead of them. They could be aggressive about it too, by packaging bulk access to their APIs with the syncing service, which would choke off Fivetran’s ability to run the syncs on their own.

Hard disagree this logic isn't valid it's what everyone has thought would happen for years and years and years. Everyone also thought all Extract and Load would be commoditized too. If everyone's target was a warehouse you'd be right Ben, it's not and guesing all those combos won't work either so you'll be back to opening an api up

Yeah, I’m also trying to imagine what managing this single sync in snowflake (or SFDC) is actually going to look like…

A final version of a better Fivetran is Fivetran…a Snowflake Company (or Amazon, or Microsoft, or Databricks Company). For the same reasons I mentioned earlier, database vendors have an obvious incentive to make it easier for customers to load data into their products. It gives them more opportunities to spin their usage meters; it means they don’t have to recommend other vendors to their customers to help them source data.

No, see oracles history

I love it 😄

Thanks for sharing, still a good read. It's fun watching people try to generalize etl / elt workloads

Interesting post - I want this world where my data just shows up clean in my warehouse but I struggle with the prediction of how that plays out. The idea that every SaaS vendor would support this and do it consistently somehow seems unlikely 🤮. As a user imagining if this existed I'd be happy for a little while but once one of my SaaS apps doesnt support it I have to figure out my EL strategy or if one vendor decides that it wants to charge more than other vendors then maybe I would prefer to pay fivetran/roll my own to save money. I'd have inconsistently manage integrations in many platforms (starts to sound like the challenges people are having with scheduling jobs across their MDS components i.e. fivetran + dbt cloud, etc.) and negotiate rates differently with all. That sounds like a head ache to me

Thanks for sharing, still a good read. It's fun watching people try to generalize etl / elt workloads

It's been interesting to see this play out over and over again in the past several years. When I was getting started, this piece about ETL being basically a solved problem was still required reading for data platform folks and since then it's gone the opposite direction where rather than ELT being completely commoditized, there's been the rise of a whole new role (analytics engineer) for just the "T" part.

It's been interesting to see this play out over and over again in the past several years. When I was getting started, this piece about ETL being basically a solved problem was still required reading for data platform folks and since then it's gone the opposite direction where rather than ELT being completely commoditized, there's been the rise of a whole new role (analytics engineer) for just the "T" part.

Crazy right? We don't learn and I think it's mainly because most folks focus on integrations just enough to get the value they need (very clear incentive and motivation which is pretty much every company running analytics today). Then we realize it's hard, point to something different that would make our lives better we think just because it's different even though the grass isn't always greener. re saas apps writing directly to your warehouse, snowflake being the magic fix for issues, etl tools being the magic fix, elt tools being the fix, data lake being the fix, realtime streaming, debezium, Apache caml, hopefully not Derek saying meltano is the way, etc etc The problem feels intractable which means break it down into it's smallest peices, iterate and share. 🚀 Meltano