<https://benn.substack.com/p/case-for-consolidatio...
# data-news
the world appears to be full of people who will pay a premium for a consolidated collection of tools, packaged and bundled into a neat little suite, sold on a single bill.
In data products, modularity, flexibility, and composability are markers of our craftsmanship. But most people “don’t care about the aesthetics of software in the same way non-plumbers don’t care about the aesthetics of plumbing—they just want their information in the right place or their hot water to work.”
d
Interesting, something we shouldn't lose sight of with our messaging (@e.kyle3)
d
I prefer open-source frameworks—unless Fivetran is just more reliable than Stitch or Airbyte
Clearly meltano got left out of this comparison since reliability isn't an issue 😏
d
Reading the article now, but just from these quotes: I'm looking forward to having Meltano projects on the hub, as pre-packaged data platforms for those who don't want to put it together themselves, or want to have a template to start with. They can then host these with us, and we can have an onboarding flow for each that doesn't require leaving the web UI, from setup through accessing and managing the tools. Web-based “meltano init” on GitHub is the first step in this, and a complete UI around Git, feature-branch playground environments, and all plug-in management is second. I imagine that the Git repo and meltano.yml backing project would still be accessible, for those who want to work through GitHub/Lab or local development, but it would no longer be required.
The same applies to other principles. I prefer open-source frameworks—unless Fivetran is just more reliable than Stitch or Airbyte. I like modularity—until it leads to “drops in signal quality, lower data transfer speeds,” and a failing user experience. As they say, a value isn’t a value until it costs you something. And if I’m honest with myself, thinking as a buyer rather than builder of data tools, I’m not willing to pay very much to hold on to them.9
Put another way, how much more would you pay for Fivetran if the product were exactly the same, but it was built on an open-source framework? How much less would you pay for dbt if it were proprietary? Yes, these changes might make vendors build better or worse software, but that’s the point: Tenets like these are a means to an end, and as a customer, I pay for the ends.
I think an open source connector ecosystem already got us to a great end: Singer taps and targets for over 300 sources and destinations, where proprietary tools like Fivetran support far fewer Of course, only a (growing!) portion of those are of the same quality and robustness of Fivetran’s, which is his point, but I'll take “mostly there and great with an hour of work” over “completely nonexistent” any day
In any case, I agree with much more in this post than I disagree. It’s easy to forget that the part of the market that cares about the modern data stack is a tiny piece of the total market for data platforms, and there’s no reason to think that winning the smaller market will mean winning the larger ones. Which is not to say that that smaller market exists and is sizable, in terms of people buying the tools, and money they’re willing to spend for it, and we have a path towards the larger market through pre-built Meltano-powered data platforms with tight integration between the components and a consistent experience