Not sure if it's kosher to talk about Not-Meltano here, but this announcement today is interesting, in that it sounds like the way they may address the community-supported extractor problem is through a revenue sharing model.

Now, regarding the participative model, there are still a lot of details to figure out, but the general idea is that maintainers of connectors would have the option to get a revenue share when the connectors are being used in the paid version of Airbyte. In exchange, maintainers would be responsible for SLAs, new features, and bug fixes for the said connector.

of course, the devil is in the "a lot of details" 🙂 I do like the idea that if I develop a tap that gets used in the paid version, I would benefit directly. But I wonder if that is the root problem, or whether the problem is that people move on to some other job and no longer have access to the service they built. Do contributors afterwards not get a cut? A bug bounty program would seem to be better in that case https://airbyte.io/blog/a-new-license-to-future-proof-the-commoditization-of-data-integration

We've thought a lot about this as well and they run into the same incentive challenges any open source project runs into. Particularly in an enterprise software situation, many customers will just want to pay someone and not have to worry about problems with specific connectors. Revenue share, if done in a transparent way is an interesting idea and would probably make sense to a lot of consultancies. That said, your point about individual contributors is interesting too. Bug bounties are a littler bit hard to imagine I think because most bugs in a connector would be solved by moving to the SDK and anything left would be considered critical to the functioning of the tap. So figuring out how to sort that out could be challenging. All that said, we really believe that a fully distributed and open source ecosystem is possible - there are thousands of client API libraries and other open source projects that companies rely on for critical infrastructure and there's no reason that can't be true for Singer and the data ecosystem. A revenue share within a future hosted version of Meltano could be one way to support the larger Singer ecosystem, but that's a very narrow view into Singer and its connector library. We're pretty confident that focusing on building up the entire ecosystem by bringing transparency to the existing standard and community, along with building great tooling around taps/targets and ways to run them, is long-term better for everyone within the Singer community.

Absolutely. I wonder if there's a way to enlarge the Singer community? dbt packages are already starting to have the same issues that Singer had -- every instance I've used a package there have either been surprise issues or I got blocked during an upgrade, etc. I agree that it's a whole ecosystem challenge, and I wonder what the boundaries of that ecosystem really are when there are literally thousands of packages and variants, and they bleed across EL into T and even elsewhere. (For example, if Meltano were to make targets reverse-etl compatible.)

Can you share more about what you mean by enlarge the Singer community? Also, regarding dbt packages, I think of this https://benn.substack.com/p/when-are-templates-going-to-happen a lot

i think ideally it would be something larger than just "EL", which is what singer covers now. when i think about the singer community (and tap-development work) it is pretty transactional and behind the scenes. Often, i am doing it to get on to the fun part. there are great discussions to be had, especially right now, but once you have a data platform owner has most of their taps figured out, dont they basically forget about the singer community? obviously meltano has a broader vision than just EL, and i think it would be awesome to see it come up as the data engineering analogue to dbt's analytics engineering community, because if the community is inviting, there will be more people to help with the long tail of taps to manage.

Ah, I see what you're saying 🙂 So, I think there's a few reasons the Singer community (and extended data community) will continue to grow. One is the clear reversal of ELT - Singer is fully capable of handling "Reverse ETL" use cases and the SDK is written in a way that you can write taps and targets for basically anything. As reverse etl is still relatively new for modern data stack teams we're not focusing on this as much, but Singer and Meltano are still great ways to manage them because it's "just" data movement. Another is that as we are able to add more lineage and observability to Meltano and its supported plugins, folks will naturally be able to understand the lifecycle of their data better. To be more specific, if Meltano has a native understanding of specific streams and how those tie into downstream dbt models, that connection is now explicit and we can surface metadata on the entire dataflow to the end user. dbt wouldn't stop at sources.yml - it could tie into specific streams from tap-hubspot (for example). And a third is our overall strategy / vision for Meltano is to be that foundation for any teams ideal data stack, so by using Meltano you're part of a larger community of data professionals, even if maybe all you're using it for is to run taps and targets. Eventually we'll get to the point where you can do a lot more with Meltano and that community will naturally grow. So yeah, you're thinking along the same lines. Expect to see a few blogs posts very soon about Mission/Vision/Strategy from us 🙂

it would be awesome to see meltano abstract the operational management of data tools in the same way dbt has abstracted much of the operational management of data platforms.

A power move would be for meltano to be compatible with singer spec and airbyte tap spec! Though maybe worth letting the dust settle and ensure that airbyte taps remain high quality... Complex problems!