interested to hear the new direction meltano is taking. As a user of pipelinewise, it's good to see someone else taking on the challenge of open source fivetran/stitch. I don't know anyone at transferwise, but have you talked to the folks behind pipelinewise? Their main innovation on top of meltano's current direction is to add fast-sync for large mysql/postgres tables (e.g. I can sync a 200m row table in 4 hours with snowflake charges of just a few cents at the end vs stitch/singer's incremental approach). This is a feature that fivetran I believe also has. That said, they don't have the upfront column filtering approach or ui that meltano has that could be useful. In any event while pipelinewise's project appears to be primarily for internal needs with features for companies with large dbs (private jira and just public github), they have spent around 2-3 years with 2-3 employees building it out. They have a high quality snowflake target and recently appear to have more or less forked the singer python repo. https://github.com/transferwise/pipelinewise-singer-python I'd say it's worth at least a conversation to potentially avoid duplication of effort.

@eamon Thanks for raising this! Meltano and PipelineWise would definitely benefit from collaboration on Singer spec and tooling matters, and specific taps and targets, even if the applications themselves are optimized for different use cases and scenarios. I haven’t reached out to them yet, but I’ll do so once I’m back to work (out sick since this weekend, unfortunately)

This is a great idea. We are also currently running Pipelinewise in production.

Can pipelinewise handle custom taps?

I haven't done it but yea you just need to wrap your tap a small bit so that pipelinewise can invoke it automatically. https://transferwise.github.io/pipelinewise/project/contribution.html

We have expanded upon the singer specification to enable some required elements like common naming conventions for stream ID’s names, catalog arguments et cetera. We need to register the connectors behaviour to be compatible with the PipelineWise common YAML configuration file structure.

That sounds similar to https://meltano.com/docs/contributor-guide.html#known-plugins, but it doesn't look like PipelineWise has an equivalent to Meltano's

meltano add --custom extractor

, to allow adding not-yet-known taps to your project without having to fork the tool and either use the fork or wait for the changes to be accepted upstream.

ah, great stuff, hopefully there'll be some scope to collaborate! It'll be great to have an open source alternative to paying $$$$ to fivetran!

So what was the outcome of the call ?

@keat Thanks for asking! @adam_roderick can comment on his takeaways regarding the reasons he set up the call in the first place; I was just happy to tag along 🙂 Peter shared that in general, the TransferWise data team (who build PipelineWise) would rather focus their efforts on building really great taps and targets for the services and DBs they use, than on building and maintaining the tool to run them (PipelineWise). In fact, they would not have built their own runner if a great one had existed when they originally started evaluating Singer, and looking into the future, they would be more interested in rallying behind a dedicated Singer runner project (like Meltano) than in continuing to push and support their own version, since that would allow them more time to work on the taps and targets they actually depend on. He said he'd find some time to play around with Meltano and see how it stacks up to PipelineWise with regard to its support for the PipelineWise taps and targets, but I don't think he's had a chance to do so yet. In any case, if we continue our efforts to truly make Meltano the best open source Singer runner, it sounds like a future collaboration may take the shape of them using their experience to improve Meltano (through feedback or actual contributions) and perhaps eventually even them making it the recommended runner of PipelineWise's own taps and targets 🙂

Whoa, thanks for sharing! Yeah, one reason why I didn’t choose pipelinewise is the lack of custom tap and custom target. This is really cool, am really excited for Meltano and thank you so much for working on this

My reason for getting the teams connected is I believe singer has a lot of potential, and I want to increase the activity and collaboration of the community. Peter and Douwe are great and both projects have a lot of merit. I've been contributing to some of PipelineWise's taps/targets and Meltano. What I love seeing is that everyone shares the vision of collaboration and an open source community around ELT