Hey Meltano! I found <this discussion>, where Mic...
# announcements
j
Hey Meltano! I found

this discussion▾

, where Micaël said:
We want to change the logging of the taps, we want to change the OAuth flow of the taps, we want to change the execution model of the taps (like using asynchronous stuff), …
And in this discussion, Douwe wrote:
… there will also be taps that will require varying amounts of changes and some amount of custom code in Meltano before they are ready to be fit into our OAuth flow. That would likely mean forking them and taking on the maintenance burden, which is not ideal, but is already something we've had to do for quite a number of them.
Some time has passed since then. 🙂 A few months later, do you think there is still a need to fork the Singer taps to suit Meltano's workflow?
d
Hi @jan_fertek, good question! A lot has happened since, indeed 🙂 At the time, we were specifically talking about letting non-technical users authenticate with taps using OAuth in Meltano UI. Since not all taps currently support OAuth, those that don't would potentially need to be forked unless we could be confident the current maintainers would be swift to accept contributions, etc. You don't have to use OAuth to authenticate with a tap though, if you're comfortable manually generating tokens and providing them to the tap as config, on the CLI or in the UI, so there's definitely no need to fork Singer taps to suit Meltano's workflow, just if you'd like to tap to support the specific OAuth-in-Meltano-UI workflow.
j
Thank you, @douwe_maan! The "OAuth-in-Meltano-UI workflow" is great, I really like it. And what about the mentioned logging, data chunking and "asynchronous stuff"? Is it possible to build these features on top of common tap or is there a barrier?
d
The better taps implement all three so there's nothing holding us back in a technical sense, but there's a practical barrier in the lack of documented best practices and example and boilerplate code to implement them, resulting in many less mature and less popular taps not feeling quite as production-ready as others. At some point we may run into limitations that can't be solved without forking/extending the Singer spec (https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md), but the spec is quite solid and the existence of multiple Singer-compatible runners is a feature not a bug, so we can still get quite some mileage out of it, to start with (as the greater Singer community) making all taps reach their maximum potential within what the spec allows today 🙂
As described in https://meltano.com/blog/2020/05/13/why-we-are-building-an-open-source-platform-for-elt-pipelines/, we think that will start with building a great production-ready runner (so that's what we're focusing on today!), followed by building better guidance and tooling around tap and target development and testing.
j
Thank you, Douwe! ☺️