what's the up-to-date guide for setting up a custom tap for an API, are these two the best? https://www.stitchdata.com/blog/how-to-build-a-singer-tap-infographic/ https://meltano.com/tutorials/create-a-custom-extractor.html#create-the-plugin-s-package

Those are probably the best. I used https://blog.zhaw.ch/splab/2019/12/20/building-a-singer-io-tap-for-an-open-data-source/ when getting started, it has a nice progression

thanks @adam_roderick . Just out of interest, roughly how long does it take to make a custom tap for an API with a few endpoints?

It depends on a LOT of things, like any software development effort. The security and authentication with the API, the number of entities and attributes, the complexity of those entities, any dynamic endpoints where you can pass in a complex body and get a dynamic result).

great, thanks @adam_roderick. So definitely sounds like a better approach than writing some custom python in airflow. This would be for an internal microservices API so would have more control. I think the team are more comfortable exposing an API rather than getting data via the underlying postgres. Another option is via an event stream. Anyway appreciate the insight, I gather there are a fair number of people using custom taps just perhaps not that much content about it online.

Yeah, they aren't too bad to write. The benefits are the consistency of the framework, and the ability to use them with existing targets

I found it much easier to start from an existing solution & adapt (easier to see how the pieces fit together): https://github.com/singer-io/tap-github Just cloned it and updated schemas & logic