<@U06CEPJEWFN> - Thanks for this question. Cross-p...
# singer-tap-developmenta
aaronsteers
09/29/2021, 8:12 PM@dav - Thanks for this question. Cross-posting here as it relates to tap development: https://meltano.slack.com/archives/CKHP6G5V4/p1632946219380100
aaronsteers
09/29/2021, 8:15 PMWe've thought of generic implementations, but these in my experience have (at least) two significant challenges: (1) pagination is generally different across APIs; there are some common patterns, for sure, but no one-size-fits-all implementation that I know of, and (2) most existing abstraction layers out there don't necessarily define incremental streams or parent-child relationships very well.
So, while a generic tap might be difficult, the above challenges are handled in a generic way in our Tap and Target SDKs (sdk.meltano.com). Sometimes a tap can be built with <100 lines of code, plus the streams' schema definitions.
v
visch
09/30/2021, 1:56 PMAnother core issue is that by making a generic tap you're pushing the mappings to those api calls to configuration. So now you have a unique configuration for every SaaS app out there, where do you maintain that list of configuration values?
visch
09/30/2021, 1:57 PMMy experience is they are never that simple, but at the same time when you just need 1 api endpoint historically throwing together a tap takes too long. Today with the SDK it's fairly close to being easy enough for that 1 endpoint to throw a tap together
d
dav
10/07/2021, 7:30 PMhi @aaronsteers / @visch
thanks for getting back.
regarding 1, pagination
a requirement would be that the API supports Cursors, sometimes called Connections. this way, modifications to the data source length shouldn't affect the current slice of data being read. A server correctly implementing this should allow introspection and a generic way to safely fetch ALL data exposed by a given schema.
re 2, parent-child relationships
like you said, to make use of the scraped data-set you'd need to define this somehow. But to just fetch ALL exposed schemas, we don't need to worry about how things are structured.
what i'm after is more like a backup / mirror of anything exposed.
a good use case would be to create a read replica on a local machine using an identical API