When we built

tap-athena

in the live coding session two weeks ago, I commented on how generic the code ended up being. The only thing Athena-specific was the creation of the sqlalchemy conection string, which concatenated the creds. We didn't yet to any Athena-specific optimizations but the result was still functional and a solid entry-level tap. Alternatively, we could get the same functionality by allowing the user to install their own driver and provide their own fully built connection string. What do folks think of a basic

tap-sqlalchemy

or (anther name for the same thing)

tap-dbapi

(emphasis on DBAPI as described in PEP-249)? Presumably, If the proper drive is installed into the virtual environment, then this could give basic interop for anything in this list or this one - as well as for community-created drivers like PyAthena which are not on either list. The tap would just accept as config input whatever connection string was needed for the specific requested driver. Questions for further discussion: 1. Would this be valuable as an entry-level connector - and/or a starting point for other taps? 2. Any thoughts on the naming

tap-sqlalchemy

vs

tap-dbapi

? I think I like referencing DBAPI but curious to other's thoughts.

Does it make sense to instead make it an extra, i.e.

singer-sdk[sqlalchemy]

or

singer-sdk[dbapi]

instead? I can imagine limits to how generic a tap can be when all those databases will have a lot of non-overlapping settings

maybe I shouldn't lump these together but it seems similar to something like

tap-files

(`tap-fs)`` or something like that. SEO and usability wise there should at a minimum be entries for each DB that's usable due to SQLAlchemy . From a marketing standpoint having every source available on SQL Alchemy would be better, but maybe the quality isn't high enough?

@edgar_ramirez_mondragon - From the SDK, yes, I think it makes sense to make the sqlalchemy/dbapi dependency an optional one - so SaaS-based taps won't have any extra bloat. That said, the SDK is not runnable as a tap, so my question was more towards the tap side: should we have a generic tap that works with anything sqlalchemy works with?

@visch that's a very similar case, thanks for sharing! The role of the abstraction layer would be played by something like fsspec or pyfilesystem.

Yep. That's what I was thinking. There may still be other generic config options like stream maps, etc., but connectivity would be driven by that one url input.

Left a comment for server side cursors: https://gitlab.com/meltano/sdk/-/merge_requests/44#note_676452948

https://github.com/MeltanoLabs/tap-sqlalchemy Untested as of yet, but here's the quick adaptation from

tap-athena

to

tap-sqlalchemy

. Basically the exact same code but expects a

database_url

config instead of the Athena-specific cred arguments. Before broadly promoting this, we need to document the limitations (no log-based replication, for instance) and the caveats regarding performance limitations using the generic, versus a source-optimized tap like tap-postgres. For high-volume streams, users will definitely need a native built tap for their source. However, I do think this has some viable use cases for relatively low-volume streams or cases where the in-built SDK functionality is more important than source-specific optimizations.