I was working on tap-bigquery following closely ta...
# singer-tap-development
a
I was working on tap-bigquery following closely tap-athena screencast. And I have couple of suggestions for SQLConnector design:
Currently method
discover_catalog_entries
relies on sqlalchemy introspection working correctly and does not support customization.
In bigquery case, due to non-straightforward logic sqlalchemy-bigquery does not work as expected
(in tables discovery it returns table name with schema e.g
schema1.table
, and in getting table properties when schema is also provided as separate argument, treats schema in table name as project name -> fails to find such table)
I think it would be nice to split
discover_catalog_entries
into several overridable functions: •
_get_all_schemas
_get_all_tables_in_schema
_get_all_columns_in_table
in this way I would be able to override just a
get_all_tables
part and reuse the rest of the logic
e
Hi @andrey_tatarinov, we're all for better API design! Would you be willing to contribute an MR for this refactor?
a
I will give it a try
e
Awesome! Feel free to tag me and ask questions 🙂
a
I'm not sure about extracted functions names, feel free to suggest alternatives