BuzzCutNorman
06/14/2023, 7:01 PMmeltano run
was complete. I found that if you add a tap_connector
property in your Tap's class in `tap.py`:
default_connector_class = mssqlConnector
_tap_connector: SQLConnector = None
@property
def tap_connector(self) -> SQLConnector:
"""The connector object.
Returns:
The connector object.
"""
if self._tap_connector is None:
self._tap_connector = self.default_connector_class(dict(self.config))
return self._tap_connector
Then you copy over catalog_dict
property and discover_streams
method from the SQLTap
class in tap_base.py
. Edit catalog_dict
to use the new self.tap_connector
and discover_streams
to pass connector=self.tap_connector
to the default_stream_class
instance as it is appended. This will look like this:
self.default_stream_class(
tap=self,
catalog_entry=catalog_entry,
connector=self.tap_connector
)
This allows all the Streams to utilize one SQLAlchemy connection pool. When doing test extracts of databases with 80 tables or more I only see a single session utilized. I just merged this change to tap-mssql--buzzcutnorman if you want to see a complete example. If you give this a try in your own SDK based SQL Tap please let me know what the result was, good or bad.