https://linen.dev logo
#singer-tap-development
Title
# singer-tap-development
b

BuzzCutNorman

06/14/2023, 7:01 PM
I wanted to pass this along incase other SDK based SQL Tap developers might have run into this. I was seeing a database session per Stream. The sessions would stay until the
meltano run
was complete. I found that if you add a
tap_connector
property in your Tap's class in `tap.py`:
Copy code
default_connector_class = mssqlConnector
    _tap_connector: SQLConnector = None

    @property
    def tap_connector(self) -> SQLConnector:
        """The connector object.

        Returns:
            The connector object.
        """
        if self._tap_connector is None:
            self._tap_connector = self.default_connector_class(dict(self.config))
        return self._tap_connector
Then you copy over
catalog_dict
property and
discover_streams
method from the
SQLTap
class in
tap_base.py
. Edit
catalog_dict
to use the new
self.tap_connector
and
discover_streams
to pass
connector=self.tap_connector
to the
default_stream_class
instance as it is appended. This will look like this:
Copy code
self.default_stream_class(
		tap=self,
		catalog_entry=catalog_entry,
		connector=self.tap_connector
	)
This allows all the Streams to utilize one SQLAlchemy connection pool. When doing test extracts of databases with 80 tables or more I only see a single session utilized. I just merged this change to tap-mssql--buzzcutnorman if you want to see a complete example. If you give this a try in your own SDK based SQL Tap please let me know what the result was, good or bad.