Question regarding SQL targets like target-postgre...
# singer-targets
s
Question regarding SQL targets like target-postgres and target-snowflake: is one database connection shared across all sinks or is a new connection created for each sink? I couldn't tell from looking at the code. https://github.com/MeltanoLabs/target-postgres/blob/main/target_postgres/sinks.py https://github.com/MeltanoLabs/target-snowflake/blob/main/target_snowflake/sinks.py
r
I believe the lifecycle of a connection is controlled by the pool configuration, and I don't think either of these targets or the SDK implementation stray from the SQLAlchemy defaults in any meaningful way. https://docs.sqlalchemy.org/en/20/core/pooling.html#connection-pool-configuration • SDK: https://github.com/meltano/sdk/blob/244bbf1960f77f883dc5e424843baa5f71b20c75/singer_sdk/sql/connector.py#L881-L912 • `target-postgres`: uses SDK default implementation • `target-snowflake`: https://github.com/MeltanoLabs/target-snowflake/blob/5a341fa06845818d75ac7396576398a67948a508/target_snowflake/connector.py#L240-L273 If you want to customise how connections are handled, unfortunately it doesn't look like either target or the SDK allow configuration of the connection pool at the moment. Targets would have to support passing params to the
create_engine
call via
config
-- specifically the
pool*
params in this case: https://docs.sqlalchemy.org/en/20/core/engines.html#sqlalchemy.create_engine.params.pool.
👍 1