I have my ingestion program (uses Meltano with multiple subprocesses, ) running periodically now, but I occasionally see the error.

This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (sqlite3.OperationalError) database is locked.

[SQL: UPDATE runs SET last_heartbeat_at=? WHERE runs.id = ?]
[parameters: ('2024-11-13 13:36:28.605212', 4)]
(Background on this error at: <https://sqlalche.me/e/20/e3q8>) (Background on this error at: <https://sqlalche.me/e/20/7s2a>)

I am using tap-mssql and target-parquet, so would I be correct assuming this error is related to

systemdb

? Each worker runs independent streams, and each run of the pipeline is launched in a new container. If so, is there anyway to avoid using systemdb? I'm using --force anyway, and I have no need for systemdb to be tracking running processes or states.

so would I be correct assuming this error is related to

systemdb

?

yeah, that seems to be the case

If so, is there anyway to avoid using systemdb? I'm using --force anyway, and I have no need for systemdb to be tracking running processes or states.

Not at the moment, but I'm open to ideas 🙂