#9947 spike: Use the state backend or a new extensible mechanism to handle job locking for concurrent runs Issue created by edgarrmondragon Feature scope Other Description Background When

meltano run

starts a pipeline it checks whether an identical pipeline (same state ID) is already running by querying the system database (

Job

table via

JobFinder.latest_running

). If one is found, the run is aborted with a message instructing the user to wait or use

--force

to bypass the check. This mechanism has two limitations: 1. Tied to the system database. The check (and the heartbeat-based staleness logic) only works when all runners share the same system database. In setups where the system DB is unavailable or not shared (e.g. distributed runners using only a cloud state backend), there is no protection against duplicate concurrent runs. 2. Not integrated with the state backend. The state backend already has an

acquire_lock

abstraction (

StateStoreManager.acquire_lock

), but it is only used to guard state reads and writes, not to prevent duplicate pipeline executions. The two locking concerns are currently completely separate. Goal of this spike Investigate whether the job-locking check for concurrent runs can be moved to, or made extensible through, the state backend (or a new dedicated lock backend), so that: • Locking works correctly for runners that share only a state backend and no system database. • The

--force

flag semantics are preserved — it should still be possible to bypass the lock check. • Each state backend can provide an appropriate distributed-lock implementation (e.g. conditional writes for S3/GCS/Azure, advisory locks or

SELECT FOR UPDATE

for databases). Questions to answer • Should the "is this pipeline already running?" check move into

StateStoreManager

(alongside

acquire_lock

), or should it be a separate

LockManager

abstraction? • What is the right default when a backend cannot provide a distributed lock (fail fast, warn, or skip)? • Does the heartbeat / staleness concept from the

Job

model need a counterpart in the state backend locking mechanism? • Is there a clean migration path from the current DB-backed check to a backend-aware one? Related code •

src/meltano/core/block/extract_load.py

— current duplicate-run check and

--force

handling •

src/meltano/core/job/finder.py

—

JobFinder.latest_running

/ staleness logic •

src/meltano/core/state_store/base.py

—

StateStoreManager.acquire_lock

and

StateIDLockedError

•

src/meltano/core/state_store/db.py

— no-op

acquire_lock

(DB backend delegates to transactions) •

src/meltano/core/state_store/filesystem.py

— file-based

acquire_lock

(also used by S3/Azure/GCS) meltano/meltano