I am trying to understand when the lock mechanism is created and checked when using a filesystem backend. When I call

meltano run

for a simple elt job it only checks the state_id and latest_running. It looks like there is some StateStore management but I don't see where in the code it would raise an error if there is a lock file.

cc @cody_hanson I think you'd know best on this

Hi @nancy_wong 👋. Whenever state is read, written, or cleared, the lock mechanism is used because we have an "bookmark writer" output that gets output from the singer plugin and adds state whenever Singer

STATE

messages are emitted. The locking mechanism for state backends does not actually prevent multiple pipelines of the same job (as determined by its state ID); it merely prevents multiple jobs running at the same time from conflicting with one another when reading and writing state to the same state backend. Does that help?

That clarifies the purpose of the lock mechanism. Thank you Cody!