Hi there, is it best to have Meltano write to a persistent disk?
We're running Meltano in AWS ECS, where we launch a container for each run, and the container stops afterward. Anything that was written to disk at that point (as opposed to our RDS Postgres metadata DB) is lost.
We just ran into an issue where our metadata DB went into auto-recovery. The DB recovered in a few minutes, but a couple hours passed until the Meltano's metadata DB connection timed out. At that point, an exception was thrown and the job failed. However, the run wasn't marked as failed in the metadata DB. As a result of it not being marked as failed, the next time the job ran it checked the DB and assumed there was already an existing job running and exited. This required manual intervention to fix.
I was wondering if there would have been some file on the local disk that Meltano would have used to recover from this automatically (if it were run again in the same container, or with access to the same disk)?