Is meltano's job table designed to record incremental state during a run? Let's say I have a large run that might take 2 hours. If it finishes, the job's status is

SUCCESS

. If it fails at 1 hour, the job status is failed, and the run has to start over again and do the entire 2 hours. Is there a pattern where interim state can be recorded, so if the job fails part way through, it could pick up where it was last successful?

Meltano should be merging those interim states with new ones afaict according to https://docs.meltano.com/guide/integration#internal-state-merge-logic. It also depends on how often the target emits the state payload.

@edgar_ramirez_mondragon maybe it is a weakness of the loader. From what I can see in the docs, even failed runs should store the incremental state (in the failed job record?) and it will be merged. Is that right?