@borna_almasi

2. Recovery from hard failures:

For long-running replications, things can go wrong on the machine itself. Hardware/virtual hardware fails.

If the machine crashes without the process having a chance to exit and the state being made available, we lose the state.

My thoughts: I can imagine a world where the persistence of state is a sort of adapter and not purely reliant on the file system. Or perhaps some “Singer middleware spec” as a pipe that sits between the tap and the target and

Has anyone else experienced these failures? How have you handled them?

Again, if you are using Meltano for orchestration, we attempt to store the emitted state securely back to the central system DB even while the tap is running and before the entire stream completes. The singer spec uses STATE messages to emit periodic checkpoints which can be used to resume incremental streams at or near the point of failure. Is that the behavior what you are looking for?