So, follow-up question - it seems to me like if the pipeline (potentially only in certain cases), it still effectively "commits" on the Postgres WAL slot. E.g. do stuff in PG, run pipeline, pipeline fails; re-run pipeline, pipeline doesn't find any data in WAL, do stuff in PG, the new stuff comes through pipeline but the previous stuff is lost. Does that line up with other's experience?

@wide-motorcycle-73593 This shouldn't happen, because pipelinewise-tap-postgres stores the WAL log's LSN in its incremental replication state (https://meltano.com/docs/integration.html#incremental-replication-state): https://github.com/transferwise/pipelinewise-tap-postgres/blob/master/tap_postgres/sync_strategies/logical_replication.py#L695 and uses it on a next run to determine where to start: https://github.com/transferwise/pipelinewise-tap-postgres/blob/master/tap_postgres/sync_strategies/logical_replication.py#L548 The tap passes this state to the target, which should only pass it to Meltano once it has actually persisted all of the records up to that point. Meaning that if the target fails halfway through the batch, Meltano should only have seen the previous LSN, and this is what would be passed to the tap on the next run, so that we'd see some duplicate records, but no missing ones. If you're seeing the pipeline fail but the next run still continuing as if the previous pipeline run had succeeded, we may be looking at the target passing the state to Meltano prematurely. pipelinewise-target-postgres only appears to do so after successfully flushing all records, but perhaps there's still a bug here: https://github.com/transferwise/pipelinewise-target-postgres/blob/9470108c89b4a6f00420327ae2c66d48279d5cee/target_postgres/__init__.py#L170

Trying to reconstruct the series of events from the logs, I may have lost the plot a bit. I'll see what I can do.