Currently Singer is extremely slow for large datasets, 2Tb is extremely large data for for this.

There's a suggestion for BATCH records, which should improve throughput, but it is not implemented yet.

I hate to say this, but in your case best solution would be to use pipelinewise, they have shortcut fastsync implementation for initial load.

After that with some trickery it is possible to move synchronisation state to Meltano and continue to do incremental syncs with it.

we've been having similar issues with our replica

https://github.com/spacecowboy/pipelinewise-tap-postgres/pull/1 Not sure if it will behave better with the read replica yet however

I suspect that some people may still want the old behavior to be available because it's more transactional - gets all the rows at once. I wonder if there are any edge cases with the limit and offset (e.g. if the underlying rows change, or the order isn't stable, or the db connection drops out or errors) 🤔 . If there are more than

itersize

rows that match a single bookmark value, then singlepage will never progress. e.g. say itersize is 10k and my incremental state field is "last_updated" and there are 20k records with the same last_updated value, then singlepage will just get the first 10k each time. One of my workmates foresaw this and suggested adding a secondary key (e.g. a primary key) so that the bookmark is always a unique point. But I think a scenario like this would be rare. But, at this stage, your code looks pretty much exactly how I imagined it. Nice work.