It looks like start_date isn't listed as an option for tap-postgres (https://github.com/transferwise/pipelinewise-tap-postgres). I presume that it isn't an option?

It does look like it accepts a state file but it's not clear to me what's actually in there.

Honestly feeling a little defeated at the moment :/

you're in the same place I was one month ago 😓 some taps are indeed not in good shape, and tap-postgres is one of them as you have also discovered in https://meltano.slack.com/archives/C01TCRBBJD7/p1656089836794389

The tap-postgres IMO looks to be in good shape. Don't get defeated, you've got this! Head over to this documentation: https://transferwise.github.io/pipelinewise/connectors/taps/postgres.html Look at the bit "*Configuring what to replicate*", and specifically the bit that discusses

schemas

and under that

tables

. Also have a look at Replication Methods. I'm assuming that you would like things that have changed to be replicated over. Looks like the

LOG_BASED

replication method is what you want. You will need to enable logs in your postgres so that rows that have been added or modified are transferred. If you indeed have a reliable updated timestamp or some other incrementing field in your table, you could also look at

INCREMENTAL

replication method, which doesn't require logs. Hope that helps.

@ilkka_peltola don't get me wrong, tap-postgres works. but maintainers are not very responsive in issues and pull requests. most importantly, there seems to be a core disagreement or misunderstanding about one key thing: who manages source schema changes. tap-postgres decided to automatically refresh the schemas every time (which caused some of these bugs), but I'm not at sure that a Singer tap should be responsible for it.

That makes sense. And actually, I'm right this moment trying to implement pipelinewise-tap-mongodb and getting a bit frustrated with the level of documentation on it 😞 😁