Does anybody know why `pipelinewise-tap-postgres` ...
# announcements
d
Does anybody know why
pipelinewise-tap-postgres
cannot get messages from the wal log with logical replication? It just echoes
Waiting for first wal message
but I see a couple of logs above it says
Request wal streaming from 0/FE7F90B8 to 0/FF49E768 (slot pipelinewise_xxxxxxx)
d
d
initially it was working, for 2-3 days, and stopped all of a sudden :
d
does incremental (key based) sync handles row updates?
It does, if you use an
updated_at
timestamp as the replication key!
d
meaning my apps should take care of the
updated_at
?
d
yeah, key-based replication will only pick up updates if it has a way to tell if a row got updated since the last sync
It's pretty common for ORM frameworks (like ActiveRecord in Rails) to automatically increment an
updated_at
column whenever a row is updated
I have no idea why the WAL approach would suddenly break though 😕 That seems like something to dive into deeper. It doesn't look like the tap is using it incorrectly
d
the tap detects everything correctly. It sees it needs to use the
pipelinewise_xxxxx
(database name instead of xxxxx) should be used, it sees that it should stream wal records from LSN X to LSN Y, but receives nul when psycopg2 tries to read from the log
d
Right. I'm sure that that tap is running in production with log-based replication in various places, so I wonder if there's something off about your postgres server
d
hmmm I’m using AWS RDS Aurora Postgres v12.4
d
Hmm, no reason to think that wouldn't be configured perfectly then
j
@dejan_pejcev finding a similar issue with
tap-postgres
. Were you able to find a solution for this?