I just set up tap postgres transferwise with LOG BASED repli Meltano #troubleshooting

I just set up tap-postgres (transferwise) with LOG...

mert_bakir

04/27/2023, 2:04 PM

I just set up tap-postgres (transferwise) with LOG-BASED replication. 1. I make a change on the source-db 2. Then run

meltano run tap-postgres target-postgres

it says:

Copy code

[info     ] time=2023-04-27 13:57:18 name=tap_postgres level=INFO message=Lastest wal message received was 3B/547F9290 cmd_type=elb consumer=False name=tap-postgres producer=True stdio=stderr string_id=tap-postgres

and repeats, and waits. 3. Apply another change on the source-db 4.

tap-postgres target-postgres

process ends. The first change is applied to the target-db but the second change is not there yet. What's the logic here? Should there be a constantly listening connection between tap and source-db? Do I just create a frequent enough schedule?

mert_bakir

04/27/2023, 8:38 PM

I think the tap breaks because of this parameter

break_at_end_lsn

. 🤔

mert_bakir

04/27/2023, 9:03 PM

So... I was thinking. Postgres writes changes to the WAL for some time (or possibly for up to some byte threshold). We just need to set up a job that reads the changes for every x minute. That's what meltano and tap-postgres does. The job, read-write operation, shouldn't take longer than the x minute interval. Because then jobs may overlap (?) and cause problem. Not sure just guessing. How do I determine the x minute interval? Other than trial and error, do I need to monitor some metric?

mert_bakir

04/27/2023, 9:06 PM

Still I don't get this. Why the process

tap-postgres

didn't end immediately after I ran it, because there were existing changes already, but wait until I make another commit on the db while the tap-postgres is running.

Open in Slack

Previous Next