I'm using tap-postgres (log based replication) and...
# troubleshooting
x
I'm using tap-postgres (log based replication) and target-s3-jsonl and run into a bug where
refresh_streams_schema
is run for each WAL message. This was triggered by a migration in the underlying table the made multiple update to the table and then dropped the column
user_id
.
Copy code
2022-12-23T00:35:02.798984Z [info     ] time=2022-12-23 00:35:02 name=tap_postgres level=INFO message=Detected new columns "{'user_id'}", refreshing schema of stream XXXXXX cmd_type=elb consumer=False name=tap-postgres producer=True stdio=stderr string_id=tap-postgres
After the migration, at the next scheduled Meltano run, a large number of
UPDATE
WAL messages exist but during discovery, the user_id column cannot be found. As a result, tap-postgres keeps triggering refresh schema after each WAL message. This slows down the whole process until it times out. Has anyone come across this problem before?