xiaozhou_wang
12/23/2022, 3:33 PMrefresh_streams_schema
is run for each WAL message.
This was triggered by a migration in the underlying table the made multiple update to the table and then dropped the column user_id
.
2022-12-23T00:35:02.798984Z [info ] time=2022-12-23 00:35:02 name=tap_postgres level=INFO message=Detected new columns "{'user_id'}", refreshing schema of stream XXXXXX cmd_type=elb consumer=False name=tap-postgres producer=True stdio=stderr string_id=tap-postgres
After the migration, at the next scheduled Meltano run, a large number of UPDATE
WAL messages exist but during discovery, the user_id column cannot be found. As a result, tap-postgres keeps triggering refresh schema after each WAL message. This slows down the whole process until it times out.
Has anyone come across this problem before?