Hi everyone! Is there a way to confirm explicitly ...
# troubleshooting
o
Hi everyone! Is there a way to confirm explicitly that tap-postgres uses the LOG_BASED replication? I was looking through log files and can't find any explicit log entry that states it. I have no errors whatsoever, but my concern is that it may be still using FULL_TABLE as a replication method.
v
A log statement would be good I think we should add that 😅 If you have a property of
_sdc_lsn
then you have log based on
Not so sure on the log statment, if you instead did
meltano invoke tap-postgres --dump=catalog
and then found your stream there's a replication_method listed for the stream you're using as well. We could still do the log statement but it should really apply to every stream then we'd say something like "Starting {stream_name} - {replication_method} sync" Which isn't a bad idea, maybe just something we should do in the sdk
o
Hey Derek, thank you so much for the quick response! I haven't expected such a friendly community 🙂 So I'm afraid my concern was confirmed since I don't see _sdc_lcn in properties
Speaking of
meltano invoke tap-postgres --dump=catalog
let me check if I have the latest version since it complains about no such option
Yea, so --dump=catalog is an option to meltano invoke, not to the tap-postgres, once I dumped the catalog and found the stream, it says "replication_method": "" which is a bummer. I'll continue my troubleshooting and will let you guys know if I manage to figure this out.
So the problem was related to the fact that I copy-pasted the config bit from the Readme
Copy code
metadata:
"*":
  replication_method: LOG_BASED
  replication_key: _sdc_lsn
It should be
Copy code
metadata:
  "*":
    replication_method: LOG_BASED
    replication_key: _sdc_lsn
I also see the following line in the log now
Copy code
2024-07-11 15:44:43,573 | INFO     | tap-postgres.public-country | Beginning log_based sync of 'public-country'...
v
We always accept PRs from folks feel free to do this kind of thing yourself looks like the spaces were just miseed!