https://linen.dev logo
Join Slack
Powered by
Hi, I am using the pipelinewise tap postgres, have...
# plugins-general
d
Hi, I am using the pipelinewise tap postgres, have configured a wal2json replication slot, did an initial full sync. On every run, it just says 
Waiting for first wal message
, and I have validated that the replication slot gets filled with changes, but somehow the tap cannot see or consume any. It even outputs 
Request wal streaming from 0/95F175D0 to 0/C02F9628
Anybody had similar issues?
d
@dejan_pejcev Can you share the excerpt of the output logs that contains those messages? Is there anything else WAL related?
d
Copy code
meltano                | Running extract & load...
meltano                | Found state from 2021-03-01 22:23:26.135969.
tap-postgres           | time=2021-03-01 22:40:27 name=tap_postgres level=INFO message=Selected streams: ['public-data']
tap-postgres           | time=2021-03-01 22:40:27 name=tap_postgres level=INFO message=No streams marked as currently_syncing in state file
tap-postgres           | time=2021-03-01 22:40:27 name=tap_postgres level=INFO message=Pure Logical Replication upto lsn 3228207456 for (['public-data'])
tap-postgres           | time=2021-03-01 22:40:27 name=tap_postgres level=INFO message=Using pg_replication_slot pipelinewise_xxxx
target-postgres-custom | time=2021-03-01 22:40:27 name=target_postgres level=INFO message=Prefer json over jsonb: true
tap-postgres           | time=2021-03-01 22:40:27 name=tap_postgres level=INFO message=Set session wal_sender_timeout = 10800000 milliseconds
tap-postgres           | time=2021-03-01 22:40:27 name=tap_postgres level=INFO message=Request wal streaming from 0/95F175D0 to 0/C06A8960 (slot pipelinewise_xxxx)
target-postgres-custom | time=2021-03-01 22:40:27 name=target_postgres level=INFO message=Table '"data"' exists
target-postgres-custom | time=2021-03-01 22:40:27 name=target_postgres level=INFO message=columns to add: []
tap-postgres           | time=2021-03-01 22:40:37 name=tap_postgres level=INFO message=Waiting for first wal message
tap-postgres           | time=2021-03-01 22:40:47 name=tap_postgres level=INFO message=Waiting for first wal message
tap-postgres           | time=2021-03-01 22:40:57 name=tap_postgres level=INFO message=Waiting for first wal message
tap-postgres           | time=2021-03-01 22:41:07 name=tap_postgres level=INFO message=Waiting for first wal message
tap-postgres           | time=2021-03-01 22:41:17 name=tap_postgres level=INFO message=Waiting for first wal message
tap-postgres           | time=2021-03-01 22:41:27 name=tap_postgres level=INFO message=Breaking - 60 seconds of polling with no data
meltano                | Incremental state has been updated at 2021-03-01 22:41:27.947174.
meltano                | Extract & load complete!
meltano                | Transformation skipped.
d
d
hmmm maybe it cannot build a list of tables for wal2json correctly?
d
That's what I'd look at next
d
lemme debug print it, I have a fork of the repo
yep, wal2json table is 
public.data
d
😕
@peter_kosztolanyi I hope you don't mind I'm roping you in here again! Any idea why wal2json wouldn't be working as expected here?
d
yep, 
msg
is always 
None
for some reason
hmmm that also seems strange, the 
read_message
in 
ReplicationCursor
class:
Copy code
def read_message(self): # real signature unknown; restored from __doc__
        """ read_message() -- Try reading a replication message from the server (non-blocking). """
        pass
d
d
hmmmm I tried to run ETL on a different table, and it seems it works. The only difference is that the table that works has an 
id
of a 
int serial
and the table that doesn’t work has an 
id
of type 
uuid
d
Interesting. I don't know wal2json well enough to know why that would make difference though 😬
Open in Slack
PreviousNext
Powered by