Hi everyone I want to setup `LOG BASED` replication from Pos Meltano #troubleshooting

Hi everyone! I want to setup `LOG_BASED` replicat...

Denis I.

10/18/2023, 9:22 PM

Hi everyone! I want to setup

LOG_BASED

replication from Postgres to Postgres (both transferwise variant) and faced some issues: 1.

_sdc_deleted_at

column works as expected, but everything other than

PK

seems `NULL`ed in the target table after deletion from the source table. Is there a way to keep values before deletion? 2. When a new table created in source DB, it totally ignored by the tap.

meltano invoke tap-postgres --discover

outputs the new table metadata. Looks like it never rewrites

.meltano/run/tap.properties.json

file. Removing

.meltano

folder and following reinstall helps to see the new table with

meltano select tap-postgres --list

command, but every time I execute

run

command the tap keeps doing initial

FULL_TABLE

load and doesn’t update state (which stored in Postgres by meltano) to start using

LOG_BASED

. Doing

meltano state clear

helps, but it starts

FULL_TABLE

for every previously added tables. Is there something that I’m missing?

Denis I.

10/18/2023, 9:22 PM

meltano.yml

Copy code

- name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    config:
      filter_schemas: public
      default_replication_method: LOG_BASED
      debug_lsn: true
      itersize: 20000
      break_at_end_lsn: true
    metadata:
        "*":
          replication-method: LOG_BASED

- name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      add_metadata_columns: true
      primary_key_required: true
      hard_delete: false
      batch_size_rows: 20000

Denis I.

10/19/2023, 11:12 AM

I’ve tried to change state manually in DB and add the new table to JSON, but it continues to do

FULL_TABLE

and removes the new table from state. At the same time

tap.properties.json

has the new table.

Denis I.

10/19/2023, 12:06 PM

2. Found it in the tap’s github: https://github.com/transferwise/pipelinewise-tap-postgres/issues/107 https://github.com/transferwise/pipelinewise-tap-postgres/pull/144 Haven’t been fixed since 2021 :(

Denis I.

10/19/2023, 8:46 PM

One more issue: due to this change in setup.py Meltano’s pypi installs only 1.8.4 version in environments using python versions > 3.9 https://github.com/transferwise/pipelinewise-tap-postgres/pull/227

Марта Сакаль

02/09/2024, 11:33 AM

Hi @Denis I.! I'm facing the same issue with full table resync for new tables added to the tap, did you find any solution for that?

Denis I.

02/09/2024, 12:21 PM

Hi @Марта Сакаль! Unfortunately I found more issues with Pipelinewise variant and maintainer seems to react very slowly to pull requests (or not to react at all). Other variants didn’t have features I need. So I ended up forking and rewriting some of the logic.

2 Views

Open in Slack

Previous Next