wave Using pipelinewise tap postgres into pipelinewise tap Meltano #troubleshooting

:wave: Using pipelinewise tap-postgres into pipeli...

Simon Schmitke

02/06/2024, 9:59 PM

👋 Using pipelinewise tap-postgres into pipelinewise tap-snowflake. I'm struggling to find out how to specify a replication-key for INCREMENTAL method as it's not specified in the docs. My meltano.yml looks like:

Copy code

...
plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    config:
      host: $TAP_POSTGRES_HOST
      port: $TAP_POSTGRES_PORT
      user: $TAP_POSTGRES_USER
      password: $TAP_POSTGRES_PASSWORD
      dbname: $TAP_POSTGRES_DATABASE
      default_replication_method: INCREMENTAL
      # default_replication_key: updated_at
    select:
    - public-my_table.*

using

default_replication_key

has no effect and I am met with the error that the replication_key is null. Any ideas?

✅ 1

Simon Schmitke

02/07/2024, 4:00 PM

Got it:

Copy code

plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    config:
      host: $TAP_POSTGRES_HOST
      port: $TAP_POSTGRES_PORT
      user: $TAP_POSTGRES_USER
      password: $TAP_POSTGRES_PASSWORD
      dbname: $TAP_POSTGRES_DATABASE
    select:
      - public-my_table.*
      # - public-*.*  # for all tables
    metadata:
      'public-my_table':
      # '*': # for all tables
        replication-method: INCREMENTAL
        replication-key: updated_at

4 Views

Open in Slack

Previous Next