hey all i am using a current tap ``` name tap reporting db s Meltano #singer-taps

hey all, i am using a current tap ``` - name: tap...

Kevin Phan

12/16/2024, 2:15 PM

hey all, i am using a current tap

Copy code

- name: tap-reporting-db-source
    inherit_from: tap-postgres-pipelinewise
    namespace: tap_postgres_pipelinewise
    config:
      <<: *reportingdb_credentials
      logical_poll_total_seconds: 300
      dbname: "postgres"
    select:
    - public-external_balances.*
    - public-direct_deposits.*

that does CDC from postgres to snowflake. the initial run was a full table sync. The replication slots have been created but the transaction log disk usage is rising. It doesnt look like the tap is consuming any wal logs. Is it possible that the initial full table sync did not properly transition to CDC?

Kevin Phan

12/16/2024, 6:00 PM

my logs indicate -

name=tap_postgres level=INFO message=Request wal streaming from 1578/7091E9B0 to 1579/64B0C2D8 (slot pipelinewise_postgres)

so it is definitely consuming. The DB doesnt have too many transactions. I wonder if it is not advancing the

confirmed_flush_lsn

fast enough or not confirming back to postgres that these have been consumed

logs.txt

Kevin Phan

12/16/2024, 6:18 PM

okay maybe we can listen in on a frequently updated table. only thing is can we listen to the table without actually ingesting it to the target? hypoyhtic config would be

Copy code

- name: tap-reporting-db-source
  inherit_from: tap-postgres-pipelinewise
  namespace: tap_postgres_pipelinewise
  config:
    <<: *reportingdb_credentials
    logical_poll_total_seconds: 300
    dbname: "postgres"
  select:
    - public-external_balances.*
    - public-direct_deposits.*
    - public-transaction_history
  replication_method:
    public-transaction_history: "logical"
  sync: false #hypothetic

Open in Slack

Previous Next