https://linen.dev logo
Join Slack
Powered by
hi all, I have a sample CDC tap like so: ``` # Co...
# singer-taps
k
hi all, I have a sample CDC tap like so:
Copy code
# Core DB taps
  - name: tap-core-db-testing
    inherit_from: tap-postgres-pipelinewise
    namespace: tap_postgres_pipelinewise
    config:
      <<: *coredb_credentials
      dbname: "postgres"
      logical_poll_total_seconds: 60
    select:
      - public-employees.id
      - public-employees.user_id
      - public-employees.created_at
      - public-employees.updated_at
      - public-employees.deleted_at

      # Default to `*` for all other tables
      - public-conversions.*
Im noticing certain fields from employees table that are still being ingested . I did not do 
-public-employees.*
Can this be achieved through adjusting the stream maps to further filter whats in the catalog? This should work
e
Im noticing certain fields from employees table that are still being ingested
Do you see those fields printed if you run 
meltano select tap-core-db-testing --list
?
k
ah okay it doesnt appear in the catalogue. so i think what happened was that the tap is CDC so i copied over tables from another schema over to the current test schema to test the tap and those tables had the column present. i guess it doesnt drop the prexisting column if it isnt included in the select
if the column exists and i exlcude that column, does the values defer to null for future ingests?
okay so i dont see it printed in the lhe stream after listing but for some odd reason, email column for this table comes in. is there a config we can set to more heavily enforce only the selected fields @Edgar Ramírez (Arch.dev)?
e
can you try running with the 
--refresh-catalog
flag?
k
yup still absent from the catalogue
hey @Edgar Ramírez (Arch.dev) quick question for you. In a DB with low transactions, I am having trouble with the meltano CDC from postgres advancing the lsn. I think we would need to ingest a more frequently updated table to prevent the oldest replication slot lag from increasing, I sthere a way in the extractor to listen to the table but not ingest it?
7 Views
Open in Slack
PreviousNext
Powered by