Hello, I am having trouble getting tap-postgres (default variant) to download only the columns I selected.

meltano select tap-postgres --list --all

shows the selected entities in green correctly and the rest in red, but when I run

meltano invoke

I still get the whole table with all columns. Has anyone had this problem before? Thanks in advance.

meltano select tap-postgres --list --all

, specifically

meltano select tap-postgres --all

adds

*.*

to your select list for the tap which selects everything. Before you run

meltano invoke

take a look at your

meltano.yml

to be sure it's right. There's a chance you are hitting some kind of cache issue if so there's some files you can explicitly remove or just

rm -rf .meltano

and then reinstall with

meltano install

I tried both ways but it didn't help. I replicated the issue in a fresh project, pasting in this configuration

version: 1
default_environment: dev
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-postgres
        select:
        - schema_name-table_name.column1.*
        - schema_name-table_name.column2.*
        - schema_name-table_name.column3.*
        - "!schema_name-table_name.column4.*"
        - "!schema_name-table_name.column5.*"
- name: staging
- name: prod
project_id: test2-44664cf2-f094-4389-a48b-67c272b11d02
plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    config:
      user: user
      dbname: dbname
      port: 99999
      filter_schemas: schema-name
      default_replication_method: FULL_TABLE

(which I created previously using straightforward cli commands), and ran just 2 commands, one to set the password and then

meltano invoke tap-postgres > output/raw

. In the output I can see all the columns regardless on the ones I selected/unselected. Please tell me if you see something immediately wrong with my config. Thanks in advance

Did you figure this out?

No, but I didn't spend too much time on it to be honest.