Hello! I am using

tap-postgres

and

target-postgres

(Pipelinewise variants in both cases). Incremental replication. Blank destination database at this stage. I would like to exclude some columns from being extracted. I have tried config like the following, which should include columns

a

,

b

and

c

, and then exclude `d`:

select:
 - public-foo.a
 - public-foo.b
 - public-foo.c
 # - public-foo.d - commented out - so should be excluded

And also:

select:
 - public-foo.a
 - public-foo.b
 - public-foo.c
 - "!public-foo.d"

As per the documentation here: https://docs.meltano.com/concepts/plugins#how-to-use-3 I’ve also tried excluding the column through

select_filter

. In each case, the output of

meltano select tap-postgres --list --all

says the column has been excluded. But it doesn’t work - I still end up with the extra column, populated with data, in the destination database. Is this a bug? Or am I missing something?

That is a bug with that tap variant. If you search the chat you can see me posting about it a couple of weeks ago and other before that. This is a PR that shows what is happening: https://github.com/transferwise/pipelinewise-tap-postgres The schema is being refreshed and thus ignoring your selections. For me this results in a lot more data coming over than I want.

Aha! Thanks @will_johnson

Hi, I'm blocked by the same bug as well when using this tap-postgres variant. Any workarounds that you recommend to exclude a specific column?

Not at the moment. There are a couple of other issues I have with that tap-postgres varient so I have been meaning to try the singer one and see if that fixes the issues. At some point maybe we need an official/unofficial fork of that tap as it seems like there are a number of pull requests outstanding that address some of the issues.

Is Meltano team interested to maintain an official fork of this tap, I imagine postgres is a common data source in many companies? Meanwhile, is there anything we can help to get the outstanding pull requests merged? 🙂

We found a workaround using this mapper to remove the stream generated by the tap - maybe that might help your usecase? https://github.com/MeltanoLabs/meltano-map-transform Something like this works:

mappers:
    - name: meltano-map-transformer
      pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git@v0.0.4>
      mappings:
        - name: column-remover
          config:
            stream_maps:
              public-<table_name>: # or whatever your stream name is from tap-postgres
                <column_name>: null

Yes this will work, thanks!