I am having a few issues with tap-postgres (pipelinewise). The first has been covered here where the schema is refreshed and thus the columns selected (using meltano select tap-postgres) are ignored and all columns are selected from the source table. The source table is about 100 columns wide and I only need about 10 so this is greatly increasing the amount of data flowing into the DW. Might hot patch this. Another issue I am having is with the default replication method. It seems I can set that to either LOG_BASED or FULL_TABLE but cannot set it to INCREMENTAL. If I then set the replication method for a table (meltano config tap-postgres set metadata schema-name-table_name replication_method INCREMENTAL), it is ignored. Instead the default replication method is set to FULL_TABLE it will do that (Beginning new Full Table replication) rather than the INCREMENTAL replication method. Anyone have any thoughts? It seems as if I can just use LOG_BASED or FULL_TABLE and only for all columns of source table with tap-postgres. (I will submit a bug report on this as soon as I have a chance to write it up.)

Have you tried

glob

syntax with the selects? https://docs.meltano.com/reference/command-line-interface#examples-6:~:text=meltano%20select%20tap%2Dgitlab%20%2D%2Dexclude%20%22*%22%20%22*_url%22 You can do the same with replication methods. I think that's what you're saying is that you want to select all columns in a table even if they change, right?

For the first issue, I only want to select certain columns from a table. Not all. This was covered in this thread: https://meltano.slack.com/archives/C01TCRBBJD7/p1652253018834649 which also seems to link to a bug report for the tap that doesn't seem to be addressed.