Hi, I am trying to replicate a table from postgres to snowflake, but no data replicated after running

meltano etl

command. I used

meltano select

to specify entities and attribute, here is the output of command `meltano select tap-postgres --list`: Legend: selected excluded automatic Enabled patterns: table2.Id Selected attributes: The Selected attributes section has nothing. I also tried to dump the catalog using

meltano invoke --dump=catalog tap-postgres

, but it didn't give the new table "table2" in the output at all. Did I miss anything? It will include the new table2 if I run command

meltano invoke tap-postgres -d

, but it gives me 0 row-count, event though there is a record in that table.

I think your select pattern needs to include the schema name... maybe something like public-table2.id?

Tried public-table2.id, still no lock. is it expected that selected attributes section is empty? will invoke --dump command refresh data from database?

I think the selected attributes section is empty because your select rules don't correctly identify any tables. When you run

--dump catalog

what are the stream names it produces? Generally it'll be <schemaName>-<tableName>

--dump catalog

won't output table2, I have to use

invoke -d

to get the catalog which shows the output for table2

Hrm. I'm not 100% sure this will fix it but try removing .meltano/run/tap-postgres/tap.properties.cache_key and see if that fixes it. The catalog does get cached so that might explain why you see it in the output of

invoke -d

but not

--dump catalog

Thanks Thomas, the empty selected attributes issue solved, now I ran into another issue:

Exception: Unrecognized replication_method None

Here is the content I have in meltano.yml - name: tap-postgres select: - public-table2.id metadata: public-table2: replication-method: FULL_TABLE I dumped the config file, and I don't see replication-method in the config file, tried deleted tap.properties.cache_key and tap.properties.json, still no luck, any suggestion?

Where are you defining those settings? Under an environment or in the plugin definition? There's at least one known bug with metadata settings not being picked up correctly so where you're putting them might matter.

I used

meltano select tap-postgres public-table2 id,

the content above is from meltano.yml

Understood. Where in meltano.yml does it appear though? That might be the issue.

version: 1 default_environment: dev project_id: 59bd65db-f577-49a9-8791-17c79da5a5b8 plugins: extractors: - name: tap-postgres variant: transferwise pip_url: pipelinewise-tap-postgres config: dbname: mydb user: postgres loaders: schedules: - name: mssql-to-snowflake extractor: tap-mssql loader: target-snowflake transform: skip interval: 0 */1 * * * start_date: 2022-06-03 035942.140356 environments: - name: dev config: plugins: extractors: - name: tap-postgres select: - public-table2.id metadata: public-table2: replication-method: FULL_TABLE loaders: - name: target-postgres config: user: postgres dbname: mydb default_target_schema: public - name: staging - name: prod

Try copying the metadata section into the extractor definition, i.e.

plugins:
  extractors:
  - name: tap-postgres
    variant: transferwise
    pip_url: pipelinewise-tap-postgres
    config:
      dbname: mydb
      user: postgres
    metadata:
      public-table2:
        replication-method: FULL_TABLE

and see if that fixes it. That would be a bug if it does, I think, but like I said I know there's at least one open bug about metadata not being read correctly so it's definitely possible.

finally it works now. Thanks @thomas_briggs

I don't know about a timeline for the fix. The only solution I know of at this point is to set all metadata in the parent plugin 😕

Thanks.