Greetings I want to store my data with a different output ta Meltano #troubleshooting

Greetings! I want to store my data with a differen...

Samuel Nogueira Farrus

01/17/2025, 2:18 PM

Greetings! I want to store my data with a different output table name. This is my

meltano.yml

Copy code

version: 1
default_environment: prod
project_id: some_project_id
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+<https://github.com/MeltanoLabs/tap-postgres.git>
    config:
      host: source_db_host
      port: source_db_port
      database: source_db_name
      user: source_db_user
      max_record_count: 5000
    select:
    - schema-table.'*'
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres
    config:
      host: dw_host
      port: dw_port
      database: dw_name
      user: dw_user
      use_copy: true
      default_target_schema: dw_schema
  mappers:
  - name: meltano-map-transformer
    variant: meltano
    pip_url: git+<https://github.com/MeltanoLabs/meltano-map-transform.git>
    mappings:
    - name: rename
      config:
        stream_maps:
          table:
            __alias__: table_new_name

But when I run

meltano run tap-postgres rename target-postgres

, the result table still is

dw_schema.table

instead of

dw_schema.table_new_name

. Anyone knows what am I doing wrong?

✅ 1

Reuben (Matatika)

01/17/2025, 3:57 PM

Maybe you need to include the schema in the referenced stream name? Not sure what the format would be, but you can run

meltano select tap-postgres --list

to check stream names.

Copy code

stream_maps:
          dw_schema__table:
            __alias__: table_new_name

Edgar Ramírez (Arch.dev)

01/17/2025, 4:59 PM

I think since streams have the pattern

schema-table

, it should be

Copy code

stream_maps:
          schema-table:
            __alias__: table_new_name

👍 1

Samuel Nogueira Farrus

01/17/2025, 5:10 PM

Thank you for the reply! With this:

Copy code

stream_maps:
  schema-table:
    __alias__: table_new_name

the new table is created, but no data is inserted. Also, the table columns are not created, only the _`_sdc_*`_ ones. Any ideas?

Samuel Nogueira Farrus

01/17/2025, 5:14 PM

Also, if there is any other way to rename the output table, I would appreciate

Edgar Ramírez (Arch.dev)

01/17/2025, 6:31 PM

what do you see with

meltano select tap-postgres --list

Samuel Nogueira Farrus

01/17/2025, 7:15 PM

Copy code

meltano select tap-postgres --list
2025-01-17T19:12:22.456161Z [warning  ] Failed to create symlink to 'meltano.exe': administrator privilege required
2025-01-17T19:12:22.463313Z [info     ] The default environment 'prod' will be ignored for `meltano select`. To configure a specific environment, please use the option `--environment=<environment name>`.
Legend:
        selected
        excluded
        automatic
        unsupported

Enabled patterns:
        schema-table.'*'

Selected attributes:

Reuben (Matatika)

01/17/2025, 7:54 PM

Can you change

schema-table.'*'

'schema-table.*'

and try it? Also, if you run with

--all

(

meltano select tap-postgres --list --all

) you can see what is deselected as well, which may help debug here.

🙌 1

👍 1

Samuel Nogueira Farrus

01/17/2025, 7:57 PM

Thank you all! Changing to

'schema-table.*'

worked like a charm!

🔥 2

22 Views

Open in Slack

Previous Next