Hi All I was wondering if someone could explain to me how me Meltano #troubleshooting

Hi All, I was wondering if someone could explain t...

drew_ipson

12/02/2021, 5:04 PM

Hi All, I was wondering if someone could explain to me how meltano.yml configurations are applied? I’m currently using tap-postgres as an extractor and have configured my meltano.yml to look like this:

Copy code

# tap-metadata
          - name: tap-metadata
            config:
              dbname: ****
              default_replication_method: FULL_TABLE
              filter_schemas: public
              host: ****
              port: "5432"
              ssl: "false"
              user: ****
              start_date: 2020-12-21 02:08:39.212793
              select:
                - public-job_descriptor.*
                - public-job_field_descriptor.*
                - public-job_query_descriptor.*
                - public-organization.*
                - public-parent_account.*
                - public-parent_account_user.*
              metadata:
                public-job_descriptor:
                  replication-method: INCREMENTAL
                  replication-key: updated_at
                  updated_at:
                    is-replication-key: true
                public-job_field_descriptor:
                  replication-method: INCREMENTAL
                  replication-key: updated_at
                  updated_at:
                    is-replication-key: true
                public-job_query_descriptor:
                  replication-method: INCREMENTAL
                  replication-key: updated_at
                  updated_at:
                    is-replication-key: true
                public-oauth_info:
                  replication-method: INCREMENTAL
                  replication-key: updated_at
                  updated_at:
                    is-replication-key: true
                public-organization:
                  replication-method: INCREMENTAL
                  replication-key: updated_at
                  updated_at:
                    is-replication-key: true
                public-parent_account:
                  replication-method: INCREMENTAL
                  replication-key: updated_at
                  updated_at:
                    is-replication-key: true
                public-parent_account_user:
                  replication-method: INCREMENTAL
                  replication-key: updated_at
                  updated_at:
                    is-replication-key: true

However, after looking at the documentation, it appears there is a CLI command

select

to configure these for the tap. After running

meltano tap-metadata select --list

I see that it is configured to pull all tables. How can I clear this list and use my meltano.yml configuration? Should I just manually set each one and remove them from the yaml?

drew_ipson

12/02/2021, 5:58 PM

Update: I was able to reset the select --list by running the

meltano select <tap-name> <table>

, but still how the meltano.yml configuration is being applied.

drew_ipson

12/02/2021, 6:49 PM

After additional testing, the select configurations are not being retained inside the system DB either. Any ideas on how to make this configurations persistent? or do I need to run

meltano select <tap-name> <table.config>

every time?

drew_ipson

12/02/2021, 7:30 PM

This was resolved! This issue dealt with formatting the

meltano.yml

tap-postgres with the select fields above the config section. Simple mistake but these setting configurations take place outside the

config

in tap-postgres.

edgar_ramirez_mondragon

12/02/2021, 7:36 PM

Hey @drew_ipson, yes indeed!

select

was incorrectly nested inside of

config

. The CLI (

meltano select <tap-name> <table>

) should always lay it out correctly. We also have https://gitlab.com/meltano/meltano/-/issues/3078 in the backlog, along with the necessary publishing of the meltano.yml JSON schema. Please leave a 👍 in the issue if you think we should prioritize it 😄

drew_ipson

12/02/2021, 7:40 PM

@edgar_ramirez_mondragon Just did and added some comments. Thanks for putting this on my radar!

Open in Slack

Previous Next