Hello all In my `meltano yml` file I have a `tap mysql` conf Meltano #troubleshooting

Hello all. In my `meltano.yml` file I have a `tap...

ian_lewis

05/31/2023, 5:14 PM

Hello all. In my

meltano.yml

file I have a

tap-mysql

config where I have a

select:

block which lists my required tables, and a

metadata:

block which provides the replication settings. However, when I do

meltano run

it appears that the

select:

block is completely ignored and the replication settings are the ones used. The result of this is that the tap tries to fetch all of the tables in the database not just those listed in the

select:

. Is this expected? Have I missed a bit of the documentation which explains this behaviour?

pat_nadolny

05/31/2023, 5:37 PM

@ian_lewis would you mind sharing your meltano.yml with any sensitive data removed? This is not expected behavior, at least that I'm aware of

edgar_ramirez_mondragon

05/31/2023, 6:06 PM

Also, running

meltano select tap-mysql --list --all

might help you debug what's selected and what's not

ian_lewis

05/31/2023, 7:58 PM

Copy code

version: 1
default_environment: dev
project_id: 3572c119-4866-4ff7-b5a0-f2fa18a5b6d4
environments:
- name: dev
- name: staging
- name: prod
- name: xxxx-test
  config:
    plugins:
      extractors:
      - name: tap-mysql
        config:
          host: <http://xxxx-test.cluster-xxxxxxx.aws-region.rds.amazonaws.com|xxxx-test.cluster-xxxxxxx.aws-region.rds.amazonaws.com>
          user: dbuser
          database: xxxxxxxx
          ssl: true
        select:
        - db1.*
        - db1_history.*
        - db1_activation.*
        - db1_activation_item.*
        - db1_cancellation.*
        - db1_cancellation_item.*
        - branch.*
        - country.*
        - currency.*
        - language.*
        - lender.*
        metadata:
          country:
            replication-method: full
            replication-key: code
            selected: true
          language:
            replication-method: full
            replication-key: code
            selected: true
          currency:
            replication-method: full
            replication-key: code
            selected: true
          '*':
            replication-method: incremental
            selected: true
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

Here is the

meltano.yml

@pat_nadolny

pat_nadolny

05/31/2023, 8:12 PM

Ah I see so you're wanting to do - what happens if you remove the

selected: true

from the

metadata entry? The metadata selected config is how the

select

feature works behind the scenes, so I think your metadata config is overriding your select config and selecting everything.

ian_lewis

06/01/2023, 6:49 AM

Thanks, I'll experiment with that.

ian_lewis

06/01/2023, 11:00 AM

@pat_nadolny I did some playing around and a bit more investigation and ended up with something like this...

Copy code

select:
        - xxxxxxxx-db1_history.*
        - xxxxxxxx-db1_activation.*
        - xxxxxxxx-db1_activation_item.*
        - xxxxxxxx-db1_cancellation.*
        - xxxxxxxx-db1_cancellation_item.*
        - xxxxxxxx-branch.*
        - xxxxxxxx-country.*
        - xxxxxxxx-currency.*
        - xxxxxxxx-language.*
        - xxxxxxxx-lender.*
        metadata:
          country:
            replication-method: full
            replication-key: code
          language:
            replication-method: full
            replication-key: code
          currency:
            replication-method: full
            replication-key: code
          '*':
            replication-method: incremental

So it appears that in order to correctly select the table in the

select:

block I must ensure that the table name is prefixed with the schema name like so

xxxxxxxx-branch

. As long as I have done that I see that in the

tap.properties.json

file the value

"selected": true

is set for the table in question. I only found this when looking at

tap-postgres

It's not clear in the documentation that it is required to prepend the schema name onto the table name.

user

06/01/2023, 11:51 AM

@ian_lewis awesome! glad you got it figured out. Yeah I hear you, its a little tricky because some taps are different than others in terms of how they name streams thats why its helpful to list available streams like Edgar suggested https://meltano.slack.com/archives/C054MK4SE8J/p1685556387371219?thread_ts=1685553260.680179&cid=C054MK4SE8J. We can definitely add more hints in the docs though, where would it have been most helpful for you? On the hub page or one of the docs pages?

ian_lewis

06/01/2023, 12:33 PM

There's definitely a good case for the stream naming convention to appear on the hub page. That's where I looked, but didn't find anything for tap-mysql.

user

06/01/2023, 12:52 PM

ok cool - thanks for the suggestion! I opened https://github.com/meltano/hub/issues/1366 to track it

Open in Slack

Previous Next