heya just scoping out meltano for the first time trying to u Meltano #troubleshooting

heya - just scoping out meltano for the first time...

ben_marengo

04/20/2021, 1:28 PM

heya - just scoping out meltano for the first time. trying to use mysql tap but getting error

Exception: only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported

. meltano.yaml:

Copy code

extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    config:
      database: bens_db
    select:
    - '*.*'
    metadata:
      bens_table:
        replication-method: INCREMENTAL
        replication-key: date_modified

the metadata doesnt seem to have been added to

.meltano/run/tap-mysql/tap.properties.json

(but the table and its columns have been) can anyone give me a hint what i might be doing wrong?

visch

04/20/2021, 1:44 PM

Output the catalog to take a look, is the easiest debug step. With meltano elt https://meltano.com/docs/command-line-interface.html#how-to-use-4

--dump=catalog > catalog

Invoke you can do the same. My guess is that you are selecting a bunch more streams then just bens_table. You could change your select to 'bens_table.*' and that might just work right away too

ben_marengo

04/20/2021, 1:49 PM

hey derek - dump=catalog shows the correct table/columns (no more no less), but the metadata

Copy code

replication-method: INCREMENTAL
replication-key: date_modified

are nowhere to be see

ben_marengo

04/20/2021, 1:51 PM

Copy code

select:
- 'bens_table.*'

shows no difference 😕

visch

04/20/2021, 1:51 PM

Is the stream name correct? Maybe bens_table isn't matching? I'm not sure if running elt in debug mode would tell you if the streams being matched or not, it might help

visch

04/20/2021, 1:52 PM

maybe bens_table isn't matching so it's not adding the INCREMENTAL metadata info

visch

04/20/2021, 1:54 PM

Could try

Copy code

metadata:
      '*':
        replication-method: INCREMENTAL

To see if that's the case. There may be a better way to tell if it's matching or not

ben_marengo

04/20/2021, 1:57 PM

Copy code

select:
- '*.*'
metadata:
  bens_db-bens_table:
    replication-method: INCREMENTAL
    replication-key: date_modified

this works

ben_marengo

04/20/2021, 1:57 PM

"tap_stream_id": "bens_db-bens_table"

ben_marengo

04/20/2021, 1:58 PM

weird, but ok!

ben_marengo

04/20/2021, 1:58 PM

thanks for your help!

visch

04/20/2021, 1:58 PM

No problemo! Glad I could help

visch

04/20/2021, 2:06 PM

And yeah the tap that you use makes up the stream name based on the data it sees. So for tap-mssql it decided to do the $dbname-$tablename for the stream names. Each tap handles these slightly different depending on the data. How I like to handle it is to run the tap in discovery mode, and then output the catalog to see what the data looks like first, then I build out my select / metadata

visch

04/20/2021, 2:08 PM

A lot of the singer / meltano stuff is getting the right mental model in your head. Someday it'd probably be good to have video tutorials or something to explain thought processes and things

ben_marengo

04/20/2021, 3:36 PM

tbh, filling in (and not truncating) the help descs for the cli would be a good step forward!

aaronsteers

04/20/2021, 5:30 PM

@ben_marengo - I can see how that could be confusing. I’ve logged an issue in our issue tracker to improve this: Add `--help `docs for meltano CLI (#2716) · Issues · meltano / Meltano · GitLab Thanks for the feedback on this.

Open in Slack

Previous Next