Hi I almost feel embarassed but I am unable to get a single Meltano #troubleshooting

Hi, I almost feel embarassed but I am unable to ge...

tomas_votava

07/13/2023, 1:18 PM

Hi, I almost feel embarassed but I am unable to get a single tap-mysql to work. I keep on getting

Exception: only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported

when I run

meltano run tap-mysql target-jsonl

I have

meltano==2.19.1

, I've tried multiple versions of

tap-mysql

. This is my tap config:

Copy code

plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql==1.3.8
    config:
      host: <http://relational.fit.cvut.cz|relational.fit.cvut.cz>
      port: 3306
      user: guest
      database: classicmodels
      filter_dbs: classicmodels
      ssl: false
      engine: mariadb
    select:
    - classicmodels-products.*
    - classicmodels-orders.*

The

1.3.8

version pin is a remnant from when I kept trying multiple versions. Could you please help me to deal with this? I've also tried deleting

.meltano

directory multiple times and re-running

meltano install

thomas_briggs

07/13/2023, 1:26 PM

It looks like you're missing the

metadata

options

thomas_briggs

07/13/2023, 1:26 PM

Something like

Copy code

select:
    - classicmodels-products.*
    - classicmodels-orders.*
    metadata:
      classicmodels-*:
        replication-method: INCREMENTAL

tomas_votava

07/13/2023, 1:39 PM

Oh, thanks, that works! Could you please point me to where in docs this is described so that I know where to look the next time?

thomas_briggs

07/13/2023, 1:47 PM

I think this covers it: https://docs.meltano.com/guide/integration#setting-metadata

thomas_briggs

07/13/2023, 1:48 PM

It's one of those things I think you can only find after you know what you're looking for. 😕

tomas_votava

07/13/2023, 1:49 PM

Oh, I see, I think I've seen this when browsing the docs before, but I never felt it to be something required, thanks a lot again, it's working now 🆗

thomas_briggs

07/13/2023, 1:51 PM

You are welcome. 🙂 There's also some useful information here: https://github.com/transferwise/pipelinewise-tap-mysql

thomas_briggs

07/13/2023, 1:52 PM

Meltano is just running the tap and feeding it the configuration it wants, after all... so technically this is a tap-mysql thing, not a meltano thing.

tomas_votava

07/13/2023, 1:56 PM

Yeah, I get it, I just didn't understand how to pass this additional config to the singer tap and thought there was something wrong with my environment rather than the config 👌

thomas_briggs

07/13/2023, 1:57 PM

I totally understand lol

user

07/13/2023, 2:23 PM

I actually just tried to address this on the hub https://github.com/meltano/hub/pull/1427 related to https://github.com/meltano/hub/issues/1366. It merged this morning. tap-mysql was an outlier here, most taps default to FULL_TABLE and so does the SDK. It also didnt help that tap-mysql's error message is wrong 😞

5 Views

Open in Slack

Previous Next