Hi Running into trouble with `tap mysql` the test of which i Meltano #getting-started

Hi! Running into trouble with `tap-mysql`, the tes...

simon_podhajsky

03/10/2023, 7:24 AM

Hi! Running into trouble with

tap-mysql

, the test of which is alternating failing with: •

Exception: only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported

• and

No RECORD message received

(although after switching around config, I can no longer get to that message)

meltano select tap-mysql --list

outputs the correct list of tables after I've added under

plugins::extractors::tap-mysql::select: '**.**'

, so it can see the database and connect to it. I can verify the local MySQL db is running and has content via DBeaver. I'm stumped. Full config in thread.

simon_podhajsky

03/10/2023, 7:25 AM

Copy code

version: 1
default_environment: dev
project_id: 9dce0c4c-b583-4450-bf28-dc6b30988d73
environments:
- name: dev
  # config:
  #   plugins:
  #     extractors:
  #     - name: tap-mysql
  #       select:
  #       - '*.*'
  #       metadata:
  #         '*.*':
  #           replication-method: FULL TABLE
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    config:
      host: localhost
      port: 3306
      user: root
      database: debatovanicz
      engine: mysql
      ssl: false
    metadata:
      '*.*':
        replication-method: FULL TABLE
    select:
    - '*.*'
  loaders:
  - name: target-duckdb
    variant: jwills
    pip_url: target-duckdb~=0.4
    config:
      filepath: C:\Users\spodh\.dbt\adk_wrapped.db
      default_target_schema: debatovanicz
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

visch

03/10/2023, 1:19 PM

Try changing

Copy code

metadata:
      '*.*':
        replication-method: FULL TABLE

Copy code

metadata:
      '*.*':
        replication-method: FULL_TABLE

Just a hunch

simon_podhajsky

03/10/2023, 1:49 PM

Thank you! I was sure that was it, but alas, still getting the first message even with this corrected. My reading suggests that i should get a catalog file to debug further - is that correct?

visch

03/10/2023, 1:54 PM

Yeah something like

meltano invoke --dump=catalog tap-mysql > catalog.json

and then look at the metadata that's being set for the streams I guess? I bet it's the filter syntax as for streams there's no field to select If I'm setting full_table for every stream I normally do this https://gitlab.com/vischous/oracle2mssql/-/blob/master/oracle2mssql/meltano.yml#L46-48

Copy code

metadata:
      '*':
        replication-method: FULL_TABLE

visch

03/10/2023, 1:55 PM

Good excercise to see what it is set to now, and what it is after the metadata filter up date to

simon_podhajsky

03/10/2023, 3:32 PM

May this be a lesson to me in continuing habits like configuration by hand - using

meltano config tap-mysql set _metadata '*' replication-method FULL_TABLE

puts the config in the right place.

visch

03/10/2023, 3:35 PM

Glad you're up and running!

Open in Slack

Previous Next