https://linen.dev logo
Join Slack
Powered by
`tap-mysql only INCREMENTAL, LOG_BASED, and FULL T...
# troubleshooting
h
tap-mysql only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported
, I got this error when i user tap-myql , can help me ?
Copy code
version: 1
default_environment: dev
project_id: 11e34570-bed3-4835-961c-aa7a1a44a2aa
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-mysql
        config:
          host: 127.0.0.1
          database: test
          user: test
          filter_dbs: test
        select:
        - '*.*'
        metadata:
          test_user:
            replication-method: FULL_TABLE
      loaders:
      - name: target-postgres
        config:
          host: 127.0.0.1
          database: test
          user: test
          default_target_schema: public
- name: staging
- name: prod
@pat_nadolny can help me ? 🤩
p
@hello_dev I wonder if you're metadata settings aren't quite configured right. Right now you have 
test_user
as the only stream with a replication-method set, could you be syncing more than just that stream? Maybe try updating to the below example to set the replication method for all streams to 
FULL_TABLE
then once its working you can update to more fine grain settings, like what your had before.
Copy code
metadata:
          "*":
            replication-method: FULL_TABLE
h
@pat_nadolny I change config , but also can't work
Copy code
metadata:
          "*":
            replication-method: FULL_TABLE
this all the json
Copy code
{
  "streams": [
    {
      "tap_stream_id": "test-test_user",
      "table_name": "test_user",
      "schema": {
        "properties": {
          "id": {
            "inclusion": "automatic",
            "minimum": -2147483648,
            "maximum": 2147483647,
            "type": [
              "null",
              "integer"
            ]
          },
          "name": {
            "inclusion": "available",
            "maxLength": 255,
            "type": [
              "null",
              "string"
            ]
          }
        },
        "type": "object"
      },
      "stream": "test_user",
      "metadata": [
        {
          "breadcrumb": [],
          "metadata": {
            "selected-by-default": false,
            "database-name": "test",
            "row-count": 1,
            "is-view": false,
            "table-key-properties": [
              "id"
            ],
            "selected": true
          }
        },
        {
          "breadcrumb": [
            "properties",
            "id"
          ],
          "metadata": {
            "selected-by-default": true,
            "sql-datatype": "int(11)",
            "datatype": "int",
            "selected": true
          }
        },
        {
          "breadcrumb": [
            "properties",
            "name"
          ],
          "metadata": {
            "selected-by-default": true,
            "sql-datatype": "varchar(255)",
            "datatype": "varchar",
            "selected": true
          }
        }
      ],
      "selected": true
    }
  ]
}
this my log file by exec 
meltano invoke tap-mysql
Copy code
2022-07-06T14:46:47.864678Z [info     ] Environment 'dev' is active
time=2022-07-06 22:46:53 name=tap_mysql level=INFO message=Server Parameters: version: 5.7.38, wait_timeout: 28800, innodb_lock_wait_timeout: 3600, max_allowed_packet: 134217728, interactive_timeout: 28800
time=2022-07-06 22:46:54 name=tap_mysql level=INFO message=Server SSL Parameters(blank means SSL is not active): [ssl_version: ], [ssl_cipher: ]
{"type": "STATE", "value": {"currently_syncing": "test-test_user"}}
time=2022-07-06 22:46:58 name=tap_mysql level=INFO message=Beginning sync for InnoDB table test.test_user
time=2022-07-06 22:46:58 name=singer level=INFO message=METRIC: {"type": "timer", "metric": "job_duration", "value": 0.49161529541015625, "tags": {"job_type": "sync_table", "database": "test", "table": "test_user", "status": "failed"}}
time=2022-07-06 22:46:58 name=tap_mysql level=CRITICAL message=only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported
Traceback (most recent call last):
  File "/meltano-projects/my-meltano-project/.meltano/extractors/tap-mysql/venv/bin/tap-mysql", line 10, in <module>
    sys.exit(main())
  File "/meltano-projects/my-meltano-project/.meltano/extractors/tap-mysql/venv/lib/python3.7/site-packages/tap_mysql/__init__.py", line 443, in main
    raise exc
  File "/meltano-projects/my-meltano-project/.meltano/extractors/tap-mysql/venv/lib/python3.7/site-packages/tap_mysql/__init__.py", line 440, in main
    main_impl()
  File "/meltano-projects/my-meltano-project/.meltano/extractors/tap-mysql/venv/lib/python3.7/site-packages/tap_mysql/__init__.py", line 429, in main_impl
    do_sync(mysql_conn, args.config, args.catalog, state)
  File "/meltano-projects/my-meltano-project/.meltano/extractors/tap-mysql/venv/lib/python3.7/site-packages/tap_mysql/__init__.py", line 383, in do_sync
    config['engine']
  File "/meltano-projects/my-meltano-project/.meltano/extractors/tap-mysql/venv/lib/python3.7/site-packages/tap_mysql/__init__.py", line 354, in sync_non_binlog_streams
    raise Exception("only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported")
Exception: only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported
p
@hello_dev thats strange, your metadata should show up in that json. What command are you running when you get the exception and also what command are you running to get that json you shared?
h
I use command 
*meltano invoke tap-mysql*
@pat_nadolny
my json file is from local file
p
I was able to replicate this locally also. @edgar_ramirez_mondragon do you have any idea whats going on here? The metadata doesnt look like its getting added to the properties file. Its a bit odd that its using a properties file when it advertises a catalog but not sure if thats related
d
The workaround right now is to move the 
metadata
and 
select
properties out of `environments and directly under the top-level plugin definitions, under 
pip_url
p
ah thanks @douwe_maan . Yeah so @hello_dev updating to something like this should work
Copy code
version: 1
default_environment: dev
project_id: 11e34570-bed3-4835-961c-aa7a1a44a2aa
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: pipelinewise-tap-mysql
    select:
    - '*.*'
    metadata:
      '*':
        replication-method: FULL_TABLE
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
environments:
- name: dev
  config:
    plugins:
      extractors:
      - name: tap-mysql
        config:
          host: 127.0.0.1
          database: test
          user: test
          filter_dbs: test
      loaders:
      - name: target-postgres
        config:
          host: 127.0.0.1
          database: test
          user: test
          default_target_schema: public
- name: staging
- name: prod
h
@pat_nadolny can work . Does this documentation need some modification ? when I use 
meltano config tap-mysql set _metadata '*' replication-method FULL_TABLE
what is generated is wrong https://docs.meltano.com/concepts/plugins#metadata-extra
meltano select *
also generated is wrong
e
what is generated is wrong
Can you share the output of both commands and what is generated incorrectly in them? They might both be related to the issue linked above in the thread if environments are at play
p
Are you saying its wrong because of the bug that doesnt allow you to put the metadata within an environment? If thats its, I think the docs are fine and it will work again once the bug has been fixed
h
Got it, looking forward to the fix ❤️
2 Views
Open in Slack
PreviousNext
Powered by