https://linen.dev logo
Join Slack
Powered by
Is there any documentation on how to config your c...
# getting-started
j
Is there any documentation on how to config your catalog.json after I run a --discover ? I'm using tap-mysql target-redshift both from transferwise, but I'm having some issues with log_based load method, since every run it runs discover again and it seems to do a full load of the table again, instead of just updating based on the binlog extractors: - name: tap-mysql variant: transferwise pip_url: git+https://github.com/transferwise/pipelinewise.git#subdirectory=singer-connectors/tap-mysql config: host: $MYSQL_HOST port: $MYSQL_PORT user: $MYSQL_USER password: $MYSQL_PASSWORD database: $MYSQL_DATABASE filter_dbs: $MYSQL_DATABASE use_gtid: true select: /*tables ##### */
✅ 1
more details if the issue: root@7598c354aa0c:/app/meltano# meltano invoke tap-mysql --catalog test.json 2025-07-07T162053.663032Z [info ] Environment 'prod' is active {"type": "STATE", "value": {"currently_syncing": "database_x-table_y"}} only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported Traceback (most recent call last): File "/app/meltano/.meltano/extractors/tap-mysql/venv/bin/tap-mysql", line 8, in <module> sys.exit(main()) File "/app/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.9/site-packages/tap_mysql/__init__.py", line 443, in main raise exc File "/app/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.9/site-packages/tap_mysql/__init__.py", line 440, in main main_impl() File "/app/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.9/site-packages/tap_mysql/__init__.py", line 429, in main_impl do_sync(mysql_conn, args.config, args.catalog, state) File "/app/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.9/site-packages/tap_mysql/__init__.py", line 379, in do_sync sync_non_binlog_streams(mysql_conn, File "/app/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.9/site-packages/tap_mysql/__init__.py", line 354, in sync_non_binlog_streams raise Exception("only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported") Exception: only INCREMENTAL, LOG_BASED, and FULL TABLE replication methods are supported i'm defining the method on the stream level: { "streams": [ { "tap_stream_id": "database_x-table_y", "table_name": "table_y", "replication_method": "LOG_BASED", "selected": true, "schema": {
e
You don't need to run discovery yourself, Meltano is doing that for you under the hood, and refreshing the catalog should not affect your catalog overrides (ie 
select
and 
metadata
in 
meltano.yml
)
j
I'm not sure why it's being ignored, but even though I set the replication method as 
LOG_BASED
in the metadata, it was only respected after I explicitly defined it in the 
catalog.json
. Until then, Meltano was defaulting to full table replication. I only discovered this after running:
Copy code
meltano el tap-mysql target-redshift --catalog custom_catalog.json
At that point, I finally received the message indicating that GTID mode was not enabled on my MySQL database. Also, it's not feasible to define all the tables directly in 
meltano.yml
since I have more than 5,000 lines of configuration. The file can't handle all of that due to its size limitations.
e
did you try setting the 
replication-method
in metadata?
j
yes I did
Copy code
metadata:
   "*":
      replication-method: LOG_BASED
just updating, it is now using LOG_BASED, with the metadata config
Copy code
metadata:
   "*":
      replication-method: LOG_BASED
Open in Slack
PreviousNext
Powered by