Hi guys, I am seeking help with a issue in tap-mysql. My goal is to replicate data based on GTID, but I'm facing an issue where the tap-mysql (transferwise variant) is failing to persist the full GTID set in the Meltano state. The Environment: Setup: MySQL Master-Replica. Meltano extracts from the Replica. Transaction Profile: The Replica processes replicated transactions from the Master AND has its own Replica-owned transactions. Database Config: Both instances have log_slave_updates = ON. Meltano Config: replication_method: LOG_BASED, use_gtid: true and engine: mysql. GTID Status: SELECT @@global.gtid_executed on the Replica correctly shows a full set containing both the Master UUID and the Replica UUID ranges(comma separated). The Issue: The tap successfully executes all transactions, but it is not stopping and the state is not persisting correctly. It appears to only store the last transaction's UUID/ID in the Meltano state, overwriting it instead of storing GTID set. Because the state is incomplete, the next sync attempts to re-read ranges it should have already cleared, leading to a loop. Expected Behavior: The state should store the full, comma-separated GTID set (Master UUID + Replica UUID) as seen in @@global.gtid_executed. Meltano Configuration Details: Meltano Version: meltano, version 2.17.0 Extractor: tap-mysql (variant: transferwise) Loader: target-postgres (variant: transferwise) MySQL Version: 8.0.41 Troubleshooting Done: Verified @@global.gtid_executed on the replica shows both UUIDs. Confirmed log_slave_updates is enabled. Confirmed use_gtid: true in meltano Does the transferwise variant support multi-source GTID strings, or is this a known limitation? What configuration changes might I be missing to enable support for GTID sets?

Any help here would be appreciated.

Seeing your

meltano.yml

would help

Sharing a redacted version of my meltano.yml:

plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    config:
      use_gtid: true
      replication_method: LOG_BASED
      engine: mysql
      filter_dbs: <redacted>
      port: 3306
    select:
      - <db>-*.*
    metadata:
      <db>-*:
        replication-method: LOG_BASED

  loaders:
  - name: target-postgres
    variant: transferwise
    config:
      dbname: <redacted>
      default_target_schema: <redacted>

Let me know if anything more specific is needed.

I saw you logged https://github.com/transferwise/pipelinewise/issues/1268, which is what I would've ultimately suggested. We have our own fork of that in https://github.com/Matatika/pipelinewise-tap-mysql. I think we'd be happy to review a PR to our fork, in case the Pipelinewise folks don't have the bandwidth.

I had a quick question regarding your fork of the PipelineWise tap-mysql repo. Can you help me understand how updates from the main PipelineWise repository are handled in the Matatika fork? Specifically: • Do you regularly sync or merge changes from the main PipelineWise repo? • Or are updates selectively incorporated based on relevance? • If selective, what criteria do you typically use? I just want to understand the process so I can align any potential fix or PR accordingly.

Rather selectively. @Reuben (Matatika) was the last to update it, so he might have a better understanding of the process.

It's not well-defined at the moment - I don't think we have ever merged from upstream since we forked the tap, but I'd imagine it would be done on a selective basis as Edgar said. I guess that would be more of an involved process since the upstream was archived and moved to the main PipelineWise repo.