I m using a forked version of the `pipelinewise tap mysql` I Meltano #troubleshooting

I’m using a forked version of the `pipelinewise-ta...

josh_lloyd

08/04/2023, 7:02 PM

I’m using a forked version of the

pipelinewise-tap-mysql

. Is anyone aware of anything in the

state

that could persist from run to the next in terms of the configuration of the connection to MySQL. I’ve been scratching my head for months on an error I’ve been getting in my pipelines:

pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '<insert-host-name-here>' (timed out)")

. It has cropped up completely unexpected from one day to the next with no explainable reason (we wouldn’t change any configuration with the pipeline). I could have 2 separate meltano instances (prod and stage). One would run successfully and the other would stop with this error. I just fixed it by running the pipeline with a

--full-refresh

flag. facepalm trek A stupid simple fix that I wish I would have tried months ago, but I don’t understand how that could have caused a connection issue like this. I’m just looking for a deeper understanding so I can submit bug reports with a recommended fix.

josh_lloyd

08/04/2023, 7:28 PM

Here’s a sample

state

when it has the timeout error:

Copy code

{
  "singer_state": {
    "bookmarks": {
      "bcg-abstractmetadatafield": {
        "version": 1681327114425,
        "log_file": "mysql-bin-changelog.001047",
        "log_pos": 79598783
      },
      "bcg-addresses": {
        "version": 1681327114500,
        "log_file": "mysql-bin-changelog.001047",
        "log_pos": 79598783
      },
...

And the same state after a full-refresh that works:

Copy code

{
  "singer_state": {
    "currently_syncing": null,
    "bookmarks": {
      "bcg-abstractmetadatafield": {
        "initial_full_table_complete": true
      },
      "bcg-addresses": {
        "initial_full_table_complete": true
      },
...

Open in Slack

Previous Next