Hi everyone. I'm new to meltano. I encountered the...
# troubleshooting
k
Hi everyone. I'm new to meltano. I encountered the following problem of synchronization by CDC based on bin logs (LOG_BASED). I use tap-mysql (variant pipelinewise) fork: https://github.com/edgarrmondragon/pipelinewise-tap-mysql since there was a problem described here https://discuss.meltano.com/t/22883458/hi-team-i-m-using-tap-mysql-to-read-from-mariadb-one-of-the- After successful synchronization and snapshot of the table, I wanted to start loading from the last bin log, but got an error.. Maybe there are any suggestions to solve this problem? My tap and target configs:
Copy code
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: git+<https://github.com/edgarrmondragon/pipelinewise-tap-mysql.git@patch-1>
    config:
      database: ***
      engine: mysql
      session_sqls:
      - SET @@session.time_zone='+0:00'
      - SET @@session.wait_timeout=86400
      - SET @@session.net_read_timeout=86400
      - SET @@session.innodb_lock_wait_timeout=3600
    select:
    - schema-table.*
    metadata:
      '*':
        replication-method: LOG_BASED

  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres
    config:
      batch_size_rows: 50000
      hard_delete: true
      load_method: upsert
      use_copy: true
      validate_records: true
      sanitize_null_text_characters: true
First sync and snapshot the table:
Copy code
2025-01-12T14:25:26.750093Z [info     ] 2025-01-12 20:25:26,749 | INFO     | target-postgres.bilimclass_schedules-schedules | Inserting with SQL: COPY "52315846_3aa6_4a1a_8f12_49d575809a0d" ("id", "edu_program_subject_uuid", "date", "is_sync", "teacher_id", "updated_at", "edu_year", "status", "subgroup_uuid", "s_max_score", "type", "uuid", "quarter_id", "time_end", "theme", "created_at", "themes_count", "group_id", "time_start", "sync_lesson_type", "school_id", "cabinet_id", "order", "subject_id", "_sdc_extracted_at", "_sdc_received_at", "_sdc_batched_at", "_sdc_deleted_at", "_sdc_sequence", "_sdc_table_version", "_sdc_sync_started_at") FROM STDIN cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:25:27.709440Z [info     ] 2025-01-12 20:25:27,708 | INFO     | target-postgres.bilimclass_schedules-schedules | Hard delete: True cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.378060Z [info     ] 2025-01-12 20:26:44,377 | INFO     | target-postgres      | Target 'target-postgres' completed reading 154110525 lines of input (1 schemas, 153956563 records, 0 batch manifests, 153959 state messages). cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.391474Z [info     ] 2025-01-12 20:26:44,391 | INFO     | target-postgres.bilimclass_schedules-schedules | Cleaning up bilimclass_schedules-schedules cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.392114Z [info     ] 2025-01-12 20:26:44,391 | INFO     | singer_sdk.metrics   | METRIC: {"type": "counter", "metric": "record_count", "value": 47783, "tags": {"stream": "bilimclass_schedules-schedules", "pid": 562963}} cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.392447Z [info     ] 2025-01-12 20:26:44,392 | INFO     | target-postgres      | Emitting completed target state {"currently_syncing": null, "bookmarks": {"bilimclass_schedules-schedules": {"version": 1736619512188, "log_file": "mysql-bin.000210", "log_pos": 271739990}}} cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.414169Z [info     ] Incremental state has been updated at 2025-01-12 14:26:44.414102+00:00.
2025-01-12T14:26:45.961649Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True
Start sync on the last bin log:
Copy code
2025-01-12T14:51:00.216668Z [info     ] 2025-01-12 20:51:00,216 | INFO     | target-postgres.bilimclass_schedules-schedules | Initializing target sink for stream 'bilimclass_schedules-schedules'... cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.417780Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418153Z [info     ]   File "/srv/meltano/my-meltano-project/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/singer_sdk/sinks/core.py", line 596, in _parse_timestamps_in_record cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418424Z [info     ]     date_val = time_fromisoformat(date_val) cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418601Z [info     ] ValueError: Invalid isoformat string: '8:00:00' cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418765Z [info     ]                                cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418914Z [info     ] During handling of the above exception, another exception occurred: cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.419081Z [info     ]                                cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.419228Z [info     ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2
e
ValueError: Invalid isoformat string: '80000'
seems to be the same problem, but in the binlog-handling code: https://github.com/edgarrmondragon/pipelinewise-tap-mysql/blob/e1c6f7ba256e11eb51696587d4d651644d6af7d3/tap_mysql/sync_strategies/binlog.py#L2[…]227
k
Ok, please could you tell me, will it be fixed as part of your pull request? I am using your fork to work with tap-mysql until your pull request has been merged into main
e
My original branch won't be merged because the source repo was archived, but I just created https://github.com/transferwise/pipelinewise/pull/1198, which incorporates the fix for binlog too
🙌 1
In the meantime, pointing to my fork:
pip_url: git+<https://github.com/edgarrmondragon/pipelinewise.git@fix-timedelta-string#subdirectory=singer-connectors/tap-mysql>
👍 1
k
Thx!