Kuanysh Zhaksylyk
01/12/2025, 4:34 PMplugins:
extractors:
- name: tap-mysql
variant: transferwise
pip_url: git+<https://github.com/edgarrmondragon/pipelinewise-tap-mysql.git@patch-1>
config:
database: ***
engine: mysql
session_sqls:
- SET @@session.time_zone='+0:00'
- SET @@session.wait_timeout=86400
- SET @@session.net_read_timeout=86400
- SET @@session.innodb_lock_wait_timeout=3600
select:
- schema-table.*
metadata:
'*':
replication-method: LOG_BASED
loaders:
- name: target-postgres
variant: meltanolabs
pip_url: meltanolabs-target-postgres
config:
batch_size_rows: 50000
hard_delete: true
load_method: upsert
use_copy: true
validate_records: true
sanitize_null_text_characters: true
First sync and snapshot the table:
2025-01-12T14:25:26.750093Z [info ] 2025-01-12 20:25:26,749 | INFO | target-postgres.bilimclass_schedules-schedules | Inserting with SQL: COPY "52315846_3aa6_4a1a_8f12_49d575809a0d" ("id", "edu_program_subject_uuid", "date", "is_sync", "teacher_id", "updated_at", "edu_year", "status", "subgroup_uuid", "s_max_score", "type", "uuid", "quarter_id", "time_end", "theme", "created_at", "themes_count", "group_id", "time_start", "sync_lesson_type", "school_id", "cabinet_id", "order", "subject_id", "_sdc_extracted_at", "_sdc_received_at", "_sdc_batched_at", "_sdc_deleted_at", "_sdc_sequence", "_sdc_table_version", "_sdc_sync_started_at") FROM STDIN cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:25:27.709440Z [info ] 2025-01-12 20:25:27,708 | INFO | target-postgres.bilimclass_schedules-schedules | Hard delete: True cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.378060Z [info ] 2025-01-12 20:26:44,377 | INFO | target-postgres | Target 'target-postgres' completed reading 154110525 lines of input (1 schemas, 153956563 records, 0 batch manifests, 153959 state messages). cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.391474Z [info ] 2025-01-12 20:26:44,391 | INFO | target-postgres.bilimclass_schedules-schedules | Cleaning up bilimclass_schedules-schedules cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.392114Z [info ] 2025-01-12 20:26:44,391 | INFO | singer_sdk.metrics | METRIC: {"type": "counter", "metric": "record_count", "value": 47783, "tags": {"stream": "bilimclass_schedules-schedules", "pid": 562963}} cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.392447Z [info ] 2025-01-12 20:26:44,392 | INFO | target-postgres | Emitting completed target state {"currently_syncing": null, "bookmarks": {"bilimclass_schedules-schedules": {"version": 1736619512188, "log_file": "mysql-bin.000210", "log_pos": 271739990}}} cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=5421d841-b2ca-4ccf-9890-84ed658d0cda stdio=stderr string_id=target-postgres
2025-01-12T14:26:44.414169Z [info ] Incremental state has been updated at 2025-01-12 14:26:44.414102+00:00.
2025-01-12T14:26:45.961649Z [info ] Block run completed. block_type=ExtractLoadBlocks err=None set_number=0 success=True
Start sync on the last bin log:
2025-01-12T14:51:00.216668Z [info ] 2025-01-12 20:51:00,216 | INFO | target-postgres.bilimclass_schedules-schedules | Initializing target sink for stream 'bilimclass_schedules-schedules'... cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.417780Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418153Z [info ] File "/srv/meltano/my-meltano-project/.meltano/loaders/target-postgres/venv/lib/python3.10/site-packages/singer_sdk/sinks/core.py", line 596, in _parse_timestamps_in_record cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418424Z [info ] date_val = time_fromisoformat(date_val) cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418601Z [info ] ValueError: Invalid isoformat string: '8:00:00' cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418765Z [info ] cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.418914Z [info ] During handling of the above exception, another exception occurred: cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.419081Z [info ] cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
2025-01-12T14:51:00.419228Z [info ] Traceback (most recent call last): cmd_type=elb consumer=True job_name=dev:tap-mysql-to-target-postgres name=target-postgres producer=False run_id=82af6905-6feb-4af3-aa58-b1f92c33564e stdio=stderr string_id=target-postgres
Edgar Ramírez (Arch.dev)
01/13/2025, 5:12 PMValueError: Invalid isoformat string: '80000'seems to be the same problem, but in the binlog-handling code: https://github.com/edgarrmondragon/pipelinewise-tap-mysql/blob/e1c6f7ba256e11eb51696587d4d651644d6af7d3/tap_mysql/sync_strategies/binlog.py#L2[…]227
Kuanysh Zhaksylyk
01/14/2025, 7:21 AMEdgar Ramírez (Arch.dev)
01/14/2025, 6:35 PMEdgar Ramírez (Arch.dev)
01/14/2025, 7:06 PMpip_url: git+<https://github.com/edgarrmondragon/pipelinewise.git@fix-timedelta-string#subdirectory=singer-connectors/tap-mysql>
Kuanysh Zhaksylyk
01/14/2025, 8:31 PM