#2117 bug: target table is being reset after a successful sync, then completes without logging an error Issue created by gkarugi ### Meltano Version 3.1.0 ### Python Version 3.10 ### Bug scope Other ### Operating System Linux Ubuntu 22 ### Description sync reset's table after completing successfully and it seems to want to start afresh but it completes without any error. It initially synced over 500k records before resetting and adding some 9k records before completing without logging any error. I suspect this is happening on the near end of the log attached where it says " Cleaning up db_schema-locations" ### Code version: 1 default_environment: dev project_id: 2951a9a9-6eb5-4526-8f96-b10d22c2be8f environments: - name: dev - name: staging - name: prod plugins: extractors: - name: tap-mysql variant: transferwise pip_url: pipelinewise-tap-mysql select: - schema-locations.* metadata: schema-locations: replication-method: LOG_BASED loaders: - name: target-clickhouse variant: shaped-ai pip_url: git+https://github.com/shaped-ai/target-clickhouse.git 2023-12-09T195701.285373Z [info ] 2023-12-09 225701,285 | INFO | target-clickhouse | Target sink for 'db_schema-locations' is full. Draining... cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195701.966531Z [info ] 2023-12-09 225701,965 | INFO | target-clickhouse.db_schema-locations | Inserting with SQL: INSERT INTO test_schema.locations cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195701.967530Z [info ] (duration, playback_session_id, platform_id, device_platform_version, internet_connection, device_manufacturer, user_id, playable_id, longitude, id, app_version, playable_type, address, event, country, content_category, device_model, created_at, updated_at, device_platform, sub_user_id, playback_time, type, ip, latitude) cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195701.968338Z [info ] VALUES (:duration, :playback_session_id, :platform_id, :device_platform_version, :internet_connection, :device_manufacturer, :user_id, :playable_id, :longitude, :id, :app_version, :playable_type, :address, :event, :country, :content_category, :device_model, :created_at, :updated_at, :device_platform, :sub_user_id, :playback_time, :type, :ip, :latitude) cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195703.005703Z [info ] time=2023-12-09 225703 name=singer level=INFO message=METRIC: {"type": "counter", "metric": "record_count", "value": 109503, "tags": {"database": "db_schema", "table": "locations"}} cmd_type=elb consumer=False name=tap-mysql producer=True stdio=stderr string_id=tap-mysql 2023-12-09T195703.006572Z [info ] time=2023-12-09 225703 name=singer level=INFO message=METRIC: {"type": "timer", "metric": "job_duration", "value": 324.30908393859863, "tags": {"job_type": "sync_table", "database": "db_schema", "table": "locations", "status": "succeeded"}} cmd_type=elb consumer=False name=tap-mysql producer=True stdio=stderr string_id=tap-mysql 2023-12-09T195706.041006Z [info ] 2023-12-09 225706,040 | INFO | target-clickhouse | Target 'target-clickhouse' completed reading 582665 lines of input (1 schemas, 582077 records, 0 batch manifests, 585 state messages). cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195706.597018Z [info ] 2023-12-09 225706,596 | INFO | target-clickhouse.db_schema-locations | Inserting with SQL: INSERT INTO test_schema.locations cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195706.597761Z [info ] (duration, playback_session_id, platform_id, device_platform_version, internet_connection, device_manufacturer, user_id, playable_id, longitude, id, app_version, playable_type, address, event, country, content_category, device_model, created_at, updated_at, device_platform, sub_user_id, playback_time, type, ip, latitude) cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195706.598541Z [info ] VALUES (:duration, :playback_session_id, :platform_id, :device_platform_version, :internet_connection, :device_manufacturer, :user_id, :playable_id, :longitude, :id, :app_version, :playable_type, :address, :event, :country, :content_category, :device_model, :created_at, :updated_at, :device_platform, :sub_user_id, :playback_time, :type, :ip, :latitude) cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195707.515184Z [info ] 2023-12-09 225707,514 | INFO | target-clickhouse.db_schema-locations | Cleaning up db_schema-locations cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195707.517276Z [info ] 2023-12-09 225707,515 | INFO | target-clickhouse | Emitting completed target state {"currently_syncing": null, "bookmarks": {"db_schema-locations": {"version": 1702151505338, "log_file": "mysql-bin.000008", "log_pos": 62472502, "gtid": "49569077-2635-11ee-8f72-6045bddb2dee:1-1589"}}} cmd_type=elb consumer=True name=target-clickhouse producer=False stdio=stderr string_id=target-clickhouse 2023-12-09T195707.732261Z [info ] Incremental state has been updated at 2023-12-09 195707.731831. 2023-12-09T195707.870317Z [info ] Block run completed. block_type=ExtractLoadBlocks err=None set_number=0 success=True meltano/sdk