Hello everyone! I have more and more problems with Meltano. I start ELT one table using CDC based on the bin log. The first snapshot of the table is taken correctly with the TZ that I specify in the meltano.yaml

- SET @@session.time_zone='+05:00',

but when I start the increment, a warning is displayed in the logs.

2025-02-15T15:32:05.554365Z [info     ] /home/kuanysh/work/meltano/my-meltano-project/.meltano/extractors/tap-mysql/venv/lib/python3.11/site-packages/tap_mysql/sync_strategies/binlog.py:215: PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see <https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html> cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-postgres name=tap-mysql producer=True run_id=beae157b-085a-49a1-a907-1ca522ef365d stdio=stderr string_id=tap-mysql
2025-02-15T15:32:05.554648Z [info     ]   local_datetime = timezone.localize(val) cmd_type=elb consumer=False job_name=dev:tap-mysql-to-target-postgres name=tap-mysql producer=True run_id=beae157b-085a-49a1-a907-1ca522ef365d stdio=stderr string_id=tap-mysql

Next, I look in the target and see that the

created_at

and

updated_at

columns take TZ UTC. This is clearly visible on the screenshot, where

the first 4

lines with TZ UTC (increment) and everything below with TZ +5 (Full table), in the configuration file I clearly indicate the creation of a session with a source of +5. I also assume that this problem can cause

gaps

during increment in the target, where 51 lines out of 200 million lines were skipped. https://meltano.slack.com/archives/C069CQNHDNF/p1739338978501569 Source: MySQL v. 5.7.44 TZ + 5 Target: PostgreSQL v. 13 TZ UTC Using tap-mysql pipelinewise variant forked by @Edgar Ramírez (Arch.dev) https://github.com/edgarrmondragon/pipelinewise.git@fix-timedelta-string#subdirectory=singer-connectors/tap-mysql Before that, I used the fork https://github.com/edgarrmondragon/pipelinewise-tap-mysql and hardcoded the string in

date_val = '0' + date_val if len(date_val) == 7 else date_val

in \my-meltano-project\.meltano\loaders\target-postgres\venv\Lib\site-packages\singer_sdk\sinks\core.py and there was no problem with TZ in the target

Related tasks: https://meltano.slack.com/archives/C069CQNHDNF/p1739338978501569 https://meltano.slack.com/archives/C069CQNHDNF/p1736699656325379 My meltano.yaml file:

version: 1
default_environment: dev
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-mysql
    variant: transferwise
    pip_url: git+<https://github.com/edgarrmondragon/pipelinewise.git@fix-timedelta-string#subdirectory=singer-connectors/tap-mysql>
    config:
      database: database      host: host
      port: 3306
      user: user
      session_sqls:
      - SET @@session.max_execution_time=0     # No limit
      - SET @@session.net_read_timeout=604800
      - SET @@session.net_write_timeout=604800
      - SET @@session.time_zone='+05:00'
      - SET @@session.wait_timeout=604800
      - SET @@session.innodb_lock_wait_timeout=3600
    select:
    - schema-table.*
    metadata:
      '*':
        replication-method: LOG_BASED
  loaders:
  - name: target-postgres
    variant: meltanolabs
    pip_url: meltanolabs-target-postgres
    config:
      batch_size_rows: 50000
      hard_delete: true
      load_method: upsert
      sanitize_null_text_characters: true
      use_copy: true
      port: 5432

datetime values do seem to be coerced to UTC: https://github.com/transferwise/pipelinewise/blob/3d8e7bc6214a6876ec3871ca4b4aca6b[…]singer-connectors/tap-mysql/tap_mysql/sync_strategies/binlog.py