I am having issues with tap-mssql variant SpaceCondor using log based replication (Change Tracking not CDC): https://hub.meltano.com/extractors/tap-mssql--spacecondor/ I have Change Tracking configured in SQL Server but am getting a full replication on each run when using meltano run (switched to target-jsonl from target-snowflake for debugging):

meltano run tap-mssql target-jsonl

The state gets updated with each run:

{"bookmarks": {"dbo-c_logical_field_user_values": {}}}

Extractor config:

extractors:

- name: tap-mssql

config:

host: PROJECT01

port: 60065

database: rss_test

username: svcTestAccount

default_replication_method: LOG_BASED

sqlalchemy_url_query_options:

- key: driver

value: ODBC Driver 18 for SQL Server

- key: TrustServerCertificate

value: yes

select:

- dbo-c_logical_field_user_values.*

I think this might be a configuration issue, not sure but perhaps isn't picking up the default replication method?

{"event": "Visiting CatalogNode.STREAM at '.streams[352]'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.910101Z"}

{"event": "Setting '.streams[352].selected' to 'False'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910162Z"}

{"event": "Setting '.streams[352].selected' to 'True'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910211Z"}

{"event": "Skipping node at '.streams[352].tap_stream_id'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910259Z"}

{"event": "Skipping node at '.streams[352].table_name'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910306Z"}

{"event": "Skipping node at '.streams[352].replication_method'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910354Z"}

{"event": "Skipping node at '.streams[352].key_properties[0]'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910402Z"}

{"event": "Visiting CatalogNode.PROPERTY at '.streams[352].schema.properties.logical_field_sid'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.910457Z"}

{"event": "Visiting CatalogNode.PROPERTY at '.streams[352].schema.properties.enabled_flag'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.910513Z"}

{"event": "Skipping node at '.streams[352].schema.properties.enabled_flag.maxLength'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910604Z"}

{"event": "Visiting CatalogNode.PROPERTY at '.streams[352].schema.properties.modified_by_user_sid'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.910779Z"}

{"event": "Visiting CatalogNode.PROPERTY at '.streams[352].schema.properties.modified_datetime'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.910924Z"}

{"event": "Skipping node at '.streams[352].schema.properties.modified_datetime.format'", "level": "debug", "timestamp": "2025-04-22T00:34:06.910988Z"}

{"event": "Visiting CatalogNode.PROPERTY at '.streams[352].schema.properties.timestamp'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.911099Z"}

{"event": "Visiting CatalogNode.PROPERTY at '.streams[352].schema.properties.system_modified_datetime'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.911160Z"}

{"event": "Skipping node at '.streams[352].schema.properties.system_modified_datetime.format'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911212Z"}

{"event": "Skipping node at '.streams[352].schema.type'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911262Z"}

{"event": "Skipping node at '.streams[352].schema.required[0]'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911312Z"}

{"event": "Skipping node at '.streams[352].schema.$schema'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911361Z"}

{"event": "Skipping node at '.streams[352].is_view'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911410Z"}

{"event": "Skipping node at '.streams[352].stream'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911458Z"}

{"event": "Visiting CatalogNode.METADATA at '.streams[352].metadata[0]'.", "level": "debug", "timestamp": "2025-04-22T00:34:06.911509Z"}

{"event": "Visiting metadata node for tap_stream_id 'dbo-c_logical_field_user_values', breadcrumb '['properties', 'logical_field_sid']'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911558Z"}

{"event": "Setting '.streams[352].metadata[0].metadata.selected' to 'False'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911616Z"}

{"event": "Setting '.streams[352].metadata[0].metadata.selected' to 'True'", "level": "debug", "timestamp": "2025-04-22T00:34:06.911665Z"}

Anyone have suggestions on troubleshooting? No errors like the previous question about not finding the state. SQL Server tables have Change Tracking enabled in SQL Server as:

ALTER TABLE dbo.' + @ls_table_name + N'

ENABLE CHANGE_TRACKING

WITH (TRACK_COLUMNS_UPDATED = OFF);

I don't think the tap does actually do anything with the

default_replication_method

value, so you'll need to define the replication method as a

metadata

override: https://docs.meltano.com/concepts/plugins/#metadata-extra

Thanks @Edgar Ramírez (Arch.dev), that was the issue. I added the following metadata section to the config for anyone looking at this issue in the future:

extractors:

- name: tap-mssql

config:

host: PROJECT01

port: 60065

database: rss_test

username: svcTestAccount

sqlalchemy_url_query_options:

- key: driver

value: ODBC Driver 18 for SQL Server

- key: TrustServerCertificate

value: yes

select:

- dbo-c_logical_field_user_values.*

metadata:

dbo-*:

replication_method: LOG_BASED

This tap doesn't appear to be able to detect the primary key, so you will need to specify the metadata for each table in order to add the key. Is that "normal" behavior for SQL taps that support log based replication?

2025-04-22T17:26:19.153818Z [info     ] ValueError: Could not detect replication key for 'dbo-c_logical_field_user_values' stream(replication method=LOG_BASED) cmd_type=elb consumer=False job_name=dev:tap-mssql-to-target-jsonl name=tap-mssql producer=True run_id=af88abc1-c2ee-46e4-a181-6b8bb21292dd stdio=stderr string_id=tap-mssql

Can you try

metadata:
  dbo-*:
    replication_method: LOG_BASED
    replication_key: _sdc_change_version

That generates this error:

singer_sdk.exceptions.InvalidReplicationKeyException: Field '_sdc_change_version' is not in schema for stream 'dbo-c_logical_field_user_values'

This configuration looks like it works correctly:

metadata:

dbo-c_logical_field_user_values:

replication_method: LOG_BASED

replication-key: logical_field_sid

logical_field_sid:

is-replication_key: true

The tap is generating something like this extracted from the logs (I wasn't running a trace) when I didn't have VIEW CHANGE TRACKING permissions set for the SQL account. It is using the replication key to join on the change tracking tables. SELECT c.SYS_CHANGE_VERSION AS _sdc_change_version, c.SYS_CHANGE_OPERATION AS _sdc_change_operation, c.logical_field_sid, tb.enabled_flag, tb.modified_by_user_sid, tb.modified_datetime, tb.timestamp, tb.system_modified_datetime FROM CHANGETABLE CHANGES dbo.c_logical_field_user_values,", "level": "info", "timestamp": "2025-04-22T210121.749032Z"} 1", "level": "info", "timestamp": "2025-04-22T210121.749182Z"} ) AS c", "level": "info", "timestamp": "2025-04-22T210121.749341Z"} LEFT JOIN", "level": "info", "timestamp": "2025-04-22T210121.749495Z"} dbo.c_logical_field_user_values AS tb", "level": "info", "timestamp": "2025-04-22T210121.749649Z"} ON", "level": "info", "timestamp": "2025-04-22T210121.749802Z"} tb.logical_field_sid = c.logical_field_sid", "level": "info", "timestamp": "2025-04-22T210121.749965Z"} ORDER BY", "level": "info", "timestamp": "2025-04-22T210121.750121Z"} c.SYS_CHANGE_VERSION ASC", "level": "info", "timestamp": "2025-04-22T210121.750245Z"}