Hi all, I am new here. I just wanted to post an issue that I had with Meltano that I still cannot figure it out. It is about the add_record_metadata configuration. Basically when I set add_record_metadata=True, I do see all the following columns showing in this link (https://sdk.meltano.com/en/latest/implementation/record_metadata.html) appear. However when I set it to False, I still see the column, __sdc__table_version in my output which is a mssql table. I started to think whether this is a bug. I Just want to ask how I can remove this column from my output. Here is my meltano.yml: version: 1 default_environment: dev project_id: 99290a45-4a7e-42ae-8208-0b2ca725d652 environments: - name: dev config: plugins: extractors: - name: tap-mssql config: database: LiftStaging host: localhost port: '1434' user: sa select: - crm-contact_extract.id - crm-contact_extract.country_code loaders: - name: target-mssql config: database: LiftStaging default_target_schema: crm host: localhost port: '1434' username: sa add_record_metadata: false - name: staging - name: prod plugins: extractors: - name: tap-mssql variant: wintersrd pip_url: tap-mssql loaders: - name: target-mssql variant: meltanolabs pip_url: git+https://github.com/MeltanoLabs/target-mssql.git mappers: - name: meltano-map-transformer variant: meltano pip_url: git+https://github.com/MeltanoLabs/meltano-map-transform.git mappings: - name: table-map config: stream_maps: crm-contact_extract: __alias__: contact_loaded

Hi @Jeff Chen! Can you run

meltano invoke tap-mssql --discover > catalog.json

? We're looking for the

_sdc_table_version

in

catalog.json

.

Hello @Edgar Ramírez (Arch.dev). We were trying this and got the following result:

{

"streams": [

{

"tap_stream_id": "crm-contact_extract",

"table_name": "contact_extract",

"schema": {

"properties": {

"country_code": {

"inclusion": "available",

"maxLength": 2,

"type": [

"null",

"string"

]

},

"id": {

"inclusion": "available",

"minimum": -2147483648,

"maximum": 2147483647,

"type": [

"null",

"integer"

]

}

},

"type": "object"

},

"stream": "contact_extract",

"metadata": [

{

"breadcrumb": [],

"metadata": {

"selected-by-default": false,

"database-name": "crm",

"is-view": false,

"table-key-properties": []

}

},

{

"breadcrumb": [

"properties",

"country_code"

],

"metadata": {

"selected-by-default": true,

"sql-datatype": "varchar"

}

},

{

"breadcrumb": [

"properties",

"id"

],

"metadata": {

"selected-by-default": true,

"sql-datatype": "int"

}

}

]

}

]

}

This is the catalog.json. There is no

_sdc_table_version

so it seems like this is added by the

target-mssql

Ok, I see what's happening. The column is added here: https://github.com/meltano/sdk/blob/5ed3be170bfd1f35b72efc40c493cd000bd00ed2/singer_sdk/sinks/sql.py#L379-L387 That is, regardless of the

add_record_metadata

setting. The rest of the "activate version" logic doesn't make sense without that column in there, so I'm not sure how we should proceed. Perhaps a new setting meant to ignore

ACTIVATE_VERSION

messages entirely? 🤔 Definitely feel free to create an issue or even send a pull request.

@Edgar Ramírez (Arch.dev) Indeed. We tried meltano invoke tap-mssql > output.json and the output looks like this:

{"type":"STATE","value":{"currently_syncing":"crm-contact"}}
{"type":"SCHEMA","stream":"crm-contact","schema":{"properties":{"country_code":{"inclusion":"available","maxLength":2,"type":["null","string"]},"id":{"inclusion":"available","minimum":-2147483648,"maximum":2147483647,"type":["null","integer"]}},"type":"object"},"key_properties":[]}
{"type":"ACTIVATE_VERSION","stream":"crm-contact","version":1732695221437}
{"type":"RECORD","stream":"crm-contact","record":{"country_code":"US","id":1},"version":1732695221437,"time_extracted":"2024-11-27T08:13:41.445465Z"}
{"type":"RECORD","stream":"crm-contact","record":{"country_code":"US","id":2},"version":1732695221437,"time_extracted":"2024-11-27T08:13:41.445465Z"}
{"type":"RECORD","stream":"crm-contact","record":{"country_code":"US","id":3},"version":1732695221437,"time_extracted":"2024-11-27T08:13:41.445465Z"}
{"type":"RECORD","stream":"crm-contact","record":{"country_code":"US","id":4},"version":1732695221437,"time_extracted":"2024-11-27T08:13:41.445465Z"}
{"type":"RECORD","stream":"crm-contact","record":{"country_code":"US","id":5},"version":1732695221437,"time_extracted":"2024-11-27T08:13:41.445465Z"}
{"type":"STATE","value":{"currently_syncing":"crm-contact"}}
{"type":"ACTIVATE_VERSION","stream":"crm-contact","version":1732695221437}
{"type":"STATE","value":{"currently_syncing":"crm-contact","bookmarks":{"crm-contact":{"initial_full_table_complete":true}}}}
{"type":"STATE","value":{"currently_syncing":null,"bookmarks":{"crm-contact":{"initial_full_table_complete":true}}}}

So even if add_record_metadata is set to false, _sdc_table_version will still appear in the table and only its value becomes NULL because of "type":"ACTIVATE_VERSION",...

I've commented on https://github.com/meltano/sdk/issues/2776 with a potential approach for addressing this.

Thank you for your quick reply @Edgar Ramírez (Arch.dev)