Hello,
I want to simply replicate data between SQL Server and Snowflake using replication method = 'LOG_BASED'. I use tap-mssql (variant=wintersrd) and target-snowflake (variant=meltanolabs) for this pipeline. The pipeline can be executed and runs without errors. But I am a little bit surprised about the structure of my target table in Snowflake. It contains a lot of _SDC* columns.
_SDC_BATCHED_AT
_SDC_DELETED_AT
_SDC_EXTRACTED_AT
_SDC_LSN_COMMIT_TIMESTAMP
_SDC_LSN_DELETED_AT
_SDC_LSN_OPERATION
_SDC_LSN_SEQ_VALUE
_SDC_LSN_VALUE
_SDC_OPERATION_TYPE
_SDC_RECEIVED_AT
_SDC_SEQUENCE
_SDC_TABLE_VERSION
I expect that this is metadata which was added to the stream by tap-mssql. It's possible to merge the changes manually into another table to get a replication of the source table in Snowflake. But I would expect that Meltano can do this for me without my intervention. How can I achieve this when using LOG_BASED replication?
Kind regards