https://linen.dev logo
Join Slack
Powered by
Hi all. I'm using `tap-mssql` with 'replication-me...
# plugins-general
p
Hi all. I'm using 
tap-mssql
with 'replication-method' set to 'LOG_BASED' using CDC, and I have 
target-snowflake
with the 'load_method' set to 'upsert'. Until now I thought that this would mean that data deleted from the source database would also be deleted in the replicated target database, but I just discovered that this only marks the 
_SDC_OPERATION_TYPE
column as 'D' for deleted I presume (and it adds a 
_SDC_DELETED_AT
value). Is it possible to have a configuration where it will actually delete the rows in the target database instead of just mark them as deleted?
Well I found the hard_delete setting for snowflake and I'm guessing this is what I want. Does anyone know if this will work retro actively or just on any newly deleted records?
When I enabled hard delete, it seems to have wiped my whole table even though there were only a few items that had been marked as deleted previosly. Looking at the snowflake Query history it seems it ran the following and I'm not sure why. It seems to only be keeping changes since the previous run and deleting the rest. How can I get it to just delete the records that are actually deleted in the soruce database? Any suggestions anyone? 
DELETE FROM "MELTANO_ETL_DEV"."MY_SCHEMA"."MY_TABLE" WHERE _sdc_table_version < 1747775431902
c
So I believe there is an existing bug logged for this, basically the tap shouldn't be issuing activate version messages in LOG_BASED mode but is. The tap will track deleted using the _SDC_LSN_DELETED_AT field and if using soft-delete in snowflake checking that field should allow you to tell if something is deleted. The issue can be found at https://github.com/wintersrd/pipelinewise-tap-mssql/issues/71 If you were able to test making the change noted there in recent comments that would be helpful. I'm only using in soft delete mode so it's not an issue for me.
👍 1
6 Views
Open in Slack
PreviousNext
Powered by