Hey, how do you handle

truncate

events from source tables in

postgres

(or other dbms’)? It’s possible to detect the event with

LOG_BASED

replication method. I’ve researched two taps from MeltanoLab and Transferwise: • Meltano variant detects truncate event and issues debug log message • Transferwise variant ignores truncate event and do nothing The first idea that comes up is to patch both tap/target to send and receive a special type message and process it accordingly, but it seems too hacky. Anyone has experience in the field?

I don't have prod experience with this, but I think a way to handle this would be to emit an activate version message from the tap when you get the truncate. There's some details to figure out there but seems right from afar

@visch cool, seems like a way to go, thanks! It looks like MeltanoLab/target-postgres supports activate_version, but tap-postgres don’t. Is there any plans around it?

I know there's an issue in the sdk for it, I'm not sure how hard it would be to just do in the tap

So, I’ve patched tap/target (Transferwise variant) to process

TRUNCATE

events with a simple approach: when the event received the target flushes all prior collected upserts to table and then updates all rows’

sdc_deleted_at

with timestamp

WHERE sdc_deleted_at IS NOT NULL

. It simply does the trick, but feels too hacky for original use case of

ACTIVATE_VERSION

. Digging deeper I’ve found how other tap/target variants do it: 1. Rename old table with postfix and create a new one 2. Add version column and soft/hard delete rows of previously active version I like the second one, it feels more straightforward to keep everything in one table.