Hey folks, using

tap-postgres

to

target-snowflake

and without a PK or UK to use, is it possible to truncate the data before inserting so there are no duplicates, or am I out of luck?

@chad_gray I think you have a few options here. If the target you’re using doesn’t support that behavior, you should be able to add some metadata fields that indicate which load each row came in from (this is available too for connectors made with the SDK). Then you can use something like dbt downstream to find the unique rows for the latest load.

doesn’t quite solve the problem of all the rows duplicating every time the job is run. i know storage is cheap but it’s annoying to have to go through more and more rows downstream. guess the best solution is fix the source table so it has a PK!

Yeah, it’s not ideal. You’re correct that the ultimate solution is to fix the source upstream 😅

@chad_gray - Just to confirm/clarify, is there a logical key you can use to dedupe while inserting? Even if it is derrived from other columns? Or does the data just not have any applicable uniqueness for deduping?