Deduplication: Hi, I would like to ask how to handle deduplication with Meltano? I could not find any hint about what would be the best approach to apply deduplication on a data import. In my work deduplication is a typical "always on" standard task of building data import pipelines, so I hoped for some hints here. I am not sure which element of the Meltano chain would be best to implement data deduplication tasks on import - the singer or the dbt stage?
Also of course it is not only about silently skipping duplicate indexes, like e.g. doing "upsert" style sql imports, but also comprehensive monitoring and reporting of the dedup process is important - you definitely want to understand if your duplicates are expected or if something is wrong. Also some optional handling of duplicates like e.g. count them, save to another data space, produce issues in tracker systems (only ONE for same error) etc. should be part of the game.
I have the strong impression that Meltano could give an tremendous boost to these kind of import / dedup jobs, but unfortunately I had no search results with "dedup" or similar words in the current docs.
Is watching for dupes not an absolutely basic part of your import pipelines? Adorable, enviable data sources you must have 🙂.