Hihi, I'm just getting started and have a very simple question. I have a CSV tap and a jsonl loader just to test, but it's not working like I'd expect. I've configured the CSV tap to identify the key/id column, but each time I run the pipeline it re-adds every row to the output creating duplicates. I thought that it would store which keys had been extracted in a state file and not load them again; I also thought that doing a --full-refresh would clear the jsonl output and re-load all the keys, but again it just appends them. Am I missing something? How do I get non-duplicating extract-loads?

Hi, @will_gittoes! And welcome! 1. Regarding duplicates. The Singer Spec promises that all records will land at least once, but not necessarily exactly once. For streams with a primary key, most database targets will dedupe by using a merge upsert operation. (I can provide more information on this and additional mitigations, but the biggest win would be from using a DB target instead of the JSONL file based one.) 2. Regarding clearing the old results after completing a full table sync, the feature that delivers that capability is called "activate version" - meaning activation of the new table "version" by removing records from prior syncs. Almost all common database targets have this feature but the JSONL target does not (to my knowledge at least). I hope this info is helpful. Do you have a specific destination database in mind?

Hi AJ, thanks so much for replying! My actual destinations will be a mix of various data warehouses (redshift, snowflake) which should be fine, and google sheets which is more problematic, since that also doesn't seem to de-dupe.

Can a CSV tap store state?? I guess I can imagine replication-mode: INCREMENTAL but surely not a distinct list of keys.