Hi everyone, Im starting to use meltano and Im developing a custom extractor. I'm trying to figure out how to run a new pipeline with a new version of my custom extractor but targeting the same target already configured (bigquery). How should I make it so that records that already exist in the destination table are not duplicated? Is it necessary to do a full-refresh? Or can I copy the state of the old version of the extractor to the new version?

I think those are actually two different things: 1. Not duplicating records in the target: That is usually taken care of by ensuring that primary keys are designed correctly. Singer spec (and therefore meltano taps and targets) allow you to set up your primary keys for each stream. 2. Copying meltano state (and thereby eliminating unnecessary source extracts): Yes, that will usually work fine. You can carefully copy state by hand using meltano's

meltano get state

and

meltano set state

commands.