Does anyone have suggestions for how to best perform a one-off sync of historical data? I have a table that we sync with incremental update and uses a timestamp column for the replication key. The table has "new" records that are not synced to our destination because their timestamp value is before the replication key value.

Can you share your

meltano.yml

config? Assume you have a

metadata

section for your tap that defines this incremental behaviour.

@Ellis Valentiner can you edit / delete your state to trigger a full refresh?

Here is a redacted version of our meltano.yml with only the 1 table

Pretty sure you would be able to do something like

TAP_POSTGRES__MY_SCHEMA__METADATA___REPLICATION_METHOD=FULL_TABLE meltano run tap-postgres--my-schema ...

I'm fuzzy on the exact environment variable, but you would be able to check with

meltano config tap-postgres--my-schema list

Yes but I don't want to perform a full table replication. It would take days and most of the data is unchanged.

Ah right, sorry I misunderstood. I think you would have to do something like create a custom state entry and reference it with

--state-id-suffix

then? Or else invoke the tap directly passing

--state state.json

.

Interesting. What would the

state.json

actually look like in this case?

Probably very similar to your existing state, just with a tweak to the `replication_key_value`:

# meltano state list
meltano state get <state id> | jq .singer_state > state.json

I understand how to modify the

replication_key_value

to override which values meltano starts to replicate from but I don't understand how to stop it from replicating data past a certain end point. As far as I know meltano only supports replication that is entirely unbounded (full table) or left-bounded ranges (incremental from a starting point) but doesn't support any form of right-bounded ranges

or actually you could just edit your existing state, given that it will update at the end of the run anyway.