Hello, Can someone helps in understanding the working of state.json.

The tap is run consecutively, the data is run from start_date until today.

If I run tap again - it runs for the last date in the state file, causing duplicate data.

Context - We are trying to run meltano as sync jobs using

tap-google-search-console

&

target-athena

Hi @hitesh! Using Meltano you shouldn’t need to worry about

state.json

, since it manages that for you, storing and retrieving it from a state backend. If the tap has the state capability, then Meltano will automatically pass the state file to it, and save the updated state at the end of the pipeline.

@hitesh I'd also recommend checking out https://sdk.meltano.com/en/latest/implementation/at_least_once.html related to singer and duplicate records

Thanks @pat_nadolny @edgar_ramirez_mondragon These docs really helped.