Hi, Any ideas on what is the best method to update state.json of a tap for incremental load after the pipeline has completed ?

@nitin_kharya Have you considered using Meltano’s automatic state management? https://meltano.com/docs/integration.html#incremental-replication-state

Hi @douwe_maan, As per the documentation it looks like it should manage the state on its own and I do not require to set anything explicitly. But every time I run my pipeline it brings the data with complete import. meltano | No state was found, complete import. However when the load completes, I do get meltano | Incremental state has been updated at 2021-08-18 174048.930257.

@nitin_kharya Are you passing a

--job_id

into

meltano elt

?

no .... how do I get that ?

See https://meltano.com/docs/integration.html#incremental-replication-not-running-as-expected and the

--job_id

flag under https://meltano.com/docs/command-line-interface.html#elt

So it could be any String basically ?

Correct!

Thanks @douwe_maan