I am new to Singer & Meltano so my questions might be basic but it seems that Singer has "state file" which persists across runs. In Meltano however, is the state file maintained under each run? If so, that seems to defeat the purpose of having the state/bookmarks since Meltano always starts anew?

@timothy_suh Have you seen https://meltano.com/docs/integration.html#pipeline-state? The state file is reused between

meltano elt

runs with the same

job_id

.

Thank you. I was already looking at the docs.

When you run

meltano elt

with

--state

and

--job_id

, that state will be passed to the extractor, which will then output new state messages based on the modified state. Meltano then stores the final state message output by the extractor and loader with the job ID, so the next time you run

meltano elt

with

--job_id

but without

state

, it will use the (updated version of) the state file you passed with

--state

on the previous run.

Muy bien, PERFECT. Let me try.

👍

Our first ELT of a big table keeps failing (maybe due to connectivity issue). I think there will be no STATE file unless the job_id=JOB_NAME successfully runs at least once. Is that true?

@timothy_suh We store state files for failed and successful runs, and take both into account on subsequent runs, but it's possible that in your case, the target isn't forwarding the tap's state if the tap fails. See the "Not seeing state picked up after a failed run?" note on https://meltano.com/docs/integration.html#pipeline-state