I should know this already, but... If a loader fails in an ELT job, what happens to the data and the state? I reviewed the Meltano ELT reference architecture and it does not say, and I have not looked at the source. Is the main Meltano process actually sitting in the middle and watching the messages from extractor stdout and sending them to loader stdin, and watching stdout of the target for that state to know that the target has processed, and stored, records up to that state? I've had that assumption all along, but didn't really know.

Hi Fred, I know the feeling. It's one of those things that you really want to know is solid and understand, but I've also often had the feeling I didn't really understand what would happen in exceptional situations. Taking the code is truth approach - these are the relevant places I think: https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/target.py#L21 https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/runner/singer.py#L90 As each STATE message flows through, meltano (or optionally delegated to the target it seems) writes the state. Would be great to have one of the <!subteam^S02BCD9FFEF> guys jump in and correct me, or even some links to detailed design docs!

Hey 👋 In having a look through the docs I agree this isn't well documented. Here is what is documented - as is expected by the Singer Spec, we only capture state emitted by the Target (i.e. we do not yet try to intercept messages between Tap and Target @fred_reimer). According to the spec, Targets should only emit State messages once records up to the bookmark have been persisted. As the note in the docs suggests however, some Targets will only emit a STATE message once all records have been persisted, which can be problematic if interrupted. Anything built with the SDK will use periodic bookmarking by default, so that mostly applies to pre-SDK Targets. In addition to storing Target state, Meltano does make an effort to pick up where a failed Target left off. It does this by updating the last completed state with any subsequent incomplete state (i.e. merging them). The code for this is actually in the SingerTap plugin definition, here, which handles state preparation before launching the Tap. Hope this helps 🙂 I have created an issue in our doc cleanup epic to note that state handling needs attention. We have docs improvements on our roadmap for December, so this should improve soon 🚀

Thanks for confirming!

Thanks @ken_payne - really good answer!

We have docs improvements on our roadmap for December, so this should improve soon

Just to add some clarity to this - we’re working on updating the style and location of the docs as well as continuing to improve the overall documentation. I’ve scheduled this specific issue but we’ll continually update the documentation as best we can (and we’d love MR’s to make it better too 😄 )