Meltano

Question - how well does Meltano handle SIGTERM/SIGKILL `docker stop` signal sequence? :thread:

I’m mulling over the idea of putting all my hourly incremements that take &lt; 60seconds to run into Fargate Spot, to save on costs.

When Fargate Spot gives the container warning that it’s about to be disrupted, it sends a SIGTERM to the running process.

What would be amazing is if meltano would handle this gracefully, ensuring that everything is cleaned up, and ideally exit with a different status code so that i can immediately reschedule the container

in fact i could run all “non critical” hourly incemements like this

Not super crazy actually, asyncio and the signal handling in Meltano "should" be able to take care of this and atleast send the signal to your down stream processes. Now do taps/targets do anything with those signals? Probably not

This would definitely be worth filing an issue in the SDK for too. I don’t know what the defaults are around handling sigkill for taps built with the sdk cc <@U06CBKL5W7L>

Meltano should handle this fine, if it doesn't, please file an issue :slightly_smiling_face:

In terms of the <#C01PKLU5D1R|>, we don’t have any special handling for SIGTERM at this point. I’ve logged an <https://gitlab.com/meltano/singer-sdk/-/issues/130|issue here> which explores what actions might be taken to gracefully respond to a shutdown message. Comments and MR’s welcome!

I have tested this feature pretty thoroughly and Meltano handles it correctly

What does it do exactly Charles?  Does it do a quick load and then save state and then gracefull commit suicide?

IIRC it interrupts the tap and target, saves the job as "failed" but including the latest state so you could pick back up where you left off, and then returns an error status code

that's for SIGTERM, for some of the other signals it isn't as graceful I think