Hello everyone, I have more questions! Does anyone know if it's possible to send a message to meltano to stop a job gracefully? I'd love to create a backfilling job which runs for 4h a day every day, but that could come to completion even if it had not loaded the entirety of the data. Is this possible?

Not sure about doing this natively, but if you're running meltano via an orchestrator then a simple job timeout would do the trick, this is definitely possible in dagster. Whether that qualifies as a 'graceful' stop, I'm not sure. 💥

Hey @Andy Carter, thanks for the response 😄 I'm running in Airflow currently, so I'm fairly confident that would work there as well, just trying to assess to the feasibility of "gracefully" 💃 😅

Yeah I have seen the 'meltano run already in progress' error message when dagster effects a timeout on a job and then retries.

Same, which is usually a great thing, but now would be a blocker for me if my objective was "please slowly backfill this data without affecting other jobs over the course of a week"

It's called a

SIGTERM

which is a signal you send to the process, if you send it to the

meltano

process it should (I think it does last I checked) propagate the

SIGTERM

to the sub processes that are currently running, and then it's up to the

tap

/

target

to properly clean up and die

What would I need to implement for that? In my tap and target to manage sigterm

Have you tried just using the signal to see if it works?

@Stéphane Burwash I’m really curious what ended up happening here - were you able to get this to die gracefully on

SIGTERM

? My concern is similar to yours - I have long running processes and want to make sure I have a way to kill a process and have it write out state before it shuts down so the next one that comes up knows where to pick up

No conclusion yet, sorry @avy_faingezicht - currently I'm looking at client-side rate limiting, so that I can ensure my long process doesn't get cancelled by a parallel one, but thats not really a solution to your current issue

Makes sense! No worries, keep us posted