<@U06CEG44Y04> I saw your post over on dagster re ...
# infra-deployment
a
@Matt Menzenski I saw your post over on dagster re dagster-meltano and the deprecation of
dagster-shell
and to use
PipesSubprocessClient
instead. https://dagster.slack.com/archives/C05RDKSEFFT/p1740419325050919 Do you think that would be a drop-in replacement for
execute_shell_command
? In-built graceful termination would be great 🙂 https://dagster.slack.com/archives/C066HKS7EG1/p1741268013311159 Worth a shot! Sorry I don't think that link works but I asked their AI bot and it came it with something passable, but probably one for Jules to assist with if he can as the ext owner.
👀 1
m
ooh, thanks for the ping here. That link works for me. I haven’t gotten to try changing from dagster-shell to the PipesSubprocessClient yet but I’m hoping to soon.
We’ve run into a lot of issues recently around weird meltano state issues in dagster-meltano jobs that use the tap-spreadsheets-anywhere extractor.I’m 99% certain that the issue is limited to how the extractor (which does not use the Meltano SDK) doesn’t get an opportunity to gracefully terminate when the parent Dagster process is canceled.
👀 1
a
I do use that tap infrequently (plus a highly modified fork of it), and not experienced any state issues but will keep an eye out. I would definitely welcome graceful interrupts, if I manually stop a pipeline meltano still thinks the job is running. Unfortunately the changes required are well above my knowledge of dagster internals - but happy to do some testing if you fancied having a go. I'll raise an issue on the repo too.
But you've beaten me to it!