Hi @Matt Menzenski just wondered if you had any luck looking at the

dagster-meltano

change? Ended up in a bit of jam last week where our dagster alerts stopped working but I couldn't update to dg 1.10.x because of the meltano issue :S

I haven’t yet had time for this. It’ll probably be another two or three weeks before I am able to really look into it.

No worries, appreciate anything you can do. Well out of my depth on this one!

@Andy Carter I’ve now got a fork with an implementation that uses the new PipesSubprocessClient. This is very much an “alpha” stage - I am just beginning to test it locally, haven’t yet tried to run it “for real”. https://github.com/menzenski/dagster-meltano/tree/to-dagster-pipes

just wanted to say that i was digging around for help after i had trouble upgrading to dagster 1.10 - i think i have the same problem with the deprecation of dagster_shell. thanks matt for trying to take a look at this!

The latest news on my side is that I think substituting the dagster-shell call for a PipesSubProcessCall is actually pretty simple/straightforward, if all you care about is that the job runs. But the current dagster-meltano implementation gets the logs (stdout lines) and exit code from the dagster-shell command and returns it to the caller, and the PipesSubprocessClient doesn’t provide a way to that that I have found. I’d like to experiment with providing Meltano a custom log handler that uses the Dagster PipeContext to send logs back to Dagster - I think that if the

dagster-pipes

library is installed in meltano’s virtual environment, and meltano is configured to use that custom log handler, that we could use the PipesSubprocessClient to invoke Meltano and still get meltano logs in Dagster, but I haven’t tried this yet.

Thanks so much for the work on this. I wonder whether it is worthwhile engaging the dagster devs /community on this too? They highlight meltano as an integration via this library.

I have been in touch with them, a bit https://dagster.slack.com/archives/C08HFHSDAJJ/p1745426942884829 I need to follow up with Colton

Can't access that link, is it a private channel?

It’s in the Dagster community Slack (in the #dg-components channel, which is public there)

Hey Matt, just checking if there's a timeline you're thinking of for the changes?

I am hoping to have time to dig into this again next week

Happy to help with some testing as and when it might help.

I still haven’t figured this out. I’ve opened an issue in the Dagster repo https://github.com/dagster-io/dagster/issues/30209

Hey Matt, just checking if you'd had any luck with the PR above or if you are looking for help testing wise

Hope there is some way to get this resolved thanks @joshua_janicas

Me again, sorry :C. A big thing that came up is there is a critical vulnerability found within one of the sub packages https://nvd.nist.gov/vuln/detail/CVE-2025-58367 (DeepDiff) that we can't get away from in Dagster 1.9. Has there been any headway on this?

Keen for something here too - @Edgar Ramírez (Arch.dev) is the dagster_meltano project something that could be brought into meltanolabs somehow? I have no idea what that would entail but ongoing compatibility with a major OSS orchestrator seems like a good selling point.

We got https://github.com/archdotdev/dagster-meltano-pipelines, which we could bring under MeltanoLabs 😉 (cc @Taylor Murphy (Arch)) but it's definitely different to

dagster_meltano

Thanks, I will take a look. From first glance it looks like it doesn't use

from dagster_shell import execute_shell_command

which is the library that was deprecated post 1.9x, that's where the issue with

dagster_meltano

arose. The other difference seems to be that

dagster_meltano

exposes the individual streams as dagster assets (even if they can't be run separately).

dagster-meltano-pipelines

seems to just expose the meltano job as a single asset. It doesn't really matter and I would prefer to be able to update dagster & dbt to be honest!

Sorry Andy, are you saying you think that package only exposes the taps and targets that we have, and not the individual components within them?

I'm guessing that based on: https://github.com/archdotdev/dagster-meltano-pipelines/blob/1da8be5ae64808948e118[…]ster_meltano_pipelines/components/meltano_pipeline/component.py and the fact that 'stream' isn't really mentioned in the code anywhere. https://github.com/search?q=repo%3Aarchdotdev%2Fdagster-meltano-pipelines%20stream&type=code It doesn't seem to expose individual taps and targets, more that each

meltano run tap-* target-*

command you list in the yaml file is exposed as a dagster asset.

I have been spending some time on this and I have been able to get the packages updated to Dagster V12. 1. I copy-pasted over Matt's changes into a local branch of the current dagster-meltano 2. I then chained it to be used from another local branch from dagster-ext 3. Use that locally to build my docker container and updated my repository.py with minimal changes It looks like Matt's changes are 👍, and gets us 90% of the way there (other updates to packages and the poetry files needed)

Amazing work @joshua_janicas and looks like we might be able to get these merged upstream after all. I can do some testing my end too. What were the repo.py changes required? Was that just for dagster v12 compatibility Is a PR to

dagster-ext

required too?

Yeah i have a PR for dagster-ext as well that is needed. I am going ahead anyways with the repo detachment as a 8-9 month turnaround for issues for a package the original maintainers no longer use is...not optimal. We will be maintaining identical licensing etc

Has

elt-dagster-ext

been published to PyPI, or are there plans to do so? The

pip_url

can simply point to the git repo, so it's ok if not.

I'm currently in the process of getting

elt-dagster-ext

to pypi, should be done latest monday (https://github.com/camisinc/elt-dagster-ext/pull/1)

Gotcha

perfect, once i get the elt-dagster-ext to pypi i'll let you know so we can rerun that test job?

Exactly, thanks!

This has been done, we now have elt-dagster-ext on PyPi https://pypi.org/project/elt-dagster-ext/

Merged!

Thank you so much! I assume it'll be something like 24-48 hrs before the changes appears on the hub?

A few minutes

Awesome, we're in business !

Happy to help! And thank you for taking up maintenance of those projects.

Wonderful news, looking forward to some testing later this week.

i haven't personally encountered that before, but if you're able to help pinpoint the steps to reproduce / where that could be happening i can take a look when i have bandwidth.

I did a quick test and appears the behaviour is the same. If you trigger a meltano run from dagster then terminate the run the pipeline run is left in the 'RUNNING' state in the meltano systemdb. I didn't expect any change really but just in case the move away from

dagster_pipes

gave us that automatically.

Yeah it was basically a port over of Matt's solution from https://github.com/quantile-development/dagster-meltano/pull/58/files#diff-38bde903a607df211a8190f891c1a5e6d9bbdf604f9859c05c2cded8f222420c ; if we're talking about the same thing. My goal was just to get things moving again

I had seen Matt's original attempt to use dagster-pipes I believe, but not the second version. Anyway great to have a future for meltano and dagster, thankyou 🙂

I'd also like to eventually resolve the meltano edk vendoring that's going on in the dagster-ext solution too