I created a first proof of concept of an integrati...
# plugins-generalj
jules_huisman
08/22/2022, 8:20 PMI created a first proof of concept of an integration between Meltano and Dagster. It automatically generates all Software Defined Assets from the taps, and dbt models and shows this in the UI.
This also allows you to easily run subsets of your data pipeline. In this example you can easily run all upstream assets that are used as input for the machine learning model (across different technologies).
I think it would also be cool to connect your dbt models to your BI tooling, to get the whole dependency graph from top to bottom 🤔
a
alexander_butler
08/22/2022, 8:39 PMHow does it work? And can it run a subset of a tap via adjusting the
selectj
jules_huisman
08/22/2022, 8:41 PMYes, it injects the assets you selected using the
selectjules_huisman
08/22/2022, 9:04 PMOh sorry, missed the first part of your comment.
It works by using the SelectService in Meltano which returns all the selected streams from a tap. These are exposed to Dagster as Assets with a certain key, these keys correspond to the sources in DBT (this allows for the edge between singer streams and dbt models)
When you materialize a stream, it uses Meltano to run the tap and target (with the capability of selecting a subset of streams).
For the DBT part I'm using `dagster_dbt`which points to the dbt instance that is installed in Meltano.
a
aaronsteers
08/23/2022, 12:41 AMWe've talked for a while about adding JSON output option for
meltano select listSelectServiceb
binoy_shah
09/01/2022, 2:43 AM@jules_huisman I am on the same path to create such integration, Would you be able to share the integration ? I have started a basic integration that reads just the
meltano.ymls
sterling_paramore
09/12/2022, 4:05 PMI would also be very interested in learning more about how you did this.
sterling_paramore
09/12/2022, 4:06 PMRight now I’m running meltano via dagster, but it’s an op that runs a shell command, so dagster doesn’t really know much about the individual pipelines.
j
jules_huisman
09/14/2022, 1:21 PMIt is going to change in the future, but you can check out the current version here: https://github.com/quantile-development/dagster-meltano/tree/development. I am a bit busy at the moment, so I will probably continue development somewhere next week.