Just onboarding to Dagster, trying to orchestrate Meltano, dbt and GoodData with it. I executed all dbt models successfully, I can see each dbt model as asset in Dagster and can display its lineage. I successfully executed Meltano as a job, I followed the Dagster tutorial(https://dagster.io/blog/dagster-meltano-integration-tutorial) and dagster-meltano README(https://github.com/quantile-development/dagster-meltano). Now I am thinking about how to connect these two world together in Dagster. My idea is to load Meltano jobs as assets and define dependencies between Meltano and dbt assets. For instance, I extract/load github data from repo and org levels (two different taps). I would like to define two dependencies: • tap-github-repo -> dbt models commits, pull_requests, ... • tap-github-org -> dbt model repositories The example in the dagster-meltano repo shows how to expose meltano jobs into Dagster. But I would like to expose assets so I can include them into the Dagster lineage. Anyone tried to do something like that?

the actual "meltano runner" looked something like this tap_powerschool_select_filter=ps-modelname (from above step)_ dbt_modelfilter=dbt-modelfilter (same as above)

meltano run tap-powerschool target-mssql

I hacked my together as I just needed to generate it once but it could be much much much better Hope that helps a bit, I didn't actually use Dagster for mine this was for something else 😕

@jan_soubusta here's how I achieved it https://github.com/quantile-development/dagster-meltano/issues/28#issuecomment-1655283987

Thanks! Meanwhile I joined the Dagster community and they answered me the same 😉

@jan_soubusta you are welcome, let me know of any improvements you make, the code is very much 'make it work', I'm sure there's efficiencies and improvements to the setup you could make. The one annoyance is that you have to list out all the separate meltano streams from a tap to get the dependencies to show. If you find a way around that, let me know 🙂

I hit another issue - Python package conflicts between Meltano and dbt. Is it possible to orchestrate both from Dagster and workaround the conflicts? How?

I installed dbt into my

dagster

env

utilities:
  - name: dagster
    variant: quantile-development
    pip_url: dagster-ext dagster-postgres dagster-dbt dbt-postgres dagster-azure
    settings:
    - name: dagster_home
      env: DAGSTER_HOME
      value: $MELTANO_PROJECT_ROOT/orchestrate/dagster
    commands:
      dev:
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker

hm, maybe I am approaching it incorrectly. I try to install everything using dagster and maybe it will work out-of-the-box

I'm not sure what the 'correct' approach is here, but having dbt in the dagster env works ok for me. It makes a bare

dbt run

from the command line when working locally a bit more tricky. I have to activate the dagster env first to achieve that.