Moving over to deploying my docker image via azure pipelines (rather than running docker locally): it emerges that my

dbt_packages

dir was not getting rebuilt and just picking up my local copy. How can I run a

dbt deps

command in my dockerfile if dbt is installed as part of my dagster utility env? Here's the relevant section of my

meltano.yml

utilities:
  - name: dagster
    variant: quantile-development
    pip_url: dagster-ext dagster-postgres dagster-dbt dbt-postgres dagster-azure pendulum==2.1.2 dagster_msteams
    settings:
    - name: dagster_home
      env: DAGSTER_HOME
      value: $MELTANO_PROJECT_ROOT/orchestrate/dagster
    commands:
      dev:
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker

You can probably add more commands that reference a

dbt

executable

Thanks. I don't think the

dbt

executable is referenceable through commands. But I'm probably overthinking it, can go the brute force method and just remove those packages folders from

.gitignore

I see dbt-postgres is in the

pip_url

, so something like:

utilities:
  - name: dagster
    variant: quantile-development
    pip_url: dagster-ext dagster-postgres dagster-dbt dbt-postgres dagster-azure pendulum==2.1.2 dagster_msteams
    settings:
    - name: dagster_home
      env: DAGSTER_HOME
      value: $MELTANO_PROJECT_ROOT/orchestrate/dagster
    commands:
      dev:
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker
      dbt-deps:
        args: deps
        executable: dbt

might work? Haven't tried myself, though 😅.

I believe the invokers need to be defined here and elsewhere in the codebase, can't be an arbitrary executable https://github.com/quantile-development/dagster-ext/blob/8ff407ec647cbb72fe2b51789026d7e18b66adb2/pyproject.toml#L46

I may be missing something particular about the dagster utility environment, but you can refer any executable that's installed in the plugin's venv. For example, all SDK-based taps install

python-dotenv

at the moment, so the

dotenv

cli is available:

- name: tap-readthedocs
    variant: edgarrmondragon
    pip_url: tap-readthedocs==0.3.0 python-json-logger
    select:
    - projects.*
    commands:
      dotenv:
        executable: dotenv
        args: ""

Then I can run

meltano invoke tap-readthedocs:dotenv list

to print the contents of my

.env

file.

Well that is cool. Should have trusted you would know what you are talking about about!

utilities:
  - name: dagster
    variant: quantile-development
    pip_url: dagster-ext dagster-postgres dagster-dbt dbt-postgres dagster-azure pendulum==2.1.2 dagster_msteams
    settings:
    - name: dagster_home
      env: DAGSTER_HOME
      value: $MELTANO_PROJECT_ROOT/orchestrate/dagster
    commands:
      dev:
        args: dev -f $REPOSITORY_DIR/repository.py --dagit-host 0.0.0.0 -d $REPOSITORY_DIR
        executable: dagster_invoker
      dbt_deps:
        executable: dbt
        args: deps --project-dir $REPOSITORY_DIR/empiric_dbt

results in

(venv) andycarter@DESKTOP-D6ERPSP:~/vscode_projects/***/***$ meltano invoke dagster:dbt_deps
08:30:10  Running with dbt=1.7.4
08:30:11  Installing dbt-labs/codegen
08:30:11  Installed from version 0.10.0
08:30:11  Updated version available: 0.12.1
08:30:11  Installing dbt-labs/dbt_utils
08:30:12  Installed from version 1.0.0
08:30:12  Updated version available: 1.1.1
08:30:12  Installing calogica/dbt_date
08:30:12  Installed from version 0.7.2
08:30:12  Updated version available: 0.10.0
08:30:12  
08:30:12  Updates available for packages: ['dbt-labs/codegen', 'dbt-labs/dbt_utils', 'calogica/dbt_date']                 
Update your versions in packages.yml, then run dbt deps

Thanks @Edgar Ramírez (Arch.dev) every day is a school day 🙂