jules_huisman
09/29/2022, 3:41 PMdagster-ext
a Meltano extension to automatically interface with Dagster.
The project is still a work in progress, but it would be great to have some beta testers to collect some first feedback.
You can "install" the extension by adding the following yaml to your `meltano.yml`:
plugins:
utilities:
- name: dagster
namespace: dagster
pip_url: dagster-ext==0.0.1a5 dbt-core dbt-postgres
executable: dagster_invoker
settings:
- name: dagster_home
env: DAGSTER_HOME
value: $MELTANO_PROJECT_ROOT/.meltano
label: The home folder of Dagster
- name: repository_dir
env: REPOSITORY_DIR
value: $MELTANO_PROJECT_ROOT/orchestrate
label: The root folder of the Dagster project
commands:
initialize:
args: initialize
executable: dagster_extension
init:
args: initialize
executable: dagster_extension
up:
args: -f $REPOSITORY_DIR/repository.py
executable: dagit_invoker
You can setup the plugin by running meltano invoke dagster:init
(If you want to use dbt, don't forget to change the dbt plugin above.)
(This version doesn't have Singer as Software Defined Assets yet, as Meltano will need some extra cli endpoints to achieve this.)