Hey community! I read the documentation but I'm not sure how I can make a transformation dependent on multiple extractions. Is it possible? Just as I would do using Airflow DAGs?

@murilo_tavolaro_de_nigris Meltano doesn't have its own concept of dependencies, and

meltano elt

always runs one specific EL pair with (if

--transform=run

is set) the related transformations. In your case, I'd suggest actually wiring up your own Airflow DAG with 2 or more operators running

meltano elt

(without

--transform=run

), followed by an operator running

dbt run

(or

meltano invoke dbt run

)

Just to be sure I understood, I should create a dag that sequences 2 bash operators to run 'meltano elt' to extract and load, and after a bash operator running 'dbt run', is that it? Where should I put this dag code?

@murilo_tavolaro_de_nigris If you're using Meltano's Airflow integration and

meltano invoke airflow scheduler

, Airflow is automatically configured to look for DAGs in your project's

orchestrate/dags

directory. In there, you can write Airflow DAGs just like you would without Meltano. You're free to remove or modify the default

meltano.py

DAG generator that iterates over the result of

meltano schedule list --format-json

and creates a new 1-step DAG for each of your project's schedules.

🤩 , thank you for the explanation Douwe, 100% clear now 🙂

@murilo_tavolaro_de_nigris Awesome 🙂