I am looking at the ways to schedule DBT models independent of an ELT pipeline (I want to run some dbt models that need to be run only after the data from multiple sources is available, and some of the source data is loaded using third party tools). Looking at this thread I can see that I can add my dbt models in transform directory and create an airflow DAG with a BashOperator that executes

meltano invoke dbt ...

. But is there a way this can be managed under

schedules

in

meltano.yml

file? Or is there a plan to add this in future? It would be nice to be able to schedule DBT models independent of export and load, but still using a common way of managing/creating schedules.

There's no concrete plan for non-EL schedules yet, but I have been thinking about reimagining

meltano elt <extractor> <loader> [--transform=run]

as

meltano run [<plugin>...]

, e.g.

meltano run tap-foo target-bar dbt

, where a pipeline could be composed of any number of plugins, including just one:

meltano run dbt

or

meltano run dbt:snapshot

. Then schedules would also be built on top of

meltano run

and support what you're looking for. Please create an issue so we can discuss this some more!

Thanks Douwe! I will create an issue.

More on

meltano run

: https://gitlab.com/meltano/meltano/-/issues/2301

Hi @douwe_maan, this issue covers my requirement. Do you think I should still add an issue?

@pankaj_saini No need, thanks!