Hi guys, is there a way to use already running AirFlow in Meltano as the orchestrator?

@or_barda Absolutely! Using Meltano's "built-in" Airflow orchestrator really just means installing it into a Meltano-managed Python venv inside

.meltano

, and invoking it through

meltano invoke

so that some default Meltano-compatible configuration can be dynamically applied at invocation time: https://gitlab.com/meltano/meltano/blob/master/src/meltano/core/bundle/discovery.yml#L840-855 That config points Airflow to ``$MELTANO_PROJECT_ROOT/orchestrate/dags`` to look for dags, which is where you'll find https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py (since

meltano add orchestrator airflow

also add the Airflow file bundle from that repo). That DAG should work with any Airflow installation as long as the

MELTANO_PROJECT_ROOT

env var is set to point at the Meltano project directory, and as long as Meltano itself is installed and

meltano install

has been run inside the project. That setup assumes you'll use

meltano schedule

to schedule your

meltano elt

pipelines, since it iterates over the result of

meltano schedule list --format=json

to create DAGs for each, but you can also create your own DAGs for any pipeline you fancy using

BashOperator

+

meltano elt

, or

DockerOperator

+

docker run <your project image> elt ...

(see https://meltano.com/#containerization)