Quick sanity-check on the Airflow integration -- if I want to include meltano-generated airflow DAGs in an existing airflow installation, is there a clean way to do so? From https://meltano.com/docs/orchestration.html#installing-airflow it looks like it wants me to upload my whole meltano repo as a replacement of the airflow dags folder? The docs say "Now, you'll want to copy the DAG generator your Airflow installation's 

dags_folder

," but it's not quite clear how I'd wire this up on an existing install

@paul_tiplady The default DAG generator (https://gitlab.com/meltano/files-airflow/-/blob/master/bundle/orchestrate/dags/meltano.py) looks for the Meltano project in the current working directory you're running

airflow scheduler

from, or the directory identified by the

MELTANO_PROJECT_ROOT

env var. That means you can keep your Meltano project directory and Airflow DAGs folder separate, as long as you set that env var correctly.

Thanks!

@paul_tiplady The venv is machine-specific, so you wouldn't just grab your local one and upload it a to a server, and you wouldn't check it into version control. In your production environment, I'd suggest installing Airflow and Meltano into their own dedicated venvs so that you don't run into dependency version conflicts. When you activate the Meltano venv, install Meltano (https://meltano.com/docs/production.html#installing-meltano) and run

meltano install

inside your Meltano project (https://meltano.com/docs/production.html#installing-plugins), it will automatically create the

.meltano/run/bin

symlink pointing at the installed

meltano

executable, so that your DAG generator will be able to find it based just on the

MELTANO_PROJECT_ROOT

path. The

meltano

package would not be a requirement of your Airflow project, but your Airflow DAG generator would depend on it being able to find a fully-functioning Meltano project somewhere else on the file system

I'll have a dig around to see how this would be done in Composer; I'm using a hosted Airflow setup. Thanks!

@paul_tiplady Ah, all right. In that case you may want to consider containerizing your Meltano project and using it through the DockerOperator instead of BashOperator, so that you don’t have to mess with getting Meltano running on the hosted environment at all.

Is there a good place for me to write up notes on this config? Happy to share my notes, I've got the first moving skeleton working using the KubernetesOperator. I could put up an MR against the tutorials section if that's useful, even if it doesn't get merged?

I could put up an MR against the tutorials section if that's useful, even if it doesn't get merged?

@paul_tiplady That'd be great! Once we have your notes, we can figure out their most appropriate place in the docs 🙂

WIP here: https://gitlab.com/meltano/meltano/-/merge_requests/1810