Hi, I have some confusions regarding using Airflow with Meltano. I did my initial POC of Ingesting Data from MySQL to Snowflake, but I realized that the scheduled jobs dont work. Reading https://docs.meltano.com/guide/orchestration, hinted that repeated scheduled jobs will require an Airflow scheduler running. I currently deploy Meltano via

meltano ui

in Kubernetes as a Deployment+Pod My doubts when I deploy it on Kubernetes 1. Does Airflow have to run only via the

meltano airflow

command 2. Does Airflow need to have meltano pipeline codebase packaged in the docker image 3. Can I Publish and run different Meltano pipelines (

meltano.yml

) on a Remote Airflow installation (helm chart) that is pre-existing and connected to its own Postgres Db

Hey @binoy_shah ! Good questions - I dont know all the answers but I'll see if I can help. The schedule feature that youre referring to does require an orchestrator to be running. You define your schedules in Meltano as configurations and those get propagated to the orchestrator (right now mostly just Airflow). Check out this blog by @ken_payne about how we deploy meltano for meltano and you can refer to this repo to see our actual code (but were still iterating on it a bit).

1. Does Airflow have to run only via the 

meltano airflow

  command

No, its common especially with a k8s deployment to use the default airflow helm chart and have Airflow just execute Meltano commands and lots of teams already have an Airflow deployment so they do the same. If youre using environment specific configurations you wont get those benefits unless you use the

meltano airflow

.

2. Does Airflow need to have meltano pipeline codebase packaged in the docker image

This depends on how you want to do it. We have a Meltano image and an Airflow image, the Meltano image contains a fully installed Meltano project that gets called by Airflow.

3. Can I Publish and run different Meltano pipelines (

meltano.yml

) on a Remote Airflow installation (helm chart) that is pre-existing and connected to its own Postgres Db

Yes you can do this also!

Thanks for the hints @pat_nadolny I will run through this further.. If I run existing Airflow out of Meltano's ecosystem, how does the Dag get pushed to Airbyte ?