Hey! can we use airflow and meltano on two different docker images and both docker images are running on two different machines ?

@ankur_adhikari It's a bit hard to answer that without knowing what you are trying to accomplish 🙂 Are you saying you'd like to run the Airflow scheduler on a different machine than the machine that ends up executing the scheduled Meltano ELT jobs? That could be a job for Airflow's KubernetesPodOperator (https://gitlab.com/meltano/files-airflow-kubernetes/-/merge_requests/1)

@douwe_maan does that mean we copy the DAG in airflow specific POD and the KubernetesPodOperator connects with Meltano POD which has specifically Meltano cli available to trigger the ELT pipelines, correct ?

@ojasvi_harsola As I understand it, the KubernetesPodOperator would manage the dynamic creation of pods for each ELT pipeline when the schedule says its time to. @niall_woodward who wrote that MR will know better!

Hey all, @ankur_adhikari the answer to your question is yes, but what is it you're trying to achieve? @ojasvi_harsola Douwe is right, the KubernetesPodOperator creates a new pod for each task it creates based on the image URI you supply. Take a look at the merge request linked to see it in practice

@niall_woodward we are just looking a way to use common airflow @douwe_maan we will test and confirm you on KubernetesPodOperator

@douwe_maan one thing which is not clear to me - like if you have Meltano & Airflow these 2 are running as 2 different PODS, how airflow has meltano cli available to get the schedules and prepare the pipelines on airflow

@ojasvi_harsola The Meltano CLI and project will need to be installed inside the Airflow pod, and all pods need to connect with the same external Meltano system database. If you install Airflow into the Meltano project and create a project-specific Docker image as described in https://meltano.com/docs/containerization.html, you can use that image both when running Airflow scheduler (with

meltano invoke airflow scheduler

as the command) and when running ELT pipelines from Airflow, (with

meltano elt <args>

or

meltano schedule run <name>

as command)