Hey! can we use airflow and meltano on two different docker images and both docker images are running on two different machines ?

@gray-apartment-22627 It's a bit hard to answer that without knowing what you are trying to accomplish 🙂 Are you saying you'd like to run the Airflow scheduler on a different machine than the machine that ends up executing the scheduled Meltano ELT jobs? That could be a job for Airflow's KubernetesPodOperator (https://gitlab.com/meltano/files-airflow-kubernetes/-/merge_requests/1)

@ripe-musician-59933 does that mean we copy the DAG in airflow specific POD and the KubernetesPodOperator connects with Meltano POD which has specifically Meltano cli available to trigger the ELT pipelines, correct ?

@adventurous-lifeguard-17150 As I understand it, the KubernetesPodOperator would manage the dynamic creation of pods for each ELT pipeline when the schedule says its time to. @careful-journalist-73001 who wrote that MR will know better!

Hey all, @gray-apartment-22627 the answer to your question is yes, but what is it you're trying to achieve? @adventurous-lifeguard-17150 Douwe is right, the KubernetesPodOperator creates a new pod for each task it creates based on the image URI you supply. Take a look at the merge request linked to see it in practice

@careful-journalist-73001 we are just looking a way to use common airflow @ripe-musician-59933 we will test and confirm you on KubernetesPodOperator

@ripe-musician-59933 one thing which is not clear to me - like if you have Meltano & Airflow these 2 are running as 2 different PODS, how airflow has meltano cli available to get the schedules and prepare the pipelines on airflow

@adventurous-lifeguard-17150 The Meltano CLI and project will need to be installed inside the Airflow pod, and all pods need to connect with the same external Meltano system database. If you install Airflow into the Meltano project and create a project-specific Docker image as described in https://meltano.com/docs/containerization.html, you can use that image both when running Airflow scheduler (with

meltano invoke airflow scheduler

as the command) and when running ELT pipelines from Airflow, (with

meltano elt <args>

or

meltano schedule run <name>

as command)