Sorry for asking again. I willing to run each meltano taps inside different containers, and one meltano ui container that run on different machine, that points to the same postgres database, and another meltano container on third machine that will run airflow scheduler. So, do the containers need to share anything else, like, log files, meltano yml or something like that? Or, they all need to point to the same database, and I will be able to see the pipelines on meltano ui, and the DAGs on airflow? How the meltano ui will know about the existing pipelines if they are not in the meltano yml?

I willing to run each meltano taps inside different containers, and one meltano ui container that run on different machine, that points to the same postgres database, and another meltano container on third machine that will run airflow scheduler.

@meir.shamay All right, we should be able to make this work 🙂

So, do the containers need to share anything else, like, log files, meltano yml or something like that?

meltano.yml

defines the Meltano project, so the containers will definitely need to have the same

meltano.yml

, although they won't be writing to it anymore, so they don't need to have a shared

meltano.yml

. See https://meltano.com/docs/production.html#and-onto-the-production-environment and the "Containerized Meltano project" section below it. If you'd like the logs to be available in Meltano UI, you're going to run into https://gitlab.com/meltano/meltano/-/issues/2107. As a workaround, you can mount the

.meltano/run/elt

directory, as discussed in https://meltano.slack.com/archives/CFG3C3C66/p1595531339303500.

Or, they all need to point to the same database,

Yep: https://meltano.com/docs/production.html#storing-metadata

and I will be able to see the pipelines on meltano ui, and the DAGs on airflow? How the meltano ui will know about the existing pipelines if they are not in the meltano yml?

If you're not using

meltano schedule

and there are no pipelines in

meltano.yml

, the Meltano UI will not be able to show you any pipelines. You will only be able to use the Airflow UI to see the status of your DAGs, which are your Meltano pipelines generated on-the-fly, which your Meltano project (

meltano.yml

and Meltano UI) don't know about.