Hi everyone, I have cloud composer environment running on GCP, I’d like to know how I can configure meltano to use this composer environment ? and if possible containerize meltano, Thanks !

Hi @fabien_ryckoort Meltano is very straightforward in this containerized approach. There are loads of threads on it actually: This is a good one: https://meltano.slack.com/archives/CMN8HELB0/p1641473284007700 Basically, the options are: 1. Build a new layer https://hub.docker.com/r/meltano/meltano and 'meltano install' a specific project in that layer, use that container with no change in production 2. Build a new layer with your python dependencies etc, but wait until runtime to install. Bit slower, but much more flexible in terms of what will be run without the container build overhead https://meltano.slack.com/archives/CMN8HELB0/p1658347639573979?thread_ts=1658329498.049159&cid=CMN8HELB0 If I were building a platform that only needed to run on Kubernetes I'd look at Argo If you want a simple UI / API for your meltano projects check out our plugin or run our community edition with docker compose: https://medium.com/@danielpdwalker/a-user-interface-for-meltano-397089465cd9

Hi @aaron_phethean Thanks for your feedback, I red this article, but, that’s not really clear for me, on how to tell composer to use dags inside

orchestrate/dags

folder… because, by default composer mount gcs fuse volume inside the container and use dags stored in GCS, right ?

Hi @fabien_ryckoort to be honest I'm not really an Airflow expert. You might need to post more of a complete project goal and someone could help - I thought you were after how to run meltano in a container... This is more like how to use Airflow best practice right?