I was able to deploy Meltano in production using Airflow as orchestrator and Postgres as the database 🏁 😄 Thank you for the help @douwe_maan and @niall_woodward! I still have a few questions though: 1) I don't think I did the best CI/CD pipeline, but I don't know how to improve. I accept suggestions 😬. What I'm doing is: 1. I develop my project in a branch in my local machine 2. I push my branch to remote 3. I do merge into master (I'm using gitlab) 4. Gitlab triggers the CI/CD script that I got from the example, pushing a docker image of my Meltano project into the gitlab registry 5. I go into my production machine (Google Cloud Compute Engine) 6. I clone the repo from gitlab into the machine 7. I start all the services (meltano UI, airflow UI, airflow orchestrator and postgres db) using the docker-compose.prod.yml example file

sudo docker-compose -f docker-compose.prod.yml up -d

the only modification I made in the docker-compose.prod.yml file was to reference the gitlab registry where I have the image of my project Apparent disadvantages of the processes above: • I create a docker image, but I still have to clone the repo manually in production • Everytime I do a new deploy, I've to restart all services, shouldn't I only restart meltano? • Doesn't seem so neat and organized. 2) Another question is: I'm deploying meltano as read-only project, however I want my team to be able to pause/play or change the schedules on the UI without having to deploy anything on code. Even if I deploy as not read-only, everytime I do a new deploy, I'll override their schedule preferences, correct? Is there a way of avoiding this? 3) As I'm creating custom DAGs they'll only be seen on Airflow, and the normal ELTs will be seen in Meltano, so my team should access both UIs?

1. Docker Compose is actually not a great option in production; it's primarily intended for local usage during development and testing. Note that https://meltano.com/docs/containerization.html#docker-compose says it let's you "_experiment_ with a production-grade setup of your containerized project" 🙂 If you want to containerize your project and only pull the image from the container registry instead of also needing to pull the repo, I'd suggest using Kubernetes. Do you have experience with that?

1. a) Ohh, I understood the opposite, I my translation just understood the 'production-grade' part not the 'experiment' one hahaha b) No, I don't have experience with Kubernetes, I tried 2 weeks ago, but it was taking me long to figure it out, so I though the docker-compose file would be easier. Actually, the docker-compose makes MUCH easier to do the complete setup, with airflow and postgres. I really liked it. In my head the ideal scenario would be, once I merge my branch to master, the gitlab ci/cd would have a 'deploy' script to create all the containers (or kubernetes pods) for meltano, airflow and postgres into google cloud/aws/azure But I don't yet how to create this script. (Maybe in the end of the project I have some time to spare to learn kubernetes and figure it out)

Some work is being done/discussed in https://gitlab.com/meltano/meltano/-/issues/2149 to make it easier to deploy Meltano/Airflow on K8s using Helm, but it's not done yet, and K8s definitely has a bit of a learning curve if you're new to it and trying to figure out how to deploy Meltano by yourself. I'm a total K8s novice as well, so I can't help you much more there unfortunately 😅

I saw that dicussion, but I understood almost nothing haha. I'll try to figure it out later. For now I'll keep with this hybrid approach hehehe