Questions regarding production in Docker: 1. Once ...
# announcements
z
Questions regarding production in Docker: 1. Once I followed the Containerization docs, I pushed it to gitlab, but then once I am on the VPS, how do I pull it? Should I pull it by using Clone? 2. When I pull it will it have preinstalled all taps, targets and scheduler, or should I run a command to install them? 3. How secure is the data in .env?
d
1. Containerizing your project is most useful if you already have experience with Kubernetes, which you could point at the GitLab container registry, so that it can take your Docker image and run it as a service with command
invoke airflow scheduler
. If you don't have Kubernetes experience yet and just have a Linux box you can SSH into, it'll be easier to skip containerization and clone the Git repo into a directory, and set up your system's service manager to make sure that
meltano invoke airflow scheduler
will always be running. The CI/CD job for your repo could then SSH into the VPS and run a little deploy script (that can live in your repo) that will run
git pull
,
pip install meltano
, and
meltano install
(as suggested in https://meltano.com/docs/production.html if you skip the "Containerized Meltano project" sections), followed by the system-appropriate command to restart the
meltano invoke airflow scheduler
service so that the latest changes are picked up.
2. If you're containerizing your project, all plugins will be installed already: https://meltano.com/docs/production.html#containerized-meltano-project-3 If not, you can do so using
meltano install
: https://meltano.com/docs/production.html#installing-plugins
3. As secure as any file on your machine! I wouldn't suggest checking it into your repo though, so you'll want to either manage that file directly on the VPS, or use a separate service to manage env vars
Note that you could also use the containerized Meltano project Docker image with Docker Compose: https://meltano.com/docs/containerization.html#docker-compose, and then add
docker-compose up -d
as a service on your VPS, but then you'd still need to clone the repo on the VPS (since the repo contains the
docker-compose.yml
file), and you'd still need your CI/CD job to tell the VPS to pull the repo and Docker image again and restart the service, so if you go that route, you won't be getting much benefit from Docker.
z
Got it, thank you!