I'm interested in making sure database credentials are rotated on a regular basis, ideally with Vault Agent running as a sidecar in a Kubernetes deployment. I see environment variables are set for the Database URI. Can they be in a file also, and will the meltano processes track any changes to the database URI? I see for airflow it supports Vault, but it does not appear it will do what we want (appears that it requires "static" credentials stored in a kv, rather than dynamic database credentials). Airflow supports local filesystem secrets for connections though, so we should be able to have Vault Agent render that file? What about the rest of the services?

I'm not sure on Vault specifically. I know it's better than what I used, https://thycotic.com/products/secret-server/ . I used to use Secret Server (but we didn't pick it for this use case). Only mention the product because we just let secret server track history of secrets. We'd populate env variables from Secret Server, and Meltano (or whatever app we were using) would pull from that. Secret server would handle the rotation and things for us (history tracking etc etc) Why do you want things to be in a file? Maybe I'm missing something. I'd love to hear if that's a common pattern seems wrong to store credentials outside of your secret manager to me

The "file" would be an in-memory volume shared with the side-car and main container for the pod, so it is not actually storing the secrets in a file. The secrets are short-lived. For database credentials, maybe 4 hours max. So the question is, if the in-memory file is updated, will the various processes see that. We can run a command to send a signal (SIGHUP or whatever) if necessary, but ideally all processes should be monitoring configuration files and noticing changes.

Are you saying you have a never ending

meltano

process running, and you want the env variables to update during the run?

Yes, for like the UI. Not a ELT run or pipeline, but the UI would always be up, correct? As well as the airflow process. But Airflow may handle this well (or not, will have to test).

You can use the UI if you'd like to. Airflow creates a new meltano process every time the DAG runs, so as long as airflow is setup, and vault agent sets the env variables (or file whatever you want to do) properly you'll be fine. I'm not sure how the UI handles local env variables and things. It's interesting, first how are you imaging you'd use the UI while having airflow running?

I don't know 😉 I've never run the UI or airflow before. Testing so far has been just running meltano from the CLI manually. The Helm chart / Terraform here (https://gitlab.com/kgpayne/poc-meltano-helm-chart) spins up both a Meltano UI pod/deployment as well as an Airflow one. So I'm assuming they can be used concurrently. The Meltano pod would deposit the dag files in a shared (NFS) volume, which the Airflow process would pick up and run the schedules. At least that's my rudimentary understanding. Just as a FYI, my background is security, so using postgres:postgres for DB credentials is a non-starter for me. It would be much more difficult to "fix" the security than figure it out correctly at the beginning. Proper PostgreSQL roles with proper rights to the database (not postgres, master, or superuser) are essential.

If you know you're going to use meltano/airflow everywhere and you already use vault, I totally agree Only issue is that you should really get some usage of Meltano under you belt first imo. Using

.env

files to set credentials (don't have to use postgres:postgres 😉 ) securely that won't get pushed up to git is a great start. I understand wanting to go with Terraform, I'm curious about a few things first. Your company is in production currently using K8's, Terraform, Helm, and Vault? I only ask as sometimes I think people get the cart before the horse, and want to set up everything with all these tools, but don't actually use the tools yet. If yes, then 100% this is the right way to go Yeah I was in charge of security at the last place I worked for a while I understand completely, it depends on the context of what you're doing. A test and something local, it's real easy to use "bad" creds, but when it's a local docker container locked down with fw rules on your local computer, I think it's alright to do imo.

Yes, we are in production with EKS, Terraform, Helm, and Vault, although we prefer Terraform modules to Helm charts. We are already using the Vault Agent sidecar for k8s pods, so it's not new to us.

😄 awesome

Vault Agent can create the .env file, both in local dev and in production (with obviously different secrets). I took a look at the Airflow Vault provider last night. It looks like it is limited to pulling secrets from a kv backend, not the database backend. So it is not ideal. If Airflow updates DB connections based on a local filesystem store, then Vault Agent rendering that file (with dynamic DB credentials) is likely the way to go. If it does not, Vault Agent can kick the container in the pod, or outright kill it if necessary and have it spin back up automatically, when the credentials change.

Hey @fred_reimer 👋 Great to have security brains thinking about these requirements for Meltano 👏 Here is my 2c's: • In general, Meltano has no direct support for secrets backends yet. Users handle secrets via environment variables, as @visch mentioned - either .env based, or using a tools like chamber or the builtin ones for platforms like ECS/kubernetes to inject secrets before the meltano cli is executed. • We currently recommend using

meltano invoke

to execute the Airflow components in your environment. This (among other things) sets additional environment variables from the

meltano.yml

project file before calling the wrapped plugin. This is how secrets set as env vars and then pulled by meltano are passed onto plugins like Airflow. It is also the reason to use env vars in the Meltano format, rather than Airflows directly - Meltano will override them. Its also worth noting, these are only evaluated on process startup. • Airflow specifically is designed to be distributed (separate schedules, workers and UI webservers) all that need to interact with the database to function. So rotation would need to happen on all components to maintain successful running. • Therefore I recommend doing rotation at deployment time. This may mean i) a tool like chamber to fetch secrets from Vault and inject them into the environment before

meltano invoke

is called, ii) a tool/snippet to rotate the db creds and update secrets on a schedule and iii) a scheduled action to redeploy (or at least restart) Airflow after rotation. • If you already have a mechanism for rotating secrets and placing them into an .env file, you simply need to add a hook to redeploy or restart Airflow (via

meltano invoke

following rotation) 🙂 Hope this helps!

I'll let you know when I get everything working 😉 Should not be long, as it's a high priority.