disclaimer: i work at a password management company, and I dogfood our cli application on the data team.

locally: the env variables are specified in the docker compose file under the key

environment

as a list of variable names normally, the variables would get passed from the host environment to the container environment. However, our cli application allows fetching the secrets and passing it into a process like below, and the benefit is that the credentials don't stick around in my host environment afterwards.

op run --env-file .creds --no-masking -- docker compose run melty tap-example target-example.

As soon as the container exits, the secrets go poof. Accessing the secrets requires biometrics / password, so they are protected before getting fetched from the vault. Rotating these development credentials is simple (we have to update the secrets in 1 place to which data team have references. the file

.creds

doesn't actually contain any secrets, just secret references. each line resembles something like this:

MELTANO_ZENDESK_CLIENT_SECRET="<op://Data/zendesk/client-secret>"

in prod, we run our meltano workload on aws-batch. For historical reasons, we use aws secrets manager, and we can specify the secrets as environment variables holding secrets as references in to the secrets in terraform. Each job is provisioned with only the secrets needed for 1 tap & 1 target that forms the EL pipeline.

non secret environment vars get specified in the docker-compose file locally or at invocation time in prod.

Locally I have been using the .env file but I need to look into using the 1Password CLI - we recently migrated to 1Password 👀