Related to the above question, but don't want to hijack the thread. Does anyone have recommendations for tutorials on proper secrets management for cloud data infrastructure? Is something like a Hashicorps Vault tutorial the best place for this? https://learn.hashicorp.com/vault

I’m not aware of anything, but do share if you find something 🙂

I haven't written anything up I can share, but the basic idea is to make things 1) based on infrastructure identity, and 2) as dynamic as possible. This means a few things: 1. You should not be storing authentication information that the service uses to authenticate anywhere. This information should come from the provider (k8s service token (JWT), Gitlab CI JWT, AWS instance profile / metadata service, etc. 2. You should not, if you can avoid it, be storing credentials for other services your service accesses as a static credential. This means DB creds should be generated on the fly, API keys for cloud services should be generated as needed, etc. What this may look like is the following flow: 1. k8s cluster created as auth backend on HC Vault (for k8s pods/containers). AWS created as auth backend on HC Vault (for Lamdba functions, Batch, etc). 2. Vault Agent is used with sidecar injector for k8s, or as stand-alone as Batch. The Agent logs in / authenticates to Vault using infrastructure credentials (k8s jwt, AWS, access key id / secret access key / session token). 3. Vault Agent pulls dynamic credentials from Vault for whatever the job needs, static credentials from a kv store only this service has access to for those that can't be dynamic. 4. Service runs, exits when done 5. Vault cleans up any dynamic credentials if service doesn't explicitly revoke them before it exits. Key is that the auth backends on Vault tie the entity (service, Lambda, Batch, whatever) to a specific role based on the infrastructure attributes (k8s ServiceAccountName, AWS Role name, etc). A Vault Policy exists that gives that entity access to only the secrets it needs to do its job, nothing else. If you structure this correctly, you can have a single policy applied to all entities, as you can substitute values in the Vault path with metadata tied to the Vault roles the entity is mapped to. So, service A may have access to: kv/services/service-a/* database/creds/service-a* database/static-creds/service-a* aws/creds/user-service-a aws/sts/sts-service-a And service-b would have access to all of the above

s/service-a/service-b/

. It's all quite elegant if done right.