Are environments necessary to specify in the project? I’m used to a twelve-factor app practice of keeping environment details out of the codebase. Is there a way to define a project in an environment-agnostic way and pass all environment-specific information at runtime?

The short answer is that you can use environment variables and templating for most things so you do have the option to essentially ignore the environment features and use a single templated project. But I would say that the environment features in Meltano act at the config level so youre not necessarily putting environment specific information in an application in the same way that the twelve factor app docs are telling you to avoid. Meltano lets you config multiple environments so you can check the config in as code vs externally managing several sets of environment variables manually. Sensitive configs are still managed in .env files and in environment variables for security reasons though. Does that make sense?

yes, it does. I guess I’m just hesitant to enumerate all my environments in one file. (We have six environments).

I totally get that - if its helpful, for our internal meltano project I chose to split out each environment to its own file https://github.com/meltano/squared/tree/main/data/environments (using the include_paths feature). I've also seen others have a single environments file with all their environment specific configs and another top level config for all their plugin installs and shared common configs. For me I split those top level configs out by plugin type in subdirectories i.e. extract/extractors.yml and load/loaders.yml all linked together using a meltano.yml at the root of my project. But I'm not sure I love that approach (still tinkering with what I like)

oh neat, I was going to ask if they could be split into a file per environment. That’s something I’d be more comfortable with (we are doing the same with our terraform stuff). Thanks for that!

@Matt Menzenski - Another thing to call out is that your environment declaration can map to env vars in the execution context. So, for instance, even though 'database_host' and 'database_port' might be configured differently per environment, your meltano.yml file(s) don't have to directly declare those values if you'd rather pull them from runtime context. So, if you'd like to keep certain aspects config entirely separate from meltano.yml, you can still do that while also declaring and using Meltano environments. The only thing I want to call out is that the addressable env vars for prod-like environments generally should probably not be named the same as your local dev/test ones. So, you might want to name env var for

PROD_POSTGRES_HOST

differently from

LOCAL_POSTGRES_HOST

to avoid mistakes if you (for instance) forget to reset env vars between executions. If you run

meltano run --environment=prod my-production-job

, it's generally better to get

Error: PROD_POSTGRES_HOST env var is not declared.

than it is to think you just updated production when you actually only updated the db on your localhost.

is there a guide anywhere on using environment variables in this way? (like, preferring them over in-file settings). I see https://docs.meltano.com/guide/configuration#configuring-settings - is that the best source for documentation?

As of today, no, I don't think we have a formal guide. cc @Sven Balnojan