Is using different environments as simple as movin...
# best-practices
i
Is using different environments as simple as moving your meltano config settings under "dev" or "prod" and then setting some env var to "DEV" and "PROD" to point your runtime environment to the right config? Like if I want to test out a tap in dev I can just add it to my dev config and test it there before I add/change it in my production config?
c
yes, as long as you are using the SQLite state storage backend. If you decide that you are going to use something like PostgreSQL for the state backend, then separate configs in the meltano.yml no longer really make sense because
MELTANO_DATABASE_URI
is an environment variable only configuration item and decades of experience have taught me never to mix environment variable configuration with multiple environment file configuration (because… I know, I know I will forget to set the correct environment variable and push dev state into production or vice versa). (Yes you could get around this by aliasing
meltano
in your shell to set the variable based on the environment switch, but frankly not a risk I am willing to take.)
i
The risk being that you keep the same MELTANO_DATABASE_URI env var and incorrectly update the state of your pipelines?
c
yes, exactly that
I will tolerate a lot of things, but a risk of pushing the incorrect state for an environment that is a carbon copy in structure but not data is never one of those things, especially if I have
autoincrement
primary key values
i
Yeah that's fair. I'm using az blob store for my state backend and the connection string remains constant across each blob container so that env var would remain constant across both my stg and prod environments. I think I should be okay since I can manually set the state backend uri in my prod/stg configs seperately to point at their own respective blob containers.
I also just generally need to move more of my config to my meltano.yml too many of my parameters are set through my env vars
c
I set pretty much everything through environment variables, but that’s my general approach on every project. There’s a guarantee that on a Linux host environment variables will work, and you can securely inject them into a container in a build environment so they are my default go to.
i
Gotcha, well thx for the input