Is there a way to set database_uri for different envs? Feels like an important part of env functionality but I’m having trouble figuring out how to do it.

Setting the environment variable

MELTANO_DATABASE_URI

should do the trick. You can see the corresponding env vars for each Meltano setting by running

meltano config meltano list

.

@quinn_batten try using the CLI instead for one or two to see how the strucutre is supposed to be. The answer is yes it does work, it's much easier to write this though

meltano --environment=dev config target-postgres database_uri abcdefg

And

env

is really overloaded at the moment so it's hard to understand what you're after exactly, but I've been through that exact pain and the config command above leads you to

environments:
- name: prod
  config:
    plugins:
      extractors:
      - name: tap-name
        config:
          config_key: config_value
      loaders:
      - name: target-azuread
        config:
          config_key: config_value
      utilities:
      - name: autoidm-transform
        env:
          EMAIL_SUFFIX: '@abc.com'
- name: dev
  config:
    plugins:
      utilities:
      - name: autoidm-transform
        env:
          EMAIL_SUFFIX: '@testabc.onmicrosoft.com'

As you can see

env

(OS Environment variables) ,

.env

(a file on the os) ,

env

as a shorthand for

environments

, and the

env

key listed above makes it hard to answer your question 😄

Re: MELTANO_DATABASE_URI— What I ideally want is to set that value in meltano.yml under the env block. Idea being that I can use the same meltano.yml in dev and prod and just set the

MELTANO_ENVIRONMENT

env var in order to switch environments, without any other finicky changes

https://meltano.slack.com/archives/C01TCRBBJD7/p1664324061356579?thread_ts=1664314402.975189&cid=C01TCRBBJD7

I think what you mean is you want to set the

database_uri

config value for your

target

. Which isn't under the

env

block it's under the

config

block as I showed above. Is that not working for you? If not please share your meltano.yml

Hey @quinn_batten ! Just to clarify are you referring to the database uri for your meltano system database that stores your state or for an actual tap/target e.g. tap-postgres?

database_uri for the meltano system database, the value that is set by MELTANO_DATABASE_URI in .env, or by database_uri in meltano.yml

Personally this feels like a feature that we should support but yeah it sounds like its not directly supported without using env vars, maybe thats intentional idk.

I want my dev environment to point at a different meltano system database than prod, so that a dev run of a pipeline doesnt affect prod state for that same pipeline. (I don’t think meltano handles that sort of differentiation automatically, right?)

This is actually already a thing depending on the version of meltano that youre on. The state IDs include your environment name if you're using

meltano run

see the docs for format https://docs.meltano.com/reference/command-line-interface#run

Thanks @pat_nadolny, that’s really helpful! I just want to confirm that Meltano does that same thing (change state_id(?)) for scheduled jobs? My pipelines are orchestrated by airflow so I’m not ever doing a “meltano run …” command. I’m looking at my system database right now and I don’t see anything that seems related to environments under the

runs

table, which I think is where state is tracked (I’m on Meltano v2.4). The job_name column is just the name of the pipeline w/out anything else, the payload doesnt reference environments either. The only thing I see that could maybe be separated by environment is run_id, but I think that’s unique on the run, not the pipeline/job