hey folks, when trying to run a custom Python script and expand an env variable that contains the

MELTANO_ENVIRONMENT

variable (defined in my

.env

) as

utilities:
    - name: run-my-script
      namespace: run_my_script
      env:
        database: "db_${MELTANO_ENVIRONMENT}"
        source: db
        table: mytable
      commands:
        update_state:
          executable: python
          args: ...

I'm getting

db_

inside the script. I know the

MELTANO_ENVIRONMENT

is defined in the context because I can log it with

os.getenv('MELTANO_ENVIRONMENT')

hmmm Also to debug your issue I'd look at the print-var option

meltano  invoke --print-var database run-my-script

Looks like a bug to me in meltano, please submit an issue and show how to replicate. A workaround for you would be to use high level env vars environments: - name : dev env: database: "db_dev" and then reference the database env var in your utility (or you could just not reference it in the utility as it'd be available now from this top level setting

Thanks @visch - yeah, I tried your debugging command as well without luck. As you mentioned, it seems like a bug. We use yaml anchors and env variables to pass env-specific variables without repeating ourselves and that didn't work either. I ended up just defining the variable in the shell and using it inside the script with a different name. I'll take the time to file the issue.

visch@DESKTOP-9BDPA9T:~/git/meltano-projects/jose_debug$ cat meltano.yml
version: 1
default_environment: dev
project_id: dd8fe0cf-30c1-4e6d-b74c-f68ac5fac7fd
environments:
- name: dev
  env:
    test_var: db_dev
- name: staging
- name: prod
plugins:
  utilities:
  - name: run-my-script
    namespace: run_my_script
    env:
      database: ${test_var}
      source: db
      table: mytable
env:
  test_var: $test
visch@DESKTOP-9BDPA9T:~/git/meltano-projects/jose_debug$ meltano invoke --print-var database run-my-script
2024-08-22T13:29:31.380095Z [info     ] Environment 'dev' is active
database=db_dev

Yeah @jose_escudero I have made it work, I can show a few examples. Just posted the above as just want to be sure you see it working

Thanks for the example. Yeah, it seems that when you explicitly define it at the env level it's correctly picked up. The problem seems to be when you are using variables from the shell (or

.env

)