Hi everyone! I'm deploying meltano in k8s gke. In order to store secrets, we use vault hashicorp. We have a sidecar that mounts the secrets into the meltano container and the secrets can be available as env variables but only during meltano startup e.g

containers:
  command:
    - /bin/bash
    - '-c
  args:
    - source /etc/priv/credentials/envvars && meltano ui

source /etc/priv/credentials/envvars

- exports the secrets as env variables. When I check for the values of meltano config

meltano config --format=env meltano

I see the meltano default values and not the ones that have been exported during start up (

meltano ui

). Is there a way for the meltano process to read from the exported env variables?

Try

meltano config meltano list

as it'll tell you where each config var is coming from. It should help you debug this

yeah, it's picking up from the default eg

ui.secret_key [env: MELTANO_UI_SECRET_KEY] current value: 'thisisnotapropersecretkey' (default)

. I think my question is why meltano isn't picking the values from the exported ones

can you define

exported ones

This command

source /etc/priv/credentials/envvars

executes

export MELTANO_UI_SECRET_KEY="anewsecret"

but it gets executed in entrypoint before

meltano ui

My guess is

source /etc/priv/credentials/envvars

isn't setting the values the way you expect them to be. Maybe it's

source /etc/priv/credentials/envvars && meltano ui

doesn't pass those vars to that command?

if I check

cat /proc/1/environ

I can see the env variables set from

source /etc/priv/credentials/envvars

Can you check the

meltano ui

proc as well?

meltano ui

is the first process

Sounds like you're better at

bash

kungfu than me. So for a test could you swap to

containers:
  command:
    - /bin/bash
    - '-c
  args:
    - source /etc/priv/credentials/envvars && meltano config meltano list

ok will try that

visch@visch-ubuntu:~/git/meltano-test-6212022$ export MELTANO_UI_SECRET_KEY=TEST && meltano config meltano list

Works

ui.secret_key [env: MELTANO_UI_SECRET_KEY] (default: 'thisisnotapropersecretkey') current value: 'TEST' (from the environment)

ok will check my entrypoint. Thank you for testing

what are your processes running before you execute

source ./envvar && meltano config meltano list

if you don't mind sharing.

Nothing interesting

and did you test it in a docker container?

No but it should be the same thing

So for my testing locally with docker-compose when I run

source ./envvar && meltano config meltano list && meltano ui

I see the secrets picked up from

./envvar

but then when I get into the docker container

docker exec ...

and I run

meltano config meltano list

I see the meltano default values. That's the same behavior I see in k8s

Works for me ```visch@visch-ubuntu:~/git/meltano-test-6212022$ podman run --entrypoint /bin/bash -it meltano/meltano root@29976156396b:/project# ls root@29976156396b:/project# meltano init project Created project Creating project files... project/ |-- .meltano |-- meltano.yml |-- README.md |-- requirements.txt |-- output/.gitignore |-- .gitignore |-- extract/.gitkeep |-- load/.gitkeep |-- transform/.gitkeep |-- analyze/.gitkeep |-- notebook/.gitkeep |-- orchestrate/.gitkeep Creating system database... Done! ████ █████ ░░███ ░░███ █████████████ ██████ ░███ ███████ ██████ ████████ ██████ ░░███░░███░░███ ███░░███ ░███ ░░░███░ ░░░░░███ ░░███░░███ ███░░███ ░███ ░███ ░███ ░███████ ░███ ░███ ███████ ░███ ░███ ░███ ░███ ░███ ░███ ░███ ░███░░░ ░███ ░███ ███ ███░░███ ░███ ░███ ░███ ░███ █████░███ █████░░██████ █████ ░░█████ ░░████████ ████ █████░░██████ ░░░░░ ░░░ ░░░░░ ░░░░░░ ░░░░░ ░░░░░ ░░░░░░░░ ░░░░ ░░░░░ ░░░░░░ ~ Your DataOps Platform Infrastructure ~ Project project has been created! Meltano Environments initialized with dev, staging, and prod. To learn more about Environments visit: https://docs.meltano.com/concepts/environments Next steps: cd project Visit https://docs.meltano.com/getting-started#create-your-meltano-project to learn where to go from here root@29976156396b:/project# cd project root@29976156396b:/project/project# meltano config meltano list 2022-06-22T140855.241076Z [info ] Environment 'dev' is active send_anonymous_usage_stats [env: MELTANO_SEND_ANONYMOUS_USAGE_STATS, !MELTANO_DISABLE_TRACKING] current value: True (default) project_id [env: MELTANO_PROJECT_ID] current value: 'b60fc83e-9cf3-4386-baa6-d78e8bd5c1c9' (from

meltano.yml

) database_uri [env: MELTANO_DATABASE_URI] current value: 'sqlite:////project/project/.meltano/meltano.db' (default: 'sqlite:///$MELTANO_PROJECT_ROOT/.meltano/meltano.db') database_max_retries [env: MELTANO_DATABASE_MAX_RETRIES] current value: 3 (default) database_retry_timeout [env: MELTANO_DATABASE_RETRY_TIMEOUT] current value: 5 (default) project_readonly [env: MELTANO_PROJECT_READONLY] current value: False (default) discovery_url [env: MELTANO_DISCOVERY_URL] current value: 'https://discovery.meltano.com/discovery.yml' (default) hub_url [env: MELTANO_HUB_URL] current value: 'https://hub.meltano.com' (default) discovery_url_auth [env: MELTANO_DISCOVERY_URL_AUTH] current value: None (default) elt.buffer_size [env: MELTANO_ELT_BUFFER_SIZE] current value: 10485760 (default) cli.log_level [env: MELTANO_CLI_LOG_LEVEL] current value: 'info' (default) cli.log_config [env: MELTANO_CLI_LOG_CONFIG] current value: 'logging.yaml' (default) ui.bind_host [env: MELTANO_UI_BIND_HOST] current value: '0.0.0.0' (default) ui.bind_port [env: MELTANO_UI_BIND_PORT] current value: 5000 (default) ui.server_name [env: MELTANO_UI_SERVER_NAME] current value: None (default) ui.session_cookie_domain [env: MELTANO_UI_SESSION_COOKIE_DOMAIN] current value: None (default) ui.session_cookie_secure [env: MELTANO_UI_SESSION_COOKIE_SECURE] current value: False (default) ui.secret_key [env: MELTANO_UI_SECRET_KEY] current value: 'thisisnotapropersecretkey' (default…

right but you're running the commands inside the container. That works for me too. But if you do this let's say docker-compose file has

...
service-local:
    <<: *service
    environment:
      <<: *service_env
    command: ["/bin/bash", "-c", "source ./envvars && meltano config meltano list && meltano ui"]
    expose:
      - 5000
    ports:
      - "5000:5000"
...

The secrets will be the ones from

./envvars

when the container starts

command: ["/bin/bash", "-c", "source ./envvars && meltano config meltano list && meltano ui"]

but then if you get into the container itself

docker exec ....

and run

meltano config meltano list

you'll see the meltano default env var values. With docker-compose testing I'm trying to mimic the k8s behavior

If you run

meltano config meltano list

and defaults are returned it means that the environment variable is not set for that shell. So however you're running it doesn't work

correct, but is the meltano process going to use the ones from the start? I guess that's my question

Now maybe there's something specific to

meltano ui

here that's causing this but if in the above

meltano config meltano list

https://meltano.slack.com/archives/C01UTUSP34M/p1655908415103819?thread_ts=1655847345.641149&cid=C01UTUSP34M command it still doesnt' get printed then there's probably something with

source ./envvars

ok. Might be an issue on the entrypoint. Will do some more debugging. Thank you for your help