Hi all, When trying to read my username and password from env vars instead of

meltano.yml

, my code is not able to read them. Do I need to explicitly read from

os.environ

or is it handled by default in the SDK ? My code is breaking at the point where I am forming the sql_alchemy connection string / validating input parameters.

url = URL.create(
            drivername='redshift+redshift_connector',  # indicate redshift_connector driver and dialect will be used
            host=self.config['host'],  # Amazon Redshift host
            port=self.config['port'],  # Amazon Redshift port
            database=self.config['database'],  # Amazon Redshift database
            username=self.config['user'],  # Amazon Redshift username
            password=self.config['password']  # Amazon Redshift password
        )

It tries to find user and pass in

self.config

and gives

KeyError

. My env vars are

TAP_REDSHIFT_USER

and `TAP_REDSHIFT_PASSWORD`and they have been specified in my

meltano.yml

like this:

default_environment: test
environments:
- name: test
  env:
    TAP_REDSHIFT_USER: foo
    TAP_REDSHIFT_PASSWORD: bar

Do I have to map these keys with env vars somewhere? I tried to find it in existing implementations but couldn’t find.

Do I need to explicitly read from os.environ or is it handled by default in the SDK ?

If you are using Meltano to invoke the plugin (i.e.

meltano invoke

), you don't need to do this - Meltano generates a temporary

config.json

which it passes to the plugin at runtime. You can actually see this in the logs if running with `--log-level devug`:

reuben@reuben-Inspiron-14-5425:~/Documents/taps/tap-spotify$ meltano --log-level debug invoke tap-spotify

...

2023-07-06T14:24:27.685623Z [debug    ] Created configuration at /home/reuben/Documents/taps/tap-spotify/.meltano/run/tap-spotify/tap.0610648a-4f84-4dad-84a7-6bbad1d4f6e1.config.json

...

2023-07-06T14:24:28.010338Z [debug    ] Deleted configuration at /home/reuben/Documents/taps/tap-spotify/.meltano/run/tap-spotify/tap.0610648a-4f84-4dad-84a7-6bbad1d4f6e1.config.json

You don't need to specify plugin configuration as environment variables in the

meltano.yml

either (in fact, I think you might be circumventing some default behaviour here unintentionally). Meltano will automatically interpret environment variables in the format

<PLUGIN_NAME>_<SETTING_NAME>=<value>

as plugin configuration, from your shell environment or a

.env

file at the root of the project (more info here). In your case, I would create a

.env

file with

TAP_REDSHIFT_USER=foo
TAP_REDSHIFT_PASSWORD=bar

and then

meltano invoke tap-redshift

or

meltano config tap-redshift test

Thanks for the explanation @Reuben (Matatika), that makes sense 🙂 But I am still getting

KeyError: 'user'

(refer earlier code segment) after running

meltano invoke tap-redshift

. I removed all the env related keys in my

meltano.yml

and putting them in

.env

file of my project root.

Have you defined

settings

for

tap-redshift

under

plugins

?

version: 1
send_anonymous_usage_stats: true
project_id: tap-spotify
default_environment: test
environments:
- name: test
plugins:
  extractors:
  - name: tap-spotify
    namespace: tap_spotify
    pip_url: -e .
    capabilities:
    - state
    - catalog
    - discover
    settings:
    - name: client_id
      kind: password
    - name: client_secret
      kind: password
    - name: refresh_token
      kind: password
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl

Thanks @Reuben (Matatika), that was it 🙂 You’re a life saver.

Nice! Good luck with the rest! 😄