I’m having trouble using environment variables to ...
# troubleshooting
d
I’m having trouble using environment variables to configure a loader.
Copy code
loaders:
  - name: target-snowflake
    pip_url: pipelinewise-target-snowflake==1.12.0
    capabilities:
    - catalog
    - discover
    - state
    config:
      file_format: FILE_FORMATS.CSV
      user: MY_USER
      role: MY_ROLE
      default_target_schema: MY_SCHEMA
In my
.env
I have
TARGET_SNOWFLAKE_DBNAME="MELTANO"
. When I run an
elt
I get a configuration error complaining that it’s not set:
Copy code
$  meltano --log-level=info elt --job_id=wip tap-postgres target-snowflake
...
target-snowflake |    * Required key is missing from config: [dbname]
meltano          | Loading failed (1):    * Required key is missing from config: [dbname]
I think I’m using the expected variable name according to the docs: https://github.com/transferwise/pipelinewise-target-snowflake#to-run-tests
e
Did you make sure to
source .env
before you ran the command?
d
Doesn’t help unfortunately
d
@dean_morin Can you add
variant: transferwise
to the definition?
d
Let me give that a try…
That works @douwe_maan I’ve added it as a custom plugin rather than a variant though because of this issue: https://gitlab.com/meltano/meltano/-/issues/2741 Is there something I need to do differently to configure a custom plugin?
d
@dean_morin The only thing to be aware of is that Meltano won't look for settings in env vars until it knows the setting exists: https://meltano.slack.com/archives/CFG3C3C66/p1615304264116800?thread_ts=1615297554.110300&cid=CFG3C3C66
@dean_morin So in your case,
TARGET_SNOWFLAKE_DBNAME
won't be picked up unless you define
dbname: null
under
config
, or
- name: dbname
under
settings
If you added the custom plugin using
meltano add --custom loader target-snowflake
, it would've asked you to provide the names of all the settings. If you had listed
dbname
there, it would've worked as expected
Note that if you run
meltano config target-snowflake list
it won't currently show
dbname
, or
TARGET_SNOWFLAKE_DBNAME
d
That worked, thanks for the help!