https://linen.dev logo
Join Slack
Powered by
If I have two variants of the same plugin, and the...
# troubleshooting
a
If I have two variants of the same plugin, and they both need the same env variable name, but with different values, what are my options? I need two variants of tap-spreadsheets-anywhere, accessing data from Azure storage in two separate storage accounts. This is normally achieved by setting 
AZURE_STORAGE_CONNECTION_STRING
env variable, but this conn string will be different for each account. Following the docs here (https://docs.meltano.com/guide/configuration/#specifying-environment-variables) I tried: .env
Copy code
BUCKET_A_CONN_STRING=abc123
and then
Copy code
- name: tap-myfeeda
    inherit_from: tap-spreadsheets-anywhere
    env:
      AZURE_STORAGE_CONNECTION_STRING: ${BUCKET_A_CONN_STRING}
but just received errors suggesting no connection (it definitely works previously). It's not clear from the docs if I can reference ANOTHER env variable in the 
env:
section. Other issues: I'm not invoking via cli so I can't inject any extra variables at runtime. Needs to be achievable with a static set of container-level env variables and meltano.yml
I think it's something to do with my tap being inherited from a base tap, if I define the 
env:
on the base things seem to work better
I think I figured a way of working, but I needed to have my variable declared at the env: level too to get it it to pull down to my plugin So if i have .env
Copy code
MY_CONN=abc
Then this works:
Copy code
environments:
- name: dev
  env:
    MY_CONN: $MY_CONN
Copy code
plugins:
  extractors:
  - name: my-tap
    env:
      AZURE_STORAGE_CONNECTION_STRING: $MY_CONN
But if I leave out the definition in 
environments:
then 
AZURE_STORAGE_CONNECTION_STRING
just appears as an empty string when I try to invoke. Is this a bug?
This is with meltano 2.20, haven't tried with 3.x yet but not sure this is a new feature.
v
Is the the azure connection string env purposely not put in the env block?
a
I need to vary the string with different inherited plugins, so if I put it in the env block directly, it will have the same value for all plugins.
Basically what I want for end state is:
Copy code
plugins:
  extractors:
  - name: my-tap-1
    env:
      AZURE_STORAGE_CONNECTION_STRING: $MY_CONN_1

  - name: my-tap-2
    env:
      AZURE_STORAGE_CONNECTION_STRING: $MY_CONN_2
h
Yeah I remember that one… the tap checks directly for an env variable because it would take a lot of rewriting to use a config. As far as I know Meltano isn’t able to set custom env variables for each run, but is it possible to overload the environment mechanism so that you have two different environments with different configs? Or you could run it in prefect/dagster etc and set the env from there.
v
I set them for each run I'm fairly certain, Just did this last week almost the same setup
There is some oddities in how env vars expand as certain levels they'll expand envs and certain they won't https://github.com/meltano/meltano/issues/6991
a
Cheers both I'll take a look
I did get it working, but I would need all the vars defined at the 
environment:
level first, then at plugin level, otherwise I just get empty strings. I'm fine with that
v
Yeah it sounds super similar to my setup, maybe you'll find another setup that works for you from that issue. Sorry for the pain you just went through, I feel you!
p
I ran into this too, today, got it work like how you mentioned here:
I did get it working, but I would need all the vars defined at the 
environment:
level first, then at plugin level, otherwise I just get empty strings. I'm fine with that
v
Can you both go thumbsup / like the issue https://github.com/meltano/meltano/issues/6991
Open in Slack
PreviousNext
Powered by