hey hey, I have env variable substitution in my me...
# troubleshooting
s
hey hey, I have env variable substitution in my meltano.yml
Copy code
select:
    - ${SNOWFLAKE_DBNAME}-RECOS-*.*
but it doesn't seem to substitute that value . I am 100% sure that the value exists in evn variable. I have
Copy code
dbname: ${SNOWFLAKE_DBNAME}
which does seem to get subsituted
d
s
I also noticed something strange, maybe. Those variables only work if they match
TAPNAME_CONFIGNAME
format. For example it won't work if in dbname if the env variable name was
SNOWFLAKE_DATABASE
d
Would it be an option to use
*-RECOS-*.*
, or would that end up matching different tables?
s
Copy code
select:
    - *-RECOS-*.*
gives me
Copy code
expected alphabetic or numeric character, but found '*'
d
Ah, try wrapping it in quotes
s
that works but I also have this
Copy code
metadata:
      ${SNOWFLAKE_DBNAME}-RECOS-RECOMMENDATIONS:
        table_name: RECOMMENDATIONS 
        replication-method: FULL_TABLE
d
You should be able to use
*
there as well
I think you can wrap that key in quotes as well
s
oh nice. Let me try that
d
Those variables only work if they match 
TAPNAME_CONFIGNAME
 format. For example it won't work if in dbname if the env variable name was 
SNOWFLAKE_DATABASE
Can you share exactly what works and what doesn't, and how you're setting the env var? Is it one you set yourself or one coming from the target-snowflake config?
s
Sure thing. Maybe i am doing something wrong here but 1. works
Copy code
.env file 
TAP_SNOWFLAKE_USER=AIRFLOW

meltano.yml 
  extractors:
  - name: tap-snowflake
    namespace: smart_recommendations 
    pip_url: pipelinewise-tap-snowflake
    executable: tap-snowflake
    capabilities:
    - properties
    - catalog
    - discover
    config:
      account: <redacted>
      user: $SNOWFLAKE_USER
2. doesnt work
Copy code
.env file 
TAP_SNOWFLAKE_USER=AIRFLOW1
meltano.yml 
  extractors:
  - name: tap-snowflake
    namespace: smart_recommendations 
    pip_url: pipelinewise-tap-snowflake
    executable: tap-snowflake
    capabilities:
    - properties
    - catalog
    - discover
    config:
      account: <redacted>
      user: $SNOWFLAKE_USER1
doesn't work =
Copy code
raise Exception("Config is missing required keys: {}".format(missing_keys))
Exception: Config is missing required keys: ['user']
d
@surya_g When you're setting
TAP_SNOWFLAKE_USER
, you're directly setting the
user
setting using the "native" env var that will show up in `meltano config tap-snowflake list`: https://meltano.com/docs/configuration.html#configuring-settings When you set
user: $SNOWFLAKE_USER
or
user: $SNOWFLAKE_USER1
, you should additionally be able to configure that setting through
SNOWFLAKE_USER
or
SNOWFLAKE_USER1
since you're using env var expansion: https://meltano.com/docs/configuration.html#expansion-in-setting-values
Are you sure your second example with
TAP_SNOWFLAKE_USER=AIRFLOW1
doesn't work?
s
oops sorry missed your last question. Yes I am 99% sure it doesn't work.
d
@surya_g in the second scenario, what does “meltano config tap-snowflake list” say about the user setting?