surya_g
06/02/2021, 3:55 PMselect:
- ${SNOWFLAKE_DBNAME}-RECOS-*.*
but it doesn't seem to substitute that value . I am 100% sure that the value exists in evn variable.
I have
dbname: ${SNOWFLAKE_DBNAME}
which does seem to get subsituteddouwe_maan
06/02/2021, 3:57 PMsurya_g
06/02/2021, 3:57 PMTAPNAME_CONFIGNAME format. For example it won't work if in dbname if the env variable name was SNOWFLAKE_DATABASEdouwe_maan
06/02/2021, 3:57 PM*-RECOS-*.* , or would that end up matching different tables?surya_g
06/02/2021, 3:59 PMselect:
- *-RECOS-*.*
gives me
expected alphabetic or numeric character, but found '*'douwe_maan
06/02/2021, 3:59 PMsurya_g
06/02/2021, 4:00 PMmetadata:
${SNOWFLAKE_DBNAME}-RECOS-RECOMMENDATIONS:
table_name: RECOMMENDATIONS
replication-method: FULL_TABLEdouwe_maan
06/02/2021, 4:01 PM* there as welldouwe_maan
06/02/2021, 4:01 PMsurya_g
06/02/2021, 4:02 PMdouwe_maan
06/02/2021, 4:02 PMThose variables only work if they matchCan you share exactly what works and what doesn't, and how you're setting the env var? Is it one you set yourself or one coming from the target-snowflake config?format. For example it won't work if in dbname if the env variable name wasTAPNAME_CONFIGNAMESNOWFLAKE_DATABASE
surya_g
06/02/2021, 4:14 PM.env file
TAP_SNOWFLAKE_USER=AIRFLOW
meltano.yml
extractors:
- name: tap-snowflake
namespace: smart_recommendations
pip_url: pipelinewise-tap-snowflake
executable: tap-snowflake
capabilities:
- properties
- catalog
- discover
config:
account: <redacted>
user: $SNOWFLAKE_USERsurya_g
06/02/2021, 4:14 PM.env file
TAP_SNOWFLAKE_USER=AIRFLOW1
meltano.yml
extractors:
- name: tap-snowflake
namespace: smart_recommendations
pip_url: pipelinewise-tap-snowflake
executable: tap-snowflake
capabilities:
- properties
- catalog
- discover
config:
account: <redacted>
user: $SNOWFLAKE_USER1surya_g
06/02/2021, 4:15 PMraise Exception("Config is missing required keys: {}".format(missing_keys))
Exception: Config is missing required keys: ['user']douwe_maan
06/02/2021, 4:23 PMTAP_SNOWFLAKE_USER , you're directly setting the user setting using the "native" env var that will show up in `meltano config tap-snowflake list`: https://meltano.com/docs/configuration.html#configuring-settings
When you set user: $SNOWFLAKE_USER or user: $SNOWFLAKE_USER1 , you should additionally be able to configure that setting through SNOWFLAKE_USER or SNOWFLAKE_USER1 since you're using env var expansion: https://meltano.com/docs/configuration.html#expansion-in-setting-valuesdouwe_maan
06/02/2021, 4:24 PMTAP_SNOWFLAKE_USER=AIRFLOW1 doesn't work?surya_g
06/03/2021, 1:52 AMdouwe_maan
06/03/2021, 5:33 PM