I'm not sure how to handle duplicate environment variables - specifically the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. I uses these two environment variables for both S3 access via tap-spreadsheets-anywhere and target-snowflake. So if I run the follow command they both need the environmental variables set, however the tap and target require different values.

meltano elt tap-spreadsheets-anywhere target-snowflake

I use chamber to inject these keys into my Docker Container. However they ultimately end up with the same name. I need to use a separate a AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for source s3 bucket used by tap-spreadsheets-anywhere to my external s3 stage used by target-snowflake. What are the exact steps configuration to achieve this please? Please note: I don't wish to put this information in my meltano.yml config file - I want to inject it via Chamber. Thanks Steve

Definitely doable, what have you tried?

Okay from what I can see is the environment variables are being overwritten - so it is not achieving what I want. Perhaps there is a different way to achieve this? So I have set some variables in SSM Parameter Store so that Meltano can access them. One set of AWS Keys for tap-spreadsheets-anywhere and one for target-snowflake. I need to have different keys supplied because they are for different AWS IAM users.

/meltano-dev/tap-spreadsheets-anywhere/aws_access_key_id
/meltano-dev/tap-spreadsheets-anywhere/aws_secret_access_key

/meltano-dev/target-snowflake/aws_access_key_id
/meltano-dev/target-snowflake/aws_secret_access_key

To run my Meltano job with meltano this is my command line that I pass to the EC2 container.

chamber,exec,meltano-dev/tap-spreadsheets-anywhere,meltano-dev/target-snowflake,--,meltano,elt,tap-spreadsheets-anywhere,target-snowflake,--job_id,spreadsheets

Here is the result - you can see that the environment variable is being overwritten by the target-snowflake environment variables.

2021-10-26T09:27:54.102+13:00	warning: service meltano-dev/target-snowflake overwriting environment variable AWS_ACCESS_KEY_ID

2021-10-26T09:27:54.102+13:00	warning: service meltano-dev/target-snowflake overwriting environment variable AWS_SECRET_ACCESS_KEY

2021-10-26T09:28:21.818+13:00	meltano | elt | Running extract & load...

2021-10-26T09:28:21.929+13:00	meltano | elt | Found state from 2021-10-20 20:51:31.501826.

2021-10-26T09:28:24.350+13:00	tap-spreadsheets-anywhere | extractor | INFO Using supplied catalog /project/.meltano/run/elt/spreadsheets/0ee09bbc-df66-4d3b-a3b2-b18373002f17/tap.properties.json.

2021-10-26T09:28:24.350+13:00	tap-spreadsheets-anywhere | extractor | INFO Processing 0 selected streams from Catalog

Is there some alias function that I can do in the meltano.yml to help with this so the variables are not overwritten? Any help would be much appreciated. Thanks

Before AJ hops in thought I"d chime in quick. Strange, I'm not sure exactly what's going on but I thought we could debug it. I"m not versed in chamber. So I"m asuming that

chamber exec meltano-dev/tap-spreadsheets-anywhere meltano-dev/target-snowflake

sets environment variables. Like is it possible that tap-sporeadsheets-anywhere has

AWS_ACCESS_KEY_ID

and

target-snowflake

is the thing overriding this (not Meltano itself) I wonder if

AWS_ACCESS_KEY_ID

, and

AWS_SECRET_ACCESS_KEY

are being set If you could, go into your meltano project and run

meltano config target-snowflake list

this will show where Meltano is getting the configurations are coming from for

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

. You could change the commands here just for some debugging to

chamber,exec,meltano-dev/tap-spreadsheets-anywhere,meltano-dev/target-snowflake,--,meltano,config,target-snowflake,list

The

meltano config list

should be helpful in debugging as @vischpoints out. I just want to add - it looks like you want two different pairs of aws cred keys, which means you would need to inject them in two places. I didn't see any explicit config option for AWS credentials in tap-spreadsheets-anywhere so you'll probably have to use the default for that - or fork that repo to provide them explicitly in config. For target-snowflake, I'm pretty sure that target will pick up from the default environment variable names (which you are already using for the tap) so you'll want to inject those to target's relevant config options rather than as the default env variable names. (You'll actually have both set, but I'd expect the config options to have priority over the default env vars names.) So: • tap-spreadsheets-anywhere: configured via env vars: AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY • target-snowflake: configured via its respective config options (or injected to those config options with prefixed env vars: TARGET_SNOWFLAKE_... and TARGET_SNOWFLAKE_.... Which variant are you using for target-snowflake? On the target-snowflake side, you'll have to check for the exact variable names. Hope this helps, but let me know if I missed or misunderstood anything

Thank you very much @visch and @aaronsteers. 👏 The meltano config list command is very useful, I was not aware of the additional list component. With this additional information I have been able to debug and resolve the issue. So I interpreted to Snowflake Environment variables incorrectly. As you rightly said AJ. • tap-spreadsheets-anywhere: configured via env vars: AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY • target-snowflake: configured via its respective config options (or injected to those config options with prefixed env vars: TARGET_SNOWFLAKE_... and TARGET_SNOWFLAKE_.... My original configuration was this:

/meltano-dev/tap-spreadsheets-anywhere/aws_access_key_id
/meltano-dev/tap-spreadsheets-anywhere/aws_secret_access_key

/meltano-dev/target-snowflake/aws_access_key_id
/meltano-dev/target-snowflake/aws_secret_access_key

The target-snowflake configuration was incorrect 😅 it should have looked like the config below. I was missing the target_snowflake prefix.

/meltano-dev/tap-spreadsheets-anywhere/aws_access_key_id
/meltano-dev/tap-spreadsheets-anywhere/aws_secret_access_key

/meltano-dev/target-snowflake/target_snowflake_aws_access_key_id
/meltano-dev/target-snowflake/target_snowflake_aws_secret_access_key

Thanks very much again. As a side note, I really do like the use of chamber to inject different environment variables for your different environments.

@steve_clarke - I'm glad to hear you got this working! I've not used Chamber before but I have used AWS Parameter store for cred storage. I'm excited to give this a try next time I have a chance.

As a FYI, these were the extra commands that I added to my Docker file to install chamber.

# Tell Chamber to use the account default KMS key, and where to find it
ENV AWS_REGION ap-southeast-2
ENV CHAMBER_KMS_KEY_ALIAS aws/ssm

# Install chamber
ENV chamber_version=2.10.6
RUN wget <https://github.com/segmentio/chamber/releases/download/v${chamber_version}/chamber-v${chamber_version}-linux-amd64> -O /bin/chamber \
        && chmod +x /bin/chamber

Once it is installed and working it can grab my encrypted parameters from the AWS Parameter Store on my behalf. Thanks again for your assistance.

Okay, that's way easier than I thought it would be. Thanks for sharing!