With the S3 state backend is access keys and shared credenti Meltano #troubleshooting

Join Slack

With the S3 state backend is access keys and share...

# troubleshooting

lukas_gust

05/04/2023, 9:33 PM

With the S3 state backend is access keys and shared credentials file the only way to fetch creds to s3?

edgar_ramirez_mondragon

05/04/2023, 10:17 PM

Hi Lukas. What other options did you have in mind?

lukas_gust

05/04/2023, 10:23 PM

We are currently using the legacy sso which does not have a shared credentials file, this is for local development. e.g. we'd use the following when running our images locally

docker run -v $HOME/.aws:/root/.aws:ro ...

Then our meltano project would run in a k8s cluster pod that has access via an IAM role

lukas_gust

05/04/2023, 10:25 PM

The first would use the profile I'm signed in with on my local machine or the default

lukas_gust

05/04/2023, 10:38 PM

In theory the code appears as though it should work, but maybe I'm missing something. https://github.com/meltano/meltano/blob/61c2d34642a1e02156d484a05eb3b9deb2f1a297/src/meltano/core/state_store/s3.py

lukas_gust

05/04/2023, 10:52 PM

Copy code

2023-05-04T21:47:09.005728Z [info     ] Writing state to AWS S3                 
2023-05-04T21:47:09.951054Z [warning  ] Unable to persist state, or received state is invalid, incremental state has not been updated
2023-05-04T21:47:09.973133Z [info     ] Block run completed.           block_type=ExtractLoadBlocks err=None set_number=0 success=True

This is after a --full-refresh

lukas_gust

05/05/2023, 4:25 AM

Running Meltano with debug output shows that it's grabbing the creds as I expected. And it's grabbing the default sso profile, turned out I had it pointed to the wrong account. So i suppose it'd be nice if we were able to optionally pass in a profile name for the state backend? Unless this is already possiblem just not documented?

ptd

05/05/2023, 1:21 PM

assuming you are hitting the

boto3.Session()

case, it should work via the credentials file, iam role, or environment variables. If you want to select your non default profile you can set

AWS_PROFILE

AWS_DEFAULT_PROFILE

in your environment and it should work

lukas_gust

05/05/2023, 4:11 PM

Yup that's my assumption as well, docs just confused me, and then it didn't work because of my above reason lol. Thank you!

Open in Slack

Previous Next