Hello everyone We re trying to set the state backend to s3 b Meltano #getting-started

Hello everyone! We’re trying to set the state back...

denis_gribanov

07/21/2023, 3:17 PM

Hello everyone! We’re trying to set the state backend to s3, but couldn’t figure out how to provide an endpoint for the connection. We see such features in logs by running pipeline with

--log-level=debug

option:

Copy code

2023-07-21T15:10:10.081783Z [debug    ] Looking for endpoint for s3 via: environment_global
2023-07-21T15:10:10.081891Z [debug    ] Looking for endpoint for s3 via: config_service
2023-07-21T15:10:10.081994Z [debug    ] Looking for endpoint for s3 via: config_global
2023-07-21T15:10:10.082107Z [debug    ] No configured endpoint found.

Also you can notice the variable for an endpoint in Github: https://github.com/meltano/meltano/blob/main/integration/meltano-state-s3/validate.sh Has anyone used this option? How do we provide it to Meltano?

edgar_ramirez_mondragon

07/21/2023, 4:41 PM

Hi Denis! Have you seen https://docs.meltano.com/reference/settings#state_backends3endpoint_url?

denis_gribanov

07/21/2023, 4:44 PM

Missed this setting, thank you!

denis_gribanov

07/21/2023, 6:16 PM

I’ve tried to use the

MELTANO_STATE_BACKEND_S3_ENDPOINT_URL

variable to specify the endpoint, but it’s no luck— it falls back to the default aws endpoint:

Copy code

2023-07-21T18:09:26.558534Z [debug    ] Looking for endpoint for s3 via: environment_service
2023-07-21T18:09:26.558688Z [debug    ] Looking for endpoint for s3 via: environment_global
2023-07-21T18:09:26.558844Z [debug    ] Looking for endpoint for s3 via: config_service
2023-07-21T18:09:26.558945Z [debug    ] Looking for endpoint for s3 via: config_global
2023-07-21T18:09:26.559068Z [debug    ] No configured endpoint found.

Though

MELTANO_STATE_BACKEND_S3_ENDPOINT_URL

persists in the.env file as well as a global variable. Am I missing something? Could you please point me in the right direction? We’re using Meltano 2.20.0 with tap-postgres and target-postgres (both meltanolabs variant)

edgar_ramirez_mondragon

07/21/2023, 6:24 PM

Can you confirm that the right value is detected by Meltano?

Copy code

$ meltano config meltano list | grep ENDPOINT_URL 
state_backend.s3.endpoint_url [env: MELTANO_STATE_BACKEND_S3_ENDPOINT_URL] current value: None (default)

denis_gribanov

07/21/2023, 6:28 PM

Yes, this is output:

Copy code

meltano config meltano list | grep ENDPOINT_URL
2023-07-21T18:27:04.153719Z [info     ] Environment 'prod' is active
state_backend.s3.endpoint_url [env: MELTANO_STATE_BACKEND_S3_ENDPOINT_URL] current value: '<https://custom_provider.com>' (from the MELTANO_STATE_BACKEND_S3_ENDPOINT_URL variable in the environment)

Though credentials can be found in the environment:

Copy code

2023-07-21T18:30:36.370586Z [debug    ] Looking for credentials via: env
2023-07-21T18:30:36.370766Z [info     ] Found credentials in environment variables.

edgar_ramirez_mondragon

07/21/2023, 7:19 PM

Hmm I can't reproduce the behavior.

MELTANO_STATE_BACKEND_S3_ENDPOINT_URL

is correctly used when I set it in

.env

(I do get an "Invalid endpoint: https://custom_provider.com" error). Are you running all commands in the same shell?

denis_gribanov

07/21/2023, 7:35 PM

Yes, all commands were in the same shell. This is strange, perhaps I will try to recreate Meltano project and try once again, thank you for your help!

edgar_ramirez_mondragon

07/21/2023, 7:57 PM

Np! Do let me know if you find anything.

ian_lewis

07/24/2023, 8:56 AM

@denis_gribanov I have set

MELTANO_STATE_BACKEND_URI='<s3://my-data-lake/>'

This works for me.

denis_gribanov

07/24/2023, 8:59 AM

@ian_lewis Thank you for the updates! It was a dependency issue - I tried installing Meltano on another instance, and it now works like a charm.

ian_lewis

07/24/2023, 9:24 AM

Great!

Open in Slack

Previous Next