christoph
03/10/2022, 9:37 AMREQUESTS_CA_BUNDLE
configured in my developer OS (needed for allowing a custom Root CA certificate for Security filtering), which works great every time that I use the requests
library in python and make HTTPS calls to the Internet.
Now I have just found that this seems to not work when launching meltano elt
and the tap (custom tap built with the Meltano SDK) uses the Python requests
module. The requests
module is seemingly ignoring the REQUEST_CA_BUNDLE
OS environment variable when running meltano elt
and the custom tap fails with an exception caused by the requests
module (requests.exceptions.SSLError: (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
)
I was hoping I can configure meltano to somehow tell it to use the REQUESTS_CA_BUNDLE
environment variable and pass it to the virtualenv when the plugin invoker launches the tap. Am I on the right path?
UPDATE
A bit of print
debugging seems to indicate that I am on the wrong path with the OS environment variables. the REQUESTS_CA_BUNDLE
OS env variable value does make it all the way into the RESTStream
class in the singer-sdk .... It must be something else afoot ...
UPDATE 2
Oh. Turns out I was on the right track indeed!
There is a requests
API that handles environment variables when using prepared requests in requests
. Looks like by default, the behaviour of prepared requests (like the singer sdk uses it in the RESTStream) is different from "normal" requests in requests
.
https://github.com/psf/requests/issues/1436