christoph
03/10/2022, 9:37 AMREQUESTS_CA_BUNDLE configured in my developer OS (needed for allowing a custom Root CA certificate for Security filtering), which works great every time that I use the requests library in python and make HTTPS calls to the Internet.
Now I have just found that this seems to not work when launching meltano elt and the tap (custom tap built with the Meltano SDK) uses the Python requests module. The requests module is seemingly ignoring the REQUEST_CA_BUNDLE OS environment variable when running meltano elt and the custom tap fails with an exception caused by the requests module (requests.exceptions.SSLError: (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate)
I was hoping I can configure meltano to somehow tell it to use the REQUESTS_CA_BUNDLE environment variable and pass it to the virtualenv when the plugin invoker launches the tap. Am I on the right path?
UPDATE
A bit of print debugging seems to indicate that I am on the wrong path with the OS environment variables. the REQUESTS_CA_BUNDLE OS env variable value does make it all the way into the RESTStream class in the singer-sdk .... It must be something else afoot ...
UPDATE 2
Oh. Turns out I was on the right track indeed!
There is a requests API that handles environment variables when using prepared requests in requests. Looks like by default, the behaviour of prepared requests (like the singer sdk uses it in the RESTStream) is different from "normal" requests in requests.
https://github.com/psf/requests/issues/1436