https://linen.dev logo
#plugins-general
Title
# plugins-general
c

christoph

03/10/2022, 9:37 AM
This should hopefully be easy .. How do I pass an environment variable from my OS shell to an extractor tap? I have
REQUESTS_CA_BUNDLE
configured in my developer OS (needed for allowing a custom Root CA certificate for Security filtering), which works great every time that I use the
requests
library in python and make HTTPS calls to the Internet. Now I have just found that this seems to not work when launching
meltano elt
and the tap (custom tap built with the Meltano SDK) uses the Python
requests
module. The
requests
module is seemingly ignoring the
REQUEST_CA_BUNDLE
OS environment variable when running
meltano elt
and the custom tap fails with an exception caused by the
requests
module (
requests.exceptions.SSLError: (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
) I was hoping I can configure meltano to somehow tell it to use the
REQUESTS_CA_BUNDLE
environment variable and pass it to the virtualenv when the plugin invoker launches the tap. Am I on the right path? UPDATE A bit of
print
debugging seems to indicate that I am on the wrong path with the OS environment variables. the
REQUESTS_CA_BUNDLE
OS env variable value does make it all the way into the
RESTStream
class in the singer-sdk .... It must be something else afoot ... UPDATE 2 Oh. Turns out I was on the right track indeed! There is a
requests
API that handles environment variables when using prepared requests in
requests
. Looks like by default, the behaviour of prepared requests (like the singer sdk uses it in the RESTStream) is different from "normal" requests in
requests
. https://github.com/psf/requests/issues/1436