Anyone ever have this issue? It looks like something to do with being on a corp machine, but wondered if there is a way around it? Im trying to simply an extractor

HTTPSConnectionPool(host='<http://hub.meltano.com|hub.meltano.com>', port=443): Max retries exceeded with url: /meltano/api/v1/plugins/loaders/index (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')))

I have had that issue before (not with Meltano but with installing Python libraries with pip) and you're right that probably is your corporate firewall. IIRC, I was able to get around it by installing software via WSL2.

That's what I'm working from. What kind of software do you mean?

I do all of my work development via WSL2 so I install Python there, libraries there, databases there, etc. rather than installing the Windows versions.

That's what I did as well, and I have another virtual environment on the same machine that doesn't have this issue for some reason

Logged: https://github.com/meltano/meltano/issues/6334

Thanks @aaronsteers !

@greg_vaslo On Linux it's relatively easy to manage custom Root CA Cert Stores for Python

request

via the

REQUESTS_CA_BUNDLE

environment variable. Depending on your Linux distribution, your custom Root CA Cert store will be in different locations, but basically, as long as you set the

REQUESTS_CA_BUNDLE

to the path that contains to the PEM file (e.g.

/etc/ssl/certs/ca-certificates.crt

on Debian/Ubuntu which is managed by

update-ca-certificates

to merge your internal custom Enterprise Root CAs) that has your complete customized Root CA bundle, then at least the majority of Python

requests

calls will work (Meltano SDK plugins won't work though, until we can do some more testing and work on this PR: https://github.com/meltano/sdk/issues/633) With Python on Windows, the

REQUESTS_CA_BUNDLE

variable will still work, but on Windows you don't get the luxury of having the OS provide you a convenient file location where Windows would store the custom Root CA Cert store bundle for you in PEM format. You will need to generate such a file on Windows yourself (which is possible with

certutil

to dump your custom Windows Root CA Cert stores into a PEM formatted file)

@christoph First, thanks for your feedback. You say "On Linux it's relatively easy to manage custom Root CA Cert Stores for Python" but I don't really understand anything you are talking about 🙂. For the past two months I've been using Meltano at work without this issue. I have tried a number of taps and targets pretty easily. I mentioned that I won't be able to get any assistance on certificates from my corporation. Before I start trying to do this, do I need to get something from my IT team? If so I guess this is a dealbreaker for me.

@greg_vaslo Unfortunately, your situation is VERY common in the corporate world. And for some reason, 99% of IT departments won't even understand what I'm talking about, because many IT people think that the topic of Public Key Cryptography is some sort of magic, due to the complex nature of this technology and how it is used on the Internet. Even some of the best video explanations of this topic, will take a few repeat viewings for the uninitiated person to understand what is going on () The summary is, your IT department is acting as an "intercepting" agent for all Internet traffic on your corporate desktop/laptop. This kind of "intercepting" activity is sometimes also referred to as "Man in the Middle (MITM)". Such MITM scenarios will ALWAYS result in a break of the "chain of trust". Once the "chain of trust" for secure Internet communication is broken, all your Internet software will fail. Your IT department will likely have introduced a piece of software or hardware somewhere in their realm which is acting as such "MITM" or "intercepting" agent. IT departments do this for many reason, most of the time they do this for regulatory compliance or in order to achieve an actual tangible improvement of the organisation's cybersecurity posture. When your IT department did this, they would have made sure that all the "stuff" (software, programs, etc ...) which are under their ownership still works and that the broken "chain of trust" is manually mended again. (The "mending" of the broken "chain of trust" is what the Root CA Cert Store is involved in). But anything that your IT department doesn't own (e.g. Python or Meltano software), they most likely don't care and/or don't know about, so it wasn't included in their scope of mending the "chain of trust". You have two options: • Fix it yourself (happy to guide you on the technical steps of doing that) • Ask your IT department to support Python on Windows or Python on WSL as part of the service they provide to the business (it's a long and painful process a lot of times, because of the first paragraph - 99% of IT departments will have no clue how to mend the "chain of trust" for Python in this scenario)

Would you be able to get a cloud VM and run Meltano on it?

I'd kill for a VM on our Cloud for lots of reasons but no one seems to like non-IT people (I'm Finance) getting them, which is silly.

Hi @greg_vaslo So basically, the first step in this process is to do a bit of discovery in order to find out what exactly the 'intercepting agent' is which is inspecting your Internet traffic. There typically are a few different general types of mechanism of how such interception is installed by the IT department: 1. Transparent (Network-based) Interception - This means somewhere on the network path between the computer that you control and the Internet, there is another network device which receives all the Internet bound traffic and inspects it (this could be something like a Network Firewall device or some Data Loss Prevention Device or a Intrusion Prevention Device) 2. Host-based Interception - The computer that you control might have some additional agent software installed (e.g. Zero Trust Network Agents, Data Loss Prevention Agent, etc.) which is doing the intercepting. 3. Plain old HTTP Proxy Configuration - The computer that you control has a custom configuration setting applied (HTTP Proxy configuration) which results in all Internet bound traffic to be sent to a HTTP Proxy server somewhere (could be a HTTP Proxy Server inside the same network or in a completely different network) Whilst knowing which kind of interception is deployed will not make a huge difference in the next steps, having this information counts as good environment knowledge and might come in handy in later stages. Now, here is the magic command to get a first clue about what is going on with your system. This

curl

command will simply make a connection to the Meltano Plugin Inventory document and the command will spit out a lot of useful verbose logging information (

-v

) which will also include a section about the

Server Certificate

.

curl -s -o /dev/null -v <https://discovery.meltano.com/discovery.yml>

Just run this above command on the same system where you are experiencing the meltano problem. The out put will include some lines that show the

Server Certificate

like this:

* Server certificate:
*  subject: CN=<http://discovery.meltano.com|discovery.meltano.com>
*  start date: Jun 29 18:33:00 2022 GMT
*  expire date: Oct 28 21:59:54 2022 GMT
*  subjectAltName: host "<http://discovery.meltano.com|discovery.meltano.com>" matched cert's "<http://discovery.meltano.com|discovery.meltano.com>"
*  issuer: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; OU=Gateway Intermediate ECC Certificate Authority
*  SSL certificate verify ok.

In this example output from my laptop, the important line is this one:

issuer: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; OU=Gateway Intermediate ECC Certificate Authority

In my example, I have a Zero Trust Network Agent (from Cloudflare) installed on my computer which does the intercepting, which is why the issuer of the SSL Server Certificate for

<http://discovery.meltano.com|discovery.meltano.com>

shows up as CloudFlare Inc. (Gateway Intermediate ECC Certificate Authority), which is actually not the Issuing Authority which Meltano has used to install the real SSL Server Certificate on

<http://discovery.meltano.com|discovery.meltano.com>

(The real issuer of the discovery.meltano.com SSL Certificate is acually "Let's Encrypt") Armed with this knowledge, you should be able to find the root cause of why the SSL connection is failing on your computer, by just running that curl command and looking at the

issuer

line in the

Server Certificate

section. That information will then help us correcting the issue on your computer in the next installment of this series. 😄

Thanks for that detail - I didn't get that nice certificate info like you did but did get the following output:

*   Trying 34.204.131.44:443...
* TCP_NODELAY set
* Connected to <http://discovery.meltano.com|discovery.meltano.com> (34.204.131.44) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [88 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [155 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [6 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3319 bytes data]
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
} [2 bytes data]
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0

That's good news. At least we now can reproduce the problem (

SSL certificate problem: self signed certificate in certificate chain

) with just a simple CURL command. What you can do now to proceed, is to tell

curl

to just ignore this SSL certificate problem and then curl will just act like the broken SSL certificate chain is not a problem. You can use the

-k

switch in curl for that. So, the command will look like this:

curl -k -s -o /dev/null -v <https://discovery.meltano.com/discovery.yml>

That should now give you the

Server Certificate

section in the verbose log output

Ok gotcha, so this one:

Server certificate:
*  subject: CN=<http://discovery.meltano.com|discovery.meltano.com>
*  start date: Jun 27 00:00:00 2022 GMT
*  expire date: Sep  7 22:00:20 2022 GMT
*  issuer: C=US; ST=Ohio; L=Cleveland; O=The Sherwin-Williams Company; OU=Information Security; CN=SW ZScaler
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.

Great. There's the intercepting agent (ZScaler, which is a Cloud-based Web Security software aka Web Proxy). Armed with that knowledge, we can make a safe assumption that this is a "Transparent Network Interception", which makes the next step easier. We can now pull out

openssl

in order to dump out the full SSL certificate chain contents from the ZScaler cloud instance, by simply repeating the same SSL handshake which

curl

performs from

openssl

.

openssl s_client -showcerts -connect <http://discovery.meltano.com:443|discovery.meltano.com:443>

That will spit out the BASE64 encoded SSL Server certificate chain which gets injected by the ZScaler cloud instance, which we will then use to configure the Operating System to trust your company's ZScaler self-signed Root CA certificate. Here's what the output looks like for me via the my Cloudflare Zero Trust Network agent doing the SSL interception: ```CONNECTED(00000003) depth=2 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc", CN = Cloudflare for Teams ECC Certificate Authority verify return:1 depth=1 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", OU = Gateway Intermediate ECC Certificate Authority verify return:1 depth=0 CN = discovery.meltano.com verify return:1 --- Certificate chain 0 s:CN = discovery.meltano.com i:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", OU = Gateway Intermediate ECC Certificate Authority -----BEGIN CERTIFICATE----- MIICOTCCAb6gAwIBAgIEBoyRpjAKBggqhkjOPQQDAjCBjjELMAkGA1UEBhMCVVMx EzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28xGTAX BgNVBAoTEENsb3VkZmxhcmUsIEluYy4xNzA1BgNVBAsTLkdhdGV3YXkgSW50ZXJt ZWRpYXRlIEVDQyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMjIwNjI5MTgzMzAw WhcNMjIxMDAxMjI1NzUzWjAgMR4wHAYDVQQDDBVkaXNjb3ZlcnkubWVsdGFuby5j b20wdjAQBgcqhkjOPQIBBgUrgQQAIgNiAARmjFVqM8V6B+26EXYHliDSChpn+nU5 4RmKQZCexvGorafKKRdJOd9/+lLX6aOpAfiMyE+FABBrPcx6jFIAOZi9ZIgOvKBf r9CQt8VInKET49Gt2ybZbzNjYGfiw0QgC/ejWjBYMBMGA1UdJQQMMAoGCCsGAQUF BwMBMCAGA1UdEQQZMBeCFWRpc2NvdmVyeS5tZWx0YW5vLmNvbTAfBgNVHSMEGDAW gBQbxiHdLaggqDajrTc7SEF0dJwYdjAKBggqhkjOPQQDAgNpADBmAjEAm/9t7KiT /xN2tVfJEBfqFOyBZiMMVzfYP/63PTClhWoPQHr56ciJMNCab9oyVVCkAjEA5Ow2 kPOHqrVSSJ9LMy6CMs5hXG6vJ6GU1CMOgAgMICi6g9cKyppnkDNAq7f4n6e3 -----END CERTIFICATE----- 1 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", OU = Gateway Intermediate ECC Certificate Authority i:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc", CN = Cloudflare for Teams ECC Certificate Authority -----BEGIN CERTIFICATE----- MIIC5zCCAkigAwIBAgIUc0Gjb2Mo7K3PbHHgeggfxKoJjjUwCgYIKoZIzj0EAwQw gY0xCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1T YW4gRnJhbmNpc2NvMRgwFgYDVQQKEw9DbG91ZGZsYXJlLCBJbmMxNzA1BgNVBAMT LkNsb3VkZmxhcmUgZm9yIFRlYW1zIEVDQyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkw HhcNMjIwNjI5MTgzMzAwWhcNMjIwODAzMTgzMzAwWjCBjjELMAkGA1UEBhMCVVMx EzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcTDVNhbiBGcmFuY2lzY28xGTAX BgNVBAoTEENsb3VkZmxhcmUsIEluYy4xNzA1BgNVBAsTLkdhdGV3YXkgSW50ZXJt ZWRpYXRlIEVDQyBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwdjAQBgcqhkjOPQIBBgUr gQQAIgNiAARzUHsD4WQ4MDWrrwkDp9iB3twGOrxKnPb92F4Ag/RdTQEHAGU3ercc mYOqXFCwvFBZSmTR3TrK5MebUnifnPBlArr6p1pUvaTfJp2QH/4CEpQyYYpjZguc TRaaU78W42SjZjBkMA4GA1UdDwEB/wQEAwICBDASBgNVHRMBAf8ECDAGAQH/AgEA MB0GA1UdDgQWBBQbxiHdLaggqDajrTc7SEF0dJwYdjAfBgNVHSMEGDAWgBRgFxCe DUAQww2165EMAwbT++AV7TAKBggqhkjOPQQDBAOBjAAwgYgCQgF2OQ3ksKsmUlH4 3axLy9ITlRBOqt8el1SYw6oe73R4n1p1Jx+k5OTLiMJh3jpPxrOErLLvEmD1EEyq ezUPhA6sNgJCAcOAIkmOANnT36iEBxxn8Di4XkbdXE6s5E+wSKSHriLNyacM5BPv oJzrVrQO80sUCXFcbWx7NQN+Oyn8SVPDQHiY -----END CERTIFICATE----- --- Server certificate subject=CN = discovery.meltano.com issuer=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", OU = Gateway Intermediate ECC Certificate Authority --- No client certificate CA names sent Peer signing digest: SHA384 Peer signature type: ECDSA Server Temp Key: X25…

I got the output just like you suggested with the openssl command. Should I post that? I only ask because I don't always know what's private and not. I do have a lot of SSH logins so I do recognize this to some degree. As for operating systems, Im on Windows 10, specifically using WSL2 Ubuntu in Windows to do my work

The output does not contain any private keys. The keys are all public keys. The public key you want to look for is the bottom most key. It should have identical

s:

and

i:

lines, thereby indicated a self-signed certificate, i.e. it is the 'Root CA Certificate' that is used by the ZScaler software (which is probably actually a software agent running on your Windows 10 machine in addition to the ZScaler Cloud Control plane, but that's just a minor implementation detail at this stage.) The BASE64 encoded public key of the ZScaler Root CA certificate now needs to be stored as a

.crt

file in the

/usr/local/share/ca-certificates

folder in your Ubuntu WSL2 instance. You can name the file whatever you want, but the file extension MUST be

.crt

, otherwise Ubuntu will not accept it as a certificate public key file. And as a final step, after you created the

.crt

file in

/usr/local/share/ca-certificates

with the BASE64 encoded public key from the openssl output, you'll need to run the magical command to actually install this new additional custom Root CA certificate in Ubuntu, which is:

sudo update-ca-certificates

The output will look like this:

Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
rehash: warning: skipping Cloudflare_CA.pem,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

With the

1 added, 0 removed; done.

line indicating the success of the installation. After that, you can run the original first

curl

command WITHOUT the

-k

switch again, and now

curl

should be able to validate the SSL Certificate chain successfully!

curl -s -o /dev/null -v <https://discovery.meltano.com/discovery.yml>

If this

curl

command works, you are now ready to install meltano plugins! 😄 All you have to do to make meltano's network library (Python

requests

library) aware of the customized Root CA configuration is to point the

requests

library to the correct location of the Ubuntu CA Certificate store (which got updated by the

update-ca-certificates

command). To do so, you'll need to use the environment variable

REQUESTS_CA_BUNDLE

like so:

export REQUESTS_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"

After that, you can run

meltano install

and the plugin downloads should work.

@christoph I have this whole other job in Finance with fiscal close that I had to do so had to step away to do for a few days but your directions were awesome and I was able to get this to work!!! I appreciate you spending so much time with me and it's just all the more reason to continue to build my skillset with Meltano. You are a great human being and if I could buy you a beer, I would. Thanks again.

Thanks @greg_vaslo Really appreciated. I always feel like there is not many people out there who understand this particular topic, so I'm more than happy to chime in and explain it in detail whenever it comes up. Glad it all worked for you. (I used to build and support Windows 10 WSL2 development environments for hundreds of offshore developers at a Telco in a past life, so this particular SSL scenario is something I had to deal with over and over again) NOTE: Any Meltano-SDK based taps that pull data over the Internet, will not automatically adhere to the

REQUESTS_CA_BUNDLE

setting. The old PR/MR is here, but I haven't had a chance to revisit it after learning more about the SDK behaviour regarding this issue in the wild. https://gitlab.com/meltano/sdk/-/merge_requests/256

I'm OOO as of now, but have kept an eye on this thread. Huge thankyou, @christoph 🏋️ , for the support and @greg_vaslo, I'm thrilled to hear you've gotten it working. 🎉

@christoph Ugh sorry to bug you again and I understand if it's too much time on this, but for some reason the issue returned. I checked the certificate and it didn't change, and I even did some work to remove it and re-add it but I don't know why this happened - any quick ideas?

Hi @greg_vaslo The most likely cause is that you might now be running into underlying issues with how the Python

requests

library is being called in certain taps. Which

meltano

command are you issuing when you get the error?