Troubleshooting cURL "error 60: SSL certificate problem: unable to get local issuer certificate" in vSphere Integrated Containers

Products

VMware vSphere ESXi

Issue/Introduction

Troubleshooting curl certificate errors inside vSphere Integrated Containers

Symptoms:

You see an error similar to the following when you try to connect to an external server from a container using the curl command:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Environment

VMware vSphere Integrated Containers 1.x

Cause

Transfer of sensitive information is typically done under the cover of digital certificates. The certificate will help confirm to the recipient that the sender is actually who they claim they are. Digital certificates are issued by certificate authorities or CAs. A list of trusted certificate authorities and their root certificates are installed on a server when a digital certificate is applied to the server. For transactions over HTTPS this information will be exchanged for communication. When a server CA is received via cURL that isn't signed by one of the trusted certificates in the installed CA certificate store, this will cause SSL to report an error ("failed to verify the legitimacy of the server") during the handshake. SSL will then refuse further communication with that server.

Resolution

If the insecure Certificate is a known public CA, attempt to re-download the curl ca-bundle directly from the curl site. You can use the following command:

curl --remote-name --time-cond cacert.pem https://curl.haxx.se/ca/cacert.pem

If the insecure certificate is a self-signed, known-trusted certificate, load the certificate to your local trusted certificate location for your OS.

Additional Information

https://curl.haxx.se/docs/sslcerts.html

Use an openssl or curl command similar to the following to identify the certificate that is untrusted:

openssl s_client -showcerts -connect vcsa1.gsslabs.org:443

Note: Replace vcsa1.gsslabs.org with the URL where the untrusted certificate is originating. You will see output similar to the following:

CONNECTED(00000003)
depth=0 CN = vcsa1.gsslabs.org, C = US
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = vcsa1.gsslabs.org, C = US
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/CN=vcsa1.gsslabs.org/C=US
i:/CN=CA/DC=vsphere/DC=local/C=US/ST=California/O=vcsa1.gsslabs.org/OU=VMware Engineering
-----BEGIN CERTIFICATE-----
MIID9zCCAt+gAwIBAgIJAOZptRqIbAK+MA0GCSqGSIb3DQEBCwUAMIGYMQswCQYD
VQQDDAJDQTEXMBUGCgmSJomT8ixkARkWB3ZzcGhlcmUxFTATBgoJkiaJk/IsZAEZ
FgVsb2NhbDELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExGjAYBgNV
BAoMEXZjc2ExLmdzc2xhYnMub3JnMRswGQYDVQQLDBJWTXdhcmUgRW5naW5lZXJp
bmcwHhcNMTkwNDEyMTE1NzE5WhcNMjkwNDA2MTE1NzE5WjApMRowGAYDVQQDDBF2
Y3NhMS5nc3NsYWJzLm9yZzELMAkGA1UEBhMCVVMwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQC08GuZMXHO3z8py4Ng1VOl2WQISOXWKd5P2JBkA4hTn6nG
mWCcyHHTw77BnT6fQJkSS2hnz2i7BgzFcq4JUPkyKSexasMBQXLLXLO3BppJIqo9
3fa4xEzRYycDissB8ze57knKbIXyDlRTJsk84WwDH3fARhhYXF9QlbPHZ/dqU/RH
qZIXdaXxwPlAZ888xHxYzH+8sRIpjTej3of6KkL0MVK9AuOITIp5pZn6HIczmfet
uKGSErVz378Gmqc/SZhMZtuva+HDAZ/TLPBTFJGKSYqaKQQoPPi275gRHSj25uu7
1Br+JhoE8ZwVT81aK/2z/x/xNgJl3dp97aqNs1iZAgMBAAGjgbEwga4wCwYDVR0P
BAQDAgOoMBwGA1UdEQQVMBOCEXZjc2ExLmdzc2xhYnMub3JnMB0GA1UdDgQWBBQ1
0Dk3bCwK6r7bpxRQD4CfllllsjAfBgNVHSMEGDAWgBRouJ7pmOjAMJJhRkdw6jFx
NVOVLzBBBggrBgEFBQcBAQQ1MDMwMQYIKwYBBQUHMAKGJWh0dHBzOi8vdmNzYTEu
Z3NzbGFicy5vcmcvYWZkL3ZlY3MvY2EwDQYJKoZIhvcNAQELBQADggEBAE6JS1CJ
azmdRifWFjKc2G5JkzUlDDn/I71j2ojwPqKf9HczxzdPZY26IabGxyRy/FWwyL1j
fmEYoVxVdjQsAR9xJOJvyFvveIadEJ6bWgom79lEf9nlHPdaJ9qlZWr9PapXtJMM
1v8HhKvVJ2MAhYqE0tldNpgfGx2u7Gsal02BwXyo/Do1I24jm0VGDEF808r4LTKZ
3jsvIkyidNut5zNyT384BI/yZs62NtdMNyjM0Gcot3/j+FtzPXbs5IWxNW6P9GTw
JvQhdTFe4ikQJsRNHBhKHszlK0BW5ihirA09JTQ75HZL93xcw3Lu6sKLUbIDLofh
fTHx9RbB3bFzEDQ=
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=vcsa1.gsslabs.org/C=US
issuer=/CN=CA/DC=vsphere/DC=local/C=US/ST=California/O=vcsa1.gsslabs.org/OU=VMware Engineering
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1499 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID:
Session-ID-ctx:
Master-Key: 2184ABA9D7CEA14802EBFB1BECA6FE7DC322C76B9F9E6AAF82553DC5CA0B26F3C9A29342CD656CC47E5AF6B58661CA4A
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1585935973
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---

curl -kvvv https://vcsa1.gsslabs.org

Note: Replace vcsa1.gsslabs.org with the URL where the untrusted certificate is originating. You will see output similar to the following:

* Rebuilt URL to: https://vcsa1.gsslabs.org/
* Trying 192.168.0.51...
* TCP_NODELAY set
* Connected to vcsa1.gsslabs.org (192.168.0.51) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* ignoring certificate verify locations due to disabled peer verification
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=vcsa1.gsslabs.org; C=US
* start date: Apr 12 11:57:19 2019 GMT
* expire date: Apr 6 11:57:19 2029 GMT
* issuer: CN=CA; DC=vsphere; DC=local; C=US; ST=California; O=vcsa1.gsslabs.org; OU=VMware Engineering
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET / HTTP/1.1
> Host: vcsa1.gsslabs.org
> User-Agent: curl/7.59.0
> Accept: */*
>

Impact/Risks:

Connecting to an unknown insecure server can be hazardous. Only use the insecure flag in curl to a server if you can verify that the server is safe.
Adding an unknown CA to the trusted CA can be risky. Only add a CA to curl trusted CAs if you verify the legitimacy of the CA