The sad story of AIA Chasing
TL:DR Work-around for misconfigured HTTPS servers
This morning Temboz warned me that it had suspended The Oatmeal’s RSS feed due to too many errors. On further investigation, it turns out Temboz was getting these OpenSSL errors:
{'bozo': True,
'bozo_exception': URLError(SSLCertVerificationError(1,
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable
to get local issuer certificate (_ssl.c:997)')),
'entries': [],
'feed': {'summary': ''},
'headers': {}}
But oddly enough the site (and its RSS feed) were loading perfectly fine in Vivaldi, Firefox or Safari, and their certificate trust path displaying properly.
Trying to fetch the site directly using the openssl command-line tool
revealed the root cause of the problem: the server is misconfigured and
sending only the first certificate, not the full path.
zbuild ~/build>openssl s_client -connect www.theoatmeal.com:443 < /dev/null
CONNECTED(00000003)
depth=0 CN = theoatmeal.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = theoatmeal.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = theoatmeal.com
verify return:1
---
[1;32mCertificate chain
0 s:CN = theoatmeal.com
i:C = US, O = "DigiCert, Inc.", CN = RapidSSL Global TLS RSA4096 SHA256 2022 CA1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 19 00:00:00 2022 GMT; NotAfter: Jul 21 23:59:59 2023 GMT
[m---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHnjCCBYagAwIBAgIQDShn/5d79/AKJLby4Rc9BzANBgkqhkiG9w0BAQsFADBc
MQswCQYDVQQGEwJVUzEXMBUGA1UEChMORGlnaUNlcnQsIEluYy4xNDAyBgNVBAMT
... (elided for clarity) ...
jxgimUVPjChZSSpf5+wu9TdZDMTpdOneWsxgDJL6fzqJgw==
-----END CERTIFICATE-----
subject=CN = theoatmeal.com
issuer=C = US, O = "DigiCert, Inc.", CN = RapidSSL Global TLS RSA4096 SHA256 2022 CA1
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 2644 bytes and written 448 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 50D50D930B6256AE492D1F764C2C873C6AFA6198BB4A8D544C57DC2B198E74B2
Session-ID-ctx:
Master-Key: 034F81BED2AEB74AD5A7A145C3C36F2D94BFECEA4D58A87F24C8390F924E041E40CD7C9C7F480EDAA8573A1EDAA499B1
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - e0 73 37 67 02 17 57 4e-f5 62 de 0a bd f7 2b 47 .s7g..WN.b....+G
...(elided for clarity) ...
00b0 - 46 0a 22 1f 3e 4d 31 c8-92 3f ff 18 20 d8 b3 15 F.".>M1..?.. ...
Start Time: 1663757669
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: no
---
DONE
The certificate chain has only one certificate instead of the expected 3 (server, intermediate, root).
It’s not just a Python thing, curl compiled from source also fails, but the
one shipped by Apple accepts it:
fafnir ~>/usr/local/bin/curl https://www.theoatmeal.com/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
fafnir ~>/usr/bin/curl https://www.theoatmeal.com/
<!DOCTYPE html>
<html lang="en">
<!--
-\-
\-- \-
\ - -\
\ \\
\ \
\ \\
\ \\
\ \\
\ \\\
\ \\
\ \\
\. . \\
\ . \\
\ . \\
\ . \\
\ . \\
\ <=)
\ <==)
\ <=)
\ .\\ _-
\ . \\ _-//
\ . \\ _-_/ /
\ . . . \\ _--_/ _/
\ \\ _- _/ _/
\ \\ ___-(O) _/ _/
\ \ __-- __ /_ / ***********************************
\ \\ ____--__---- / \_ I AM A MOTHERFUCKING PTERODACTYL
\ \\ ------- / \_ \_ HERE TO PTERO-YOU A NEW ASSHOLE
\ \ // // \__ \_ **********************************
\ \\ // // \_ \_
\ \\ /// // \__-
\ - \\///////// //
\ - \_ //
/ - //
/ - ///
/ - //
__--/ ///
__________/ // |
//-_________ ___ //// |
____\__--/ ///// |
-----______ -/---________//// |
_______/ --/ \ |
/_________-/ \ |
// \ /
\. /
\ . /
\ . /
\\ . /
\ /
\ __|
\ ==/
/ //
/ . //
/ . . //
/. /
/ //
/ /
/ //
/ //
--/ /
/ //
//// //
///_________////
-->
Apple uses its own Secure Transport library in curl, which must implement
AIA because mainline curl still has AIA on its TODO as of 2021-09-21.
SSL Labs’ server test tool confirmed this:
So how do the browsers manage despite lacking the full chain? They use a work-around called Authority Information Access fetching (usually shortened to AIA fetching or AIA chasing). The server certificate has an optional X.509 field that has a URL to fetch the next certificate, in this case:
zbuild ~/build>openssl s_client -connect www.theoatmeal.com:443 < /dev/null | openssl x509 -text -noout
depth=0 CN = theoatmeal.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = theoatmeal.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = theoatmeal.com
verify return:1
DONE
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0d:28:67:ff:97:7b:f7:f0:0a:24:b6:f2:e1:17:3d:07
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = "DigiCert, Inc.", CN = RapidSSL Global TLS RSA4096 SHA256 2022 CA1
Validity
Not Before: Jul 19 00:00:00 2022 GMT
Not After : Jul 21 23:59:59 2023 GMT
Subject: CN = theoatmeal.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:ad:01:37:58:d7:6a:f8:32:a0:26:c9:fd:8f:f3:
... (elided for clarity) ...
7e:ef:cd:17:14:dc:55:d4:ff:a9:66:c4:96:57:02:
ca:a1
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
F0:9C:85:FD:A2:9F:7D:8F:C9:68:BB:D5:D4:89:4D:1D:BE:D3:90:FF
X509v3 Subject Key Identifier:
19:35:30:10:7D:D9:89:64:B5:A6:53:2C:76:6F:51:37:3B:2B:08:1C
X509v3 Subject Alternative Name:
DNS:theoatmeal.com, DNS:www.theoatmeal.com
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl3.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crl
Full Name:
URI:http://crl4.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crl
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
CPS: http://www.digicert.com/CPS
[1;32mAuthority Information Access:
OCSP - URI:http://ocsp.digicert.com
CA Issuers - URI:http://cacerts.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crt[m
X509v3 Basic Constraints:
CA:FALSE
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : AD:F7:BE:FA:7C:FF:10:C8:8B:9D:3D:9C:1E:3E:18:6A:
B4:67:29:5D:CF:B1:0C:24:CA:85:86:34:EB:DC:82:8A
Timestamp : Jul 19 17:18:02.819 2022 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:E8:D0:A9:EB:65:7D:19:73:9A:BC:F8:
0B:6D:30:FD:DB:47:83:79:D5:43:0C:92:00:1C:BF:E5:
E9:58:F2:B8:0E:02:21:00:8C:C1:69:33:FB:97:F4:E3:
A5:4A:8A:FD:AB:7E:E7:B9:17:0E:95:EF:BC:27:41:CE:
6C:EA:86:57:13:94:ED:C4
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 35:CF:19:1B:BF:B1:6C:57:BF:0F:AD:4C:6D:42:CB:BB:
B6:27:20:26:51:EA:3F:E1:2A:EF:A8:03:C3:3B:D6:4C
Timestamp : Jul 19 17:18:02.762 2022 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:20:31:CC:04:98:48:B2:61:9C:66:7E:88:18:
06:08:30:72:E6:A7:F8:1C:9C:C6:65:BB:80:EF:41:F6:
E9:A3:3E:26:02:21:00:C6:3A:AB:5E:00:0C:DF:4B:E5:
70:39:3E:B6:2D:60:DF:9D:A0:DA:DE:A1:56:C4:87:D7:
49:EB:AF:BF:6F:3D:86
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : B3:73:77:07:E1:84:50:F8:63:86:D6:05:A9:DC:11:09:
4A:79:2D:B1:67:0C:0B:87:DC:F0:03:0E:79:36:A5:9A
Timestamp : Jul 19 17:18:02.824 2022 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:E3:D4:CF:BF:D0:8A:C5:BC:A6:28:F8:
49:87:75:F7:6B:A7:9B:21:7F:DB:6A:E6:69:C1:EC:D8:
F7:52:D7:4B:EA:02:21:00:E8:45:9A:7E:7E:2A:A6:EA:
64:96:60:95:1D:54:DE:2A:2F:3E:5F:25:C4:9E:02:2E:
0A:D1:6C:1F:93:17:51:EB
Signature Algorithm: sha256WithRSAEncryption
Signature Value:
7d:35:f0:0a:96:36:17:4d:de:7b:95:14:5b:67:9d:b5:f5:27:
... (elided for clarity) ...
60:0c:92:fa:7f:3a:89:83
In this case the browser will fetch
http://cacerts.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crt and
recursively until it has the complete chain.
zbuild ~/build>curl -s --output - http://cacerts.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crt | openssl x509 -text -noout | ggrep -A 2 "Authority Information Access:"
Authority Information Access:
OCSP - URI:http://ocsp.digicert.com
CA Issuers - URI:http://cacerts.digicert.com/DigiCertGlobalRootCA.crt
and http://cacerts.digicert.com/DigiCertGlobalRootCA.crt resolves to a root
certificate with no AIA but in the trust store.
Since The Oatmeal’s site may not remain broken forever (I have reported the issue to Inman), I created a site https://aia.majid.org/ deliberately broken to not include a full certificate chain, for testing purposes.