Fazal Majid's low-intensity blog · Fazal Majid's low-intensity blog

Fiber for your home network

2024-03-17 Network

TL:DR Fiber as the backbone of your home network is easier than you’d think

My apartment, like many, is elongated. The living room is on one end, the bedrooms (one of which is my home office), on the other side. This makes it hard to cover both sides with a single WiFi access point, or to have uniform Internet access speed on the wired network. I have a semi-pro Ubiquiti UniFi network of WiFi access points and switches, which makes it realtively easy, but only if you have good backhaul connectivity between the APs.

For the longest time I used G.hn powerline networking bridges made by Devolo. Unfortunately, powerline is at least as unreliable as wireless networking and this made for frequent brown-outs requiring unplugging the Devolo Magic 2 boxes to power-cycle them. I know Devolo doesn’t make the actual PLC controllers and their firmware, probably made by Broadcom from the MAC addresses, but surely they could implement something as simple as a watchdog timer to reboot the PLC if no heartbeats are seen in a while?

At any rate, at some point I flipped the bozo switch on the Devolos and decided it was long past time to install proper Ethernet across the two halves of the apartment, which is easier said than done in a rental. To make things more complex, my Internet connection that used to be absymal Vodafone/BT OpenReach VDSL ending in my office was upgraded to a fiber ISP but this terminated by the door in half-way no-man’s land.

This was around the time I was experimenting with 10G Ethernet in the core of my home network, using Ubiquiti’s relatively inexpensive (for the time) USW-Aggregation switches with 8 SFP+ 10G ports. Speaking of which, while you can buy 10GBase-T SFP+ modules allowing you to use copper 10G Ethernet like the one on my Mac Studio, their power draw exceeds the specs of the SFP+ standard and they are unreliable, stick with fiber or use a switch with actual 10G ports (in my case a ZyXEL XGS1250-12, although it has an unfortunate tendency to overheat).

Contrary to what you may think, multimode fiber is much thinner (thus more discreet) and far more flexible than copper Ethernet cable (fiber above in the picture, copper below).

I conceived the idea of running a 30m pre-terminated fiber cable, made by a French company, as it turns out, along the crown molding in the ceiling, held in place with transparent plastic 3M Command hooks originally meant for holding Christmas lights, and easily removed without damage to the paintwork (this is a rental, remember).

I had to run it along a snaking route in red to stay along the crown molding, but even with my tyro DIY skills it only took a couple of hours to set up and is barely visible unless you know to look for it. While I don’t actually have any 10G devices in my living room yet, I do have a WiFi7 access point and it won’t be bottlenecked by the Ethernet network.

I still have a Devolo link between my office and the AP in my bedroom, but that’s a much shorter distance and much less unreliable.

Things I can do on my Mac but not on my iPad

2022-11-20 Mac Soapbox

See the full SMTP headers of an email, specially useful when you are trying to track down which email alias spam was addressed to.
Use an external UVC camera instead of the built-in one. I don’t trust the Mac Zoom app with its horrendous security record and the iOS sandbox is much more secure, but I’d also like to use my Sony RX1RII camera with an Elgato Camlink 4K for its superior image quality. Yes, I know using the web-based Zoom client is much more secure for you (because it is also sandboxes, in this case by the browser), but it also lacks end-to-end encryption.
SSH using ed25519-sk keys backed by a U2F hardware key like a Yubikey or trusting SSH certificates (yes, I know Termius supports it but subscription software is never acceptable).

PSA: LinkedIn single-sign-on dangers

2022-11-17 IT Soapbox

I have a work-issued computer that I keep rigorously separate from my personal stuff. It belongs to my employer and thus I do not keep personal files on it, or access personal email and certainly don’t save personal passwords on it. I even have it on a separate VLAN on my home network.

This is why I was horrified when I went to the LinkedIn website on my work computer (to look at a colleague’s posting) and it automatically started a single sign-on with my company’s GMail (my work address is of course linked to my LinkedIn profile).

This means a company with Google Apps can potentially access your LinkedIn account without your permission. Considering LinkedIn’s past record of egregious security failures¹, it shouldn’t be too surprising, but still…

I couldn’t find any setting to disable SSO, and it seems the only way to prevent this is to turn on two-factor authentication (where the only options are the grossly insecure phone SMS text message method or the equally phishable TOTP Authenticator app codes, not the actually secure Webauthn/FIDO U2F USB keys).

A colleague had built a GPU mining rig for fun and profit, and run the LinkedIn hashed password dump through it using hashcat. He found Donald Trump’s was a variation on “You’re fired!”… ↩︎

Funding the vetting of the Software Supply-Chain

2022-09-21 IT Economics

TL:DR A way out of our software supply-chain security mess

As memorably illustrated by XKCD, the way most software is built today is by bolting together reusable software packages (dependencies) with a thin layer of app-specific integration code that glues it all together. Others have described more eloquently than I can the mess we are in, and the technical issues.

Crises like the log4j fiasco or the Solarwinds debacle are forcing the community to wake up to something security experts have been warning about for decades: this culture of promiscuous and undiscriminating code reuse is unsustainable. On the other hand, for most software developers without the resources of a Google or Apple behind them, being able to leverage third-parties for 80% of their code is too big an advantage to abandon.

This is fundamentally an economic problem:

To secure a software project to commercial standards (i.e. not the standards required for software that operates a nuclear power plant or the NSA’s classified systems, or that requires validation by formal methods like TLA+), some form of vetting and code reviews of each software dependency (and its own dependencies, and the transitive closure thereof) needs to happen.
Those code reviews are necessary, difficult, boring, labor-intensive, require expertise and somebody needs to pay for that hard work.
We cannot rely entirely on charitable contributions like Google’s Project Zero or volunteer efforts.
Each version of a dependency needs to be reviewed. Just because version 11 of foo is secure doesn’t mean a bug or backdoor wasn’t introduced in version 12. On the other hand, reviewing changes takes less effort than the initial review.
It makes no sense for every project that consumes a dependency to conduct its own duplicative independent code review.
Securing software is a public good, but there is a free-rider problem.
Because security is involved, there will be bad actors trying to actively subvert the system, and any solution needs to be robust to this.
This is too important to allow a private company to monopolize.
It is not just the Software Bill of Materials that needs to be vetted, but also the process. Solarwinds was probably breached because state-sponsored hackers compromised their Continuous Integration infrastructure, and there is Ken Thompson’s classic paper on the risks of Trusting Trust (original ACM article as a PDF).
Trust depends on the consumer and the context. I may trust Google on security, but I certainly don’t on privacy.

I believe the solution will come out of insurance, because that is the way modern societies handle diffuse risks. Cybersecurity insurance suffers from the same adverse-selection risk that health insurance does, which is why premiums are rising and coverage shrinking.

If insurers require companies to provide evidence that their software is reasonably secure, that creates a market-based mechanism to fund the vetting. This is how product safety is handled in the real world, with independent organizations like Underwriters Laboratories or the German TÜVs emerging to provide testing services.

Governments can ditch their current hand-wavy and unfocused efforts and push for the emergence these solutions, notably by long-overdue legislation on software liability, and at a minimum use their purchasing power to make them table stakes for government contracts (without penalizing open-source solutions, of course).

What we need is, at a minimum:

Standards that will allow organizations like UL or individuals like Tavis Ormandy to make attestations about specific versions of dependencies.
These attestations need to have licensing terms associated with them, so the hard work is compensated. Possibly something like copyright or Creative Commons so open-source projects can use them for free but commercial enterprises have to pay.
Providers of trust metrics to assess review providers. Ideally this would be integrated with SBOM standards like CycloneDX, SPDX or SWID.
A marketplace that allows consumers of dependencies to request audits of a version that isn’t already covered.
A collusion-resistant way to ensure there are multiple independent reviews for critical components.
Automated tools to perform code reviews at lower cost, possibly using Machine Learning heuristics, even if the general problem can be proven the be computationally untractable.

The sad story of AIA Chasing

2022-09-21 Network Python

TL:DR Work-around for misconfigured HTTPS servers

This morning Temboz warned me that it had suspended The Oatmeal’s RSS feed due to too many errors. On further investigation, it turns out Temboz was getting these OpenSSL errors:

{'bozo': True,
 'bozo_exception': URLError(SSLCertVerificationError(1,
   '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable
   to get local issuer certificate (_ssl.c:997)')),
 'entries': [],
 'feed': {'summary': ''},
 'headers': {}}

But oddly enough the site (and its RSS feed) were loading perfectly fine in Vivaldi, Firefox or Safari, and their certificate trust path displaying properly.

Trying to fetch the site directly using the openssl command-line tool revealed the root cause of the problem: the server is misconfigured and sending only the first certificate, not the full path.

zbuild ~/build>openssl s_client -connect www.theoatmeal.com:443 < /dev/null 
CONNECTED(00000003)
depth=0 CN = theoatmeal.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = theoatmeal.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = theoatmeal.com
verify return:1
---
[1;32mCertificate chain
 0 s:CN = theoatmeal.com
   i:C = US, O = "DigiCert, Inc.", CN = RapidSSL Global TLS RSA4096 SHA256 2022 CA1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jul 19 00:00:00 2022 GMT; NotAfter: Jul 21 23:59:59 2023 GMT
[m---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHnjCCBYagAwIBAgIQDShn/5d79/AKJLby4Rc9BzANBgkqhkiG9w0BAQsFADBc
MQswCQYDVQQGEwJVUzEXMBUGA1UEChMORGlnaUNlcnQsIEluYy4xNDAyBgNVBAMT
... (elided for clarity) ...
jxgimUVPjChZSSpf5+wu9TdZDMTpdOneWsxgDJL6fzqJgw==
-----END CERTIFICATE-----
subject=CN = theoatmeal.com
issuer=C = US, O = "DigiCert, Inc.", CN = RapidSSL Global TLS RSA4096 SHA256 2022 CA1
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 2644 bytes and written 448 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 50D50D930B6256AE492D1F764C2C873C6AFA6198BB4A8D544C57DC2B198E74B2
    Session-ID-ctx: 
    Master-Key: 034F81BED2AEB74AD5A7A145C3C36F2D94BFECEA4D58A87F24C8390F924E041E40CD7C9C7F480EDAA8573A1EDAA499B1
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - e0 73 37 67 02 17 57 4e-f5 62 de 0a bd f7 2b 47   .s7g..WN.b....+G
    ...(elided for clarity) ...
    00b0 - 46 0a 22 1f 3e 4d 31 c8-92 3f ff 18 20 d8 b3 15   F.".>M1..?.. ...

    Start Time: 1663757669
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---
DONE

The certificate chain has only one certificate instead of the expected 3 (server, intermediate, root).

It’s not just a Python thing, curl compiled from source also fails, but the one shipped by Apple accepts it:

fafnir ~>/usr/local/bin/curl https://www.theoatmeal.com/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

fafnir ~>/usr/bin/curl https://www.theoatmeal.com/
<!DOCTYPE html>
<html lang="en">
<!--
    -\-                                                     
    \-- \-                                                  
     \  - -\                                                
      \      \\                                             
       \       \                                            
        \       \\                                              
         \        \\                                            
         \          \\                                        
         \           \\\                                      
          \            \\                                                 
           \            \\                                              
           \. .          \\                                  
            \    .       \\                                 
             \      .    \\                                            
              \       .  \\                                 
              \         . \\                                           
              \            <=)                                         
              \            <==)                                         
              \            <=)                                           
               \           .\\                                           _-
               \         .   \\                                        _-//
               \       .     \\                                     _-_/ /
               \ . . .        \\                                 _--_/ _/
                \              \\                              _- _/ _/
                \               \\                      ___-(O) _/ _/ 
                \                \                  __--  __   /_ /      ***********************************
                \                 \\          ____--__----  /    \_       I AM A MOTHERFUCKING PTERODACTYL
                 \                  \\       -------       /   \_  \_     HERE TO PTERO-YOU A NEW ASSHOLE
                  \                   \                  //   // \__ \_   **********************************
                   \                   \\              //   //      \_ \_ 
                    \                   \\          ///   //          \__- 
                    \                -   \\/////////    //            
                    \            -         \_         //              
                    /        -                      //                
                   /     -                       ///                  
                  /   -                       //                      
             __--/                         ///
  __________/                            // |               
//-_________      ___                ////  |                
        ____\__--/                /////    |                
   -----______    -/---________////        |                
     _______/  --/    \                   |                 
   /_________-/       \                   |                 
  //                  \                   /                 
                       \.                 /                 
                       \     .            /                 
                        \       .        /                  
                       \\           .    /                  
                        \                /                  
                        \              __|                  
                        \              ==/                  
                        /              //                   
                        /          .  //                    
                        /   .  .    //                      
                       /.           /                       
                      /            //                       
                      /           /
                     /          //
                    /         //
                 --/         /
                /          //
            ////         //
         ///_________////


-->

Apple uses its own Secure Transport library in curl, which must implement AIA because mainline curl still has AIA on its TODO as of 2021-09-21.

SSL Labs’ server test tool confirmed this:

So how do the browsers manage despite lacking the full chain? They use a work-around called Authority Information Access fetching (usually shortened to AIA fetching or AIA chasing). The server certificate has an optional X.509 field that has a URL to fetch the next certificate, in this case:

zbuild ~/build>openssl s_client -connect www.theoatmeal.com:443 < /dev/null | openssl x509 -text -noout
depth=0 CN = theoatmeal.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = theoatmeal.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = theoatmeal.com
verify return:1
DONE
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0d:28:67:ff:97:7b:f7:f0:0a:24:b6:f2:e1:17:3d:07
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = "DigiCert, Inc.", CN = RapidSSL Global TLS RSA4096 SHA256 2022 CA1
        Validity
            Not Before: Jul 19 00:00:00 2022 GMT
            Not After : Jul 21 23:59:59 2023 GMT
        Subject: CN = theoatmeal.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ad:01:37:58:d7:6a:f8:32:a0:26:c9:fd:8f:f3:
                    ... (elided for clarity) ...
                    7e:ef:cd:17:14:dc:55:d4:ff:a9:66:c4:96:57:02:
                    ca:a1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                F0:9C:85:FD:A2:9F:7D:8F:C9:68:BB:D5:D4:89:4D:1D:BE:D3:90:FF
            X509v3 Subject Key Identifier: 
                19:35:30:10:7D:D9:89:64:B5:A6:53:2C:76:6F:51:37:3B:2B:08:1C
            X509v3 Subject Alternative Name: 
                DNS:theoatmeal.com, DNS:www.theoatmeal.com
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 
                Full Name:
                  URI:http://crl3.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crl
                Full Name:
                  URI:http://crl4.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crl
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                  CPS: http://www.digicert.com/CPS
            [1;32mAuthority Information Access: 
                OCSP - URI:http://ocsp.digicert.com
                CA Issuers - URI:http://cacerts.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crt[m
            X509v3 Basic Constraints: 
                CA:FALSE
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : AD:F7:BE:FA:7C:FF:10:C8:8B:9D:3D:9C:1E:3E:18:6A:
                                B4:67:29:5D:CF:B1:0C:24:CA:85:86:34:EB:DC:82:8A
                    Timestamp : Jul 19 17:18:02.819 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:E8:D0:A9:EB:65:7D:19:73:9A:BC:F8:
                                0B:6D:30:FD:DB:47:83:79:D5:43:0C:92:00:1C:BF:E5:
                                E9:58:F2:B8:0E:02:21:00:8C:C1:69:33:FB:97:F4:E3:
                                A5:4A:8A:FD:AB:7E:E7:B9:17:0E:95:EF:BC:27:41:CE:
                                6C:EA:86:57:13:94:ED:C4
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 35:CF:19:1B:BF:B1:6C:57:BF:0F:AD:4C:6D:42:CB:BB:
                                B6:27:20:26:51:EA:3F:E1:2A:EF:A8:03:C3:3B:D6:4C
                    Timestamp : Jul 19 17:18:02.762 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:31:CC:04:98:48:B2:61:9C:66:7E:88:18:
                                06:08:30:72:E6:A7:F8:1C:9C:C6:65:BB:80:EF:41:F6:
                                E9:A3:3E:26:02:21:00:C6:3A:AB:5E:00:0C:DF:4B:E5:
                                70:39:3E:B6:2D:60:DF:9D:A0:DA:DE:A1:56:C4:87:D7:
                                49:EB:AF:BF:6F:3D:86
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : B3:73:77:07:E1:84:50:F8:63:86:D6:05:A9:DC:11:09:
                                4A:79:2D:B1:67:0C:0B:87:DC:F0:03:0E:79:36:A5:9A
                    Timestamp : Jul 19 17:18:02.824 2022 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:E3:D4:CF:BF:D0:8A:C5:BC:A6:28:F8:
                                49:87:75:F7:6B:A7:9B:21:7F:DB:6A:E6:69:C1:EC:D8:
                                F7:52:D7:4B:EA:02:21:00:E8:45:9A:7E:7E:2A:A6:EA:
                                64:96:60:95:1D:54:DE:2A:2F:3E:5F:25:C4:9E:02:2E:
                                0A:D1:6C:1F:93:17:51:EB
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        7d:35:f0:0a:96:36:17:4d:de:7b:95:14:5b:67:9d:b5:f5:27:
        ... (elided for clarity) ...
        60:0c:92:fa:7f:3a:89:83

In this case the browser will fetch http://cacerts.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crt and recursively until it has the complete chain.

zbuild ~/build>curl -s --output - http://cacerts.digicert.com/RapidSSLGlobalTLSRSA4096SHA2562022CA1.crt | openssl x509 -text -noout | ggrep -A 2 "Authority Information Access:"
            Authority Information Access: 
                OCSP - URI:http://ocsp.digicert.com
                CA Issuers - URI:http://cacerts.digicert.com/DigiCertGlobalRootCA.crt

and http://cacerts.digicert.com/DigiCertGlobalRootCA.crt resolves to a root certificate with no AIA but in the trust store.

Since The Oatmeal’s site may not remain broken forever (I have reported the issue to Inman), I created a site https://aia.majid.org/ deliberately broken to not include a full certificate chain, for testing purposes.