Chain Validation Pitfalls (And Why Cross-Signing Bites)
When you write a letter of recommendation, the letter only works if the recipient recognises the recommender. A letter from Dr. Smith means something to the admissions committee that knows Dr. Smith. It means less to one that doesn't. A letter from a friend of Dr. Smith - vouching for Dr. Smith, who vouches for the candidate - works if the committee accepts that chain of "this person trusts this person trusts this person."
If any link in that chain breaks, the whole letter loses its weight.
TLS certificate validation works the same way.
When a browser sees a certificate for example.com, it doesn't decide on its own whether to trust it. It checks who signed the cert - usually an intermediate certificate authority. Then it checks who signed that - a root CA. The root has to be in the browser's trust store. If the chain is broken anywhere - a missing intermediate, an expired link, a cross-sign that doesn't validate - the connection fails.
This sounds simple. It is not. The places where chains break in production are subtle, the symptoms are inconsistent across clients, and the fixes are sometimes politically complicated by industry-wide cross-signing arrangements.
Here's how chain validation actually works, the four places it breaks, and what proper testing looks like.
How chain validation actually works
A TLS chain has three parts:
The leaf certificate. The one for your specific domain. Issued by an intermediate CA. This is the one you install on your server.
One or more intermediate certificates. Signed by a root CA, used to sign leaf certs. There are intermediates because root CAs are kept offline for security - they don't sign leaf certs directly. They sign intermediates, which do the day-to-day work.
A root certificate. At the top of the chain. The root is what browsers, OSes, and TLS libraries actually trust. Roots live in trust stores: Mozilla's CA store (used by Firefox, many tools), the Microsoft store, the Apple store, the Android store, the Java store. Each is slightly different.
When a client connects to your server, the server sends the leaf cert and (usually) the intermediates. The client tries to build a chain from the leaf up to a root in its trust store. If it can, the connection is trusted. If it can't, the connection breaks.
The catch: different clients have different root stores. A cert that validates in Chrome on a modern laptop might fail in an old Android phone, an old Java application, an old curl, or a corporate proxy with a custom trust store.
Where chains break
Four places, in rough order of frequency.
Missing intermediates. Your server is supposed to send the full chain - leaf plus intermediates - in the TLS handshake. Some servers don't. Modern Chrome can sometimes recover by fetching the missing intermediate from a URL embedded in the cert, but other clients can't. The connection then fails on those clients. Symptom: site works for some users, fails for others.
Intermediate rotation. CAs rotate their intermediates periodically - sometimes because of compromise, sometimes because of policy changes. Your server is still serving the old intermediate. Modern clients fall back, older ones don't. Symptom: site works in browsers but fails for older mobile apps or B2B integrations.
Expired intermediates or cross-signs. Intermediates can expire too. Cross-signed roots can expire. The Let's Encrypt cross-sign drama in 2024 was exactly this - an expiring cross-sign caused chain validation failures on older clients. Symptom: site suddenly fails on a specific subset of clients with no apparent change on your end.
Wrong intermediate served for the wrong leaf. Multi-cert servers (multiple sites on one IP via SNI) sometimes get configured to serve the wrong intermediate for one of the certs. The leaf is right. The intermediate is the one for a different leaf. Some clients tolerate this; others don't.
The common thread: in all four cases, the cert itself looks valid. The expiry date is in the future. The cipher is fine. The basic checks pass. But somewhere in the chain of trust, a link doesn't validate, and a subset of users see broken connections.
The cross-signing complication
Cross-signing is when a CA's root is also signed by another CA's root, to bootstrap trust in older client trust stores.
Why it exists: new root CAs need a long time - sometimes a decade - before their roots get added to all major trust stores. In the meantime, the new CA can have its root signed by an established CA's root. Then old trust stores that already trust the established CA implicitly trust the new one, via the cross-sign.
Why it bites:
Cross-signs expire. Eventually the cross-signed root reaches its expiry date. Old clients that were validating via the cross-sign suddenly can't anymore. New clients are fine because they have the new root directly. Old clients break.
The expiry day is known years in advance but rarely planned for. The 2024 Let's Encrypt cross-sign expiry caused real problems for older Android devices, certain Java applications, and some enterprise security appliances. It was a known date for years. The breaks happened anyway because nobody owned testing against the affected client base.
Cross-sign behaviour differs by client. Some clients prefer the cross-sign; some prefer the direct root. The behaviour can be inconsistent in subtle ways that only show up under load.
In practice, cross-signing is the messy compromise between "we need new CAs to work in old clients" and "we need certificates to actually validate cleanly." The messy compromise works most of the time. It bites when it stops.
How to test chains properly
Single-client testing - opening your site in Chrome and seeing it works - catches almost none of the chain problems above. Real chain testing requires:
Multiple browsers, current versions. Chrome, Firefox, Safari, Edge.
Multiple OSes. Different Android versions matter especially. Older OpenSSL versions matter. Java applications using JKS trust stores matter.
Command-line tools with controllable trust stores. openssl s_client against your endpoint, with -CAfile pointing at different root stores. testssl.sh does a good job of this. sslyze similarly.
Continuous external scanning that checks chains across multiple trust store configurations. This is what TLS Radar does, and what most internal monitoring doesn't.
The reason it matters: chain breaks often affect a subset of users, not all of them. If you test from one client and see "works fine," you might be missing a 15% client population for whom your site is silently failing.
What modern monitoring catches
Good chain monitoring checks:
- That all intermediates are present in the server's response. That each intermediate is valid (not expired, not revoked, properly signed by the next link). That the full chain validates against multiple trust stores (modern browsers, modern Android, old Android, Java, Windows, others). That the cross-signs in play haven't expired and aren't about to. That the chain depth is reasonable (very deep chains have their own performance and compatibility issues).
If your current monitoring only checks "is the cert valid?" it's missing most of this. The cert can be valid and the chain still be broken for a meaningful percentage of clients.
One small ask
Chain validation is the most common failure mode that pure expiry monitoring misses. The cert dashboard says green; users see warnings; the on-call engineer has nothing to look at. The fix is multi-client chain testing that runs continuously. TLS Radar checks chains against modern and legacy trust stores every day, surfaces cross-sign expiry well in advance, and alerts when a client subset starts failing. Free tier covers three domains, which is enough to test whether your current chains hold up across the audiences you actually serve.
Catch a broken chain before your users do
A chain that validates on your server can still fail on an older Android phone, a Java client, or a browser with a different trust store. TLS Radar fetches and validates the full chain from the outside against modern and legacy trust stores and tells you which clients would refuse it.
Get the next post in your inbox
TLS monitoring tips and product updates. No spam, unsubscribe anytime.