Post-Mortem Patterns: Why TLS Outages Are Always "Surprising
You don't usually discover a roof leak in the middle of a sunny afternoon. You discover it during the storm - water dripping onto the kitchen counter, a brown stain spreading across the ceiling. The leak feels sudden. It feels like the storm caused it.
The storm did not cause the leak. The leak had been forming for months, possibly years. A loose shingle. A failed seal. A small displacement after last winter's freeze. The water had been finding its way in slowly, accumulating behind the ceiling, looking for somewhere to drip. The storm just provided enough water at once for the leak to become visible.
TLS outage post-mortems follow the same pattern. The opening line is almost always some variant of "the cert expired unexpectedly" or "an unforeseen chain validation issue" or "we were surprised by..." The word "surprised" appears more in TLS post-mortems than in any other type of incident report.
The certs were not surprising. The patterns are highly predictable. The surprise is a function of what wasn't being watched, not what happened.
Here are the five patterns that show up in nearly every cert outage post-mortem, why teams keep calling them surprising, and how to write one that's actually useful.
The five patterns that aren't surprising at all
Pattern 1: Ownership had drifted. The post-mortem says "the engineer responsible had moved teams six months earlier." It doesn't say "we had no process for transferring cert ownership." Those are the same thing, and the second framing is the actionable one.
The ownership drift is not surprising. It is the predictable consequence of organisations where certs are owned by people rather than teams. Every cert older than three years in any organisation with normal turnover has at least one drifted ownership. The outage is downstream of a structural choice that nobody named.
Pattern 2: Monitoring had a blind spot. The post-mortem says "the cert was not being monitored." It doesn't say "our monitoring strategy assumed all certs were on the registered list and didn't catch the ones that weren't." Same fact, different actionability.
This is the inventory problem. Certs exist that the monitoring doesn't know about - internal CAs, vendor-embedded, subdomains added since the last review. The outage is downstream of an inventory that doesn't update itself.
Pattern 3: Renewal automation had failed silently. The post-mortem says "the ACME renewal had been failing for X weeks." It doesn't say "we had no monitoring of our monitoring." The renewal pipeline reported failures into a Slack channel that nobody read, or to an email account that no longer routes.
The silent failure isn't surprising. Any automation will eventually fail. The question is whether anything notices.
Pattern 4: Chain validation broke for a subset of clients. The post-mortem says "users on older Android devices started experiencing connection failures." It doesn't say "we had not tested chain validation across our actual client population." The cert was valid; the chain was broken for 14% of clients; the monitoring only checked from one client environment.
This is the chain monitoring gap covered elsewhere on the blog. It is not surprising. It is predictable from "we do not test multi-client chain validation."
Pattern 5: The CA had a bad day. The post-mortem says "we were affected by an upstream CA incident." It doesn't say "we had no contingency plan for CA-level failures." The May 2026 Let's Encrypt issuance halt caught teams that hadn't thought about CA dependency. The 2024 cross-sign drama caught teams that hadn't thought about chain validation.
CA incidents are not surprising as a category. The specific timing is unpredictable; the existence of such incidents is well-documented.
Why teams call them surprising anyway
If the patterns are so predictable, why does every post-mortem describe them as surprising?
Three reasons.
Cognitive economy. Teams have a lot to monitor. Cert hygiene is, for most teams, not their primary work. The mental model includes "we have certs, they renew" without granular awareness of the failure modes. When one of those failure modes manifests, it genuinely is surprising to the people involved - even though it's been predictable to people who study the category.
Organisational framing. "We didn't see it coming" sounds better in a post-mortem than "we hadn't built the monitoring that would have seen it." The first is exonerating; the second is actionable. Teams prefer the first because it doesn't sound like an accusation.
The "this never happens" defence. Many teams have had years of clean cert operations. The absence of incidents reinforces the belief that the current setup is sufficient. The first outage is then framed as unprecedented, even though the underlying conditions had been deteriorating quietly for the entire clean period.
The result: post-mortems that genuinely feel like discoveries of bad luck, when they're really discoveries of accumulated debt.
How to write a useful post-mortem
A post-mortem that names the structural condition is more useful than one that names the proximate cause.
Specifically, don't write: "The cert expired because nobody renewed it."
Write: "The cert expired because cert ownership was held by an individual who left, and we had no process for redistributing ownership when individuals leave. The renewal alert went to an email address that no longer routes. This is the third cert-related incident in two years with the same structural cause."
The first version names a fact. The second names a pattern. The second leads to a fix that prevents the next incident. The first leads to a fix that prevents this specific cert from expiring again.
For TLS post-mortems specifically, the questions worth answering:
- Who owned this cert at the time of the incident, and who owned it when it was first issued? Was the cert in our monitored inventory? If not, why not? Did automation report success on the relevant renewal? If so, why didn't the deployment succeed? What's the next cert that would fail for the same reason? If you can name one, the structural problem still exists. What changes prevent this category of incident, not just this incident?
The answers tend to be uncomfortable. They name systems that don't work, not just one event that went wrong. The discomfort is the value - it's what produces structural fixes rather than cosmetic ones.
One small ask
The next TLS outage post-mortem you write should include the sentence "this was not surprising." If it's not true, the structural conditions were already in place. The post-mortem should name those conditions. TLS Radar provides continuous monitoring that catches the patterns above before they become post-mortems - silent renewal failures, chain breaks across client populations, drifted ownership, inventory gaps. The 30-day free trial covers three domains, which is enough to see how many of the "predictable" patterns are already running in your environment.
Related reading
Get the next post in your inbox
TLS monitoring tips and product updates. No spam, unsubscribe anytime.