Building a TLS Incident Runbook (Free Template)
The time to plan a fire drill is not during a fire.
This sounds obvious. It is. And yet most TLS incidents - expired certs, broken chains, CA issues, weak cipher findings during an audit - get handled the way people handle smoke alarms going off at 3am: in a panic, in pyjamas, with the wrong people in the room, trying to remember which valve to turn.
The fix is the same one the fire department uses. You write down what to do, who does it, what tools they reach for, what counts as "fixed," and what the post-fire report looks like. You print it. You test it. When the alarm goes off, you don't think - you reach for the document.
This piece is the document. It is a runbook template for TLS-related incidents that you can copy, adapt, and put in your team's wiki. Use it as-is or modify it for your environment. The point isn't to be perfectly written. The point is to have something to reach for when the alarm goes off at 3am and nobody can find Bob.
We'll cover the five questions every runbook must answer, then walk through the actual template section by section, then talk about how to keep it useful.
Why you need a TLS-specific runbook
You probably already have a general incident response runbook. That's good. A TLS-specific one is still useful because cert incidents are weird.
They are usually loud and obvious - every visitor sees a browser warning. But they are sometimes silent - the cert is valid, the chain is broken for 14% of clients, nobody notices for a week. Or the cert was renewed, but the new one isn't being served. Or the CA is having a bad day and your renewal is stuck.
The diagnostic flow is different from a typical service outage. You can't just "restart the thing." You have to figure out what failed: the cert itself, the chain, the cipher, the CA, the load balancer, the cron job. Each leads to a different fix.
And the communication is different. A TLS issue affecting customers is also a trust issue. The status page wording, the post-incident customer email, the contractual notification - all of these need to be right. The 3am improvisation usually isn't.
The five questions a runbook must answer
Every operational runbook, TLS or otherwise, has to answer five questions in order. If yours doesn't, it isn't a runbook - it's a vague guide.
One: how do we know there's an incident? What triggers the process? An alert? A customer report? A status page change? Multiple sources?
Two: who's responsible? Who is on-call? Who else needs to be in the loop? Who has the authority to make calls (issue a new cert, change DNS, talk to the CA)?
Three: what do we check, and in what order? Triage steps. What's most likely. What's quickest to rule out.
Four: how do we fix it? Branching paths by failure type. Each path is a series of concrete actions.
Five: how do we know it's actually fixed? Validation steps. What evidence confirms the resolution.
A runbook missing any of these will fail on the night it matters.
The runbook template
What follows is a copy-paste-and-adapt template. Things in [brackets] are placeholders for your environment.
---
INCIDENT TYPE: TLS / Certificate Issue SEVERITY: [SEV-1 if customer-facing TLS is broken; SEV-2 if internal; adjust to your scheme] ON-CALL ROLE: [e.g. SRE-on-call, Platform Engineering On-Call] INCIDENT CHANNEL: [e.g. #incident-tls or auto-create from PagerDuty]
---
STEP 1 - Initial Confirmation (first 5 minutes)
The alert says we have a TLS problem. Before doing anything else, confirm.
- [ ] Reach the affected endpoint from outside your network (mobile data, a different country, your phone). Do you see a browser warning? [ ] Check the alert source. Is it our monitoring (TLS Radar / internal), a customer report, or a public status page? [ ] Check
[STATUS PAGE URL]to see if a status incident has already been opened. [ ] Runopenssl s_client -connect [HOSTNAME]:443 -servername [HOSTNAME]and note: expiry date, chain length, cipher negotiated, any errors.
If the issue is confirmed real and customer-facing, escalate to SEV-1 and proceed.
---
STEP 2 - Notify (next 5 minutes)
- [ ] Page [INCIDENT MANAGER ROLE] if not already in the channel. [ ] Post a brief update to
[STATUS PAGE]: "We're investigating a connection issue affecting some users. Updates to follow." [ ] Notify[INTERNAL STAKEHOLDERS CHANNEL](Customer Success, Sales, Support). [ ] If the issue affects a partner integration, notify [PARTNER LIAISON ROLE].
Do not say "expired certificate" publicly until you've confirmed that's the cause. "Connection issue" is the safer framing during diagnosis.
---
STEP 3 - Diagnose (next 15 minutes)
Pick the right diagnostic path based on what openssl s_client showed in Step 1.
Path A - Certificate is expired or expiring imminently: - Confirm the expiry date is in the past or very near. - Check if a renewal was attempted recently in [CERT MANAGEMENT TOOL / ACME LOGS]. - If the renewal succeeded but isn't deployed, you have a deployment problem (jump to Path D). - If the renewal didn't run, jump to Path E (CA / automation issue).
Path B - Chain is broken (cert valid, chain validation fails): - Check the chain depth in openssl s_client output. Are all intermediates present? - Check the intermediate CA: has it been rotated recently? - Test from multiple clients: Chrome, Firefox, mobile, an old Android device, curl with old OpenSSL. Different clients may behave differently.
Path C - Cipher / protocol issue: - Run testssl.sh [HOSTNAME] or sslyze [HOSTNAME] for a full cipher scan. - Compare against your documented cipher baseline. What changed? - Check recent config deployments to the affected service.
Path D - Cert is fine, but not being served (deployment issue): - Check the load balancer config. Is the new cert installed? - Check if the web server / app server has been reloaded since the renewal. - Check for SNI mismatches if multiple certs are on the same IP.
Path E - CA or automation issue: - Check the CA's status page (Let's Encrypt, DigiCert, your internal CA). - Check ACME logs for failed challenges. - Check DNS for any recent changes that might have broken DNS-01 validation. - Check firewall logs for HTTP-01 challenge failures.
---
STEP 4 - Resolve
Based on diagnosis:
For Path A (expired cert): issue an emergency renewal via [CA / ACME COMMAND]. Verify deployment via openssl s_client.
For Path B (chain broken): redeploy the correct intermediate chain. Verify across multiple client environments.
For Path C (cipher issue): roll back the recent config change or apply the documented hardening baseline. Verify with a fresh cipher scan.
For Path D (deployment issue): push the cert to the load balancer or web server; reload the service. Verify via openssl s_client and from an external network.
For Path E (CA issue): if your CA is down, switch to a backup CA if you have one, or wait. Update the status page. Communicate to customers and contractually-bound partners as appropriate.
---
STEP 5 - Validate
Before closing the incident:
- [ ] Run
openssl s_clientfrom outside your network. Confirm cert details match expectations. [ ] Runtestssl.shor your standard cipher scan. Confirm grade matches baseline. [ ] Test from at least three different client environments (Chrome, Firefox, mobile). [ ] Check[MONITORING DASHBOARD]is green. [ ] Wait 5 minutes. Re-verify. CDN cache and DNS TTLs can hide partial fixes.
---
STEP 6 - Close
- [ ] Update status page with resolution. [ ] Notify internal stakeholders. [ ] Post a brief internal summary to the incident channel. [ ] Schedule the post-mortem for 24–72 hours later.
---
STEP 7 - Post-incident (within 1 week)
- [ ] Write the post-mortem in
[POST-MORTEM TEMPLATE LOCATION]. [ ] Identify the root cause AND the missing alarm. Why didn't monitoring catch this earlier? [ ] Identify two preventive actions and assign owners. [ ] Update this runbook if any step turned out to be wrong or missing. [ ] If the incident affected customers materially, draft a customer communication.
---
How to test it
A runbook nobody has read is not a runbook.
Test it twice a year, at minimum. The format is a tabletop exercise - one person plays "incident commander," the others play their roles, and someone (often a security lead) plays "the universe" by introducing changes ("now the CA is also down").
The first time you do this, the runbook will reveal gaps. That's the point. Update it. Test again next quarter.
A specific scenario worth testing: "the on-call engineer is unreachable." Who picks up? Does the runbook still work without them? If the answer is "ask Bob," you've found the gap.
Feeding incidents back in
The runbook gets better the more it gets used. Every real incident should produce at least one runbook edit:
- A diagnostic step that was missing. A communication channel that was wrong. A tool that wasn't installed where it needed to be. A role that wasn't on call.
A runbook frozen at version 1 is a fossil. A runbook that's been edited fifteen times in the last year is a tool people trust.
One small ask
The template above is yours to copy. You can grab the markdown source at [REPO/DOCS LINK] and put it in your wiki today.
The bigger principle: monitoring catches problems; runbooks resolve them; post-mortems prevent the next one. All three need to be in place. TLS Radar handles the first one for any CA and any setup - including the kind of silent failures that are hardest for internal monitoring to spot. Free tier covers three domains, which is enough to test whether you'd have caught the last incident you had.
Related reading
Get the next post in your inbox
TLS monitoring tips and product updates. No spam, unsubscribe anytime.