Network incident postmortem template (blameless and evidence-first)

Blameless and evidence-first

Blameless means the review asks why an action made sense at the time, with the information the person had, and what in the system allowed it to cause harm. The finding is the set of conditions that let a reasonable action fail, not the person who took it. This is not leniency; it is how you hear what actually happened.

Evidence-first means every statement in the timeline and the analysis points to something checkable: command output, a log line, a configuration diff, a ticket entry, a graph with its time range. A statement without evidence is labeled as a recollection, and a conclusion without evidence is labeled as a hypothesis.

Write one for every P1, every repeat incident, every incident that followed a change, and every event where redundancy did not work as designed, even when users did not notice.

The structure

1. Summary. 3 to 5 sentences: what happened, who was affected, for how long, and the current status (restored, root cause confirmed, or fixed). Write it last.
2. Impact. Services, sites and users affected; impact start and end with the time zone; how the impact was measured.
3. Detection and response times. Impact start, detection, acknowledgement, restoration and root-cause confirmation, each with its source.
4. Timeline. Timestamped events from every source, in one time zone.
5. Evidence list. Numbered items that the timeline and the analysis refer to.
6. What changed. Configuration, software, hardware, traffic and dependencies, whether or not a change record exists.
7. Root cause and contributing factors. The mechanism, and the conditions that let it happen, go undetected and last as long as it did.
8. What went well and what was hard. Detection, diagnosis, communication, restoration.
9. Actions. Each with a type, an owner, a due date and a definition of done.
10. Status: restored or fixed. And the evidence that would move it to fixed.

Build the timeline with timestamps

• Use one time zone for the whole document and state it at the top, with its UTC offset.
• Give every line a source: device log, monitoring alert, ITSM entry, chat message, bridge notes, change record.
• Separate four moments that are easy to blur: when impact started, when it was detected, when someone acknowledged it, and when service was restored.
• Include the hypotheses tested and rejected, with their times. They show where the diagnosis time went.
• Check device clocks before trusting device timestamps. A router 4 minutes off can reorder cause and effect; note any offset you corrected for.
• Collect volatile evidence early. Logging buffers wrap, and on many platforms are lost on reload; routing, ARP and MAC tables change by the minute.

The commands below check clock synchronization and pull configuration-change history, two sources most network timelines need.

The evidence list

Number every item (E1, E2 and so on) and refer to the numbers in the timeline and the analysis. For each item, record:

• when it was captured, and the time range it covers;
• the device or system, and the exact command, query or file;
• the relevant lines, quoted, not paraphrased;
• what it shows, in one sentence;
• where the full item is stored (attach it to the incident or problem record).

Example, from a fictional network: E4 · 02:08:05 · plant-a-core-2 · show ip access-lists ACL-ERP-IN, two reads 25 seconds apart · object-group entry 0 matches; final deny 3,412 then 3,918 · return traffic from the ERP servers is hitting the final deny.

Include evidence that contradicts the conclusion, and say how you resolved it. Remove passwords, keys and community strings from configurations before attaching them.

Contributing factors: what changed and what detection missed

Expect one trigger and several contributing factors. List them all: a single root-cause sentence hides the factors that are cheapest to fix. Work through these questions:

• What changed? Check change records and the devices’ own history (commit logs, configuration archives, accounting logs), because not every change has a record. Include software, hardware, traffic and upstream providers. See what changed.
• Why did the change have this effect? The mechanism, shown with evidence: which line, which table, which packet path.
• Why did the pre-change checks not catch it? Was there a check for this behavior, and was it run?
• What did detection miss? If users reported it first, what signal existed earlier, and what alert or probe would have caught it?
• What slowed diagnosis? Missing access, missing topology, logs that had wrapped, the one engineer who knew the design being off shift.
• What slowed restoration? Approvals, a rollback that was not ready, a standby that was not ready.

“Human error” is where analysis starts, not where it ends. Ask what made the error possible and what should have caught it.

Actions with owners and dates

• Every action has one named owner (a person, not a team), a due date and a ticket.
• Write what done means and what evidence proves it: “probe on the ERP path alerting to the NOC, test alert received” rather than “improve monitoring”.
• Label each action: prevent (remove the cause), detect (catch it sooner), mitigate (limit the impact) or process (change how the work is done).
• If detection was late, include at least one detect action.
• Prefer 3 actions that will be done over 12 that will not. Track them in the problem record and review them until they close.

How to run the review meeting

1. Send the draft a business day ahead, with the timeline and the evidence list, so the meeting corrects facts instead of discovering them.
2. Invite the people who did the work: the incident lead, the engineers involved, the service owner, the change owner if a change was involved, and a facilitator who was not involved.
3. Open with the ground rule: the goal is to understand the system, not to find a person.
4. Walk the timeline (about 15 minutes) and correct it. Ask “what did you see at that moment?” and “what made that the right next step?”
5. Agree the contributing factors (about 15 minutes), each linked to evidence.
6. Agree the actions (about 15 minutes), with owners and dates confirmed by the owners in the room.
7. Decide the status: restored, root cause confirmed or fixed, and what evidence would move it forward.

Keep it to 60 minutes. Record disagreements in the document rather than settling them by seniority.

Restored versus fixed

Restored is not the same as fixed. A reload, a failover or a rollback restores service; none of them removes the cause. Use three explicit states:

• Restored: service is back, possibly through a workaround. The cause may still be present.
• Root cause confirmed: the mechanism is shown with evidence, not inferred from timing alone.
• Fixed: the corrective change is made and verified, and the conditions that allowed the incident are covered by owned actions.

Keep the problem record open until the incident is fixed. A recurrence of the same cause within 90 days is a repeat incident, and the plainest test of whether the postmortem worked.