At 03:40 the change record said Completed. Both core switches at the NYC-2 site were on the new IOS XE release, HSRP had flipped once per core as planned, and 48 of 48 reachability probes answered. On nyc2-core-2, one line was gone: the DHCP relay address under Vlan220. Nothing alarmed, and nothing would have until the next time core-1 failed over and the guest VLAN stopped handing out addresses.

That example comes from a fictional network, but the pattern is real: a change that ran cleanly and still left the network worse. This post is the written version of my talk at ONUG Fall 2026 in New York, “Restored Is Not Fixed: how to verify a network change with evidence,” on the Tools, Technologies & Techniques stage. The method is vendor-neutral: SSH and a spreadsheet will do, as long as it ends in evidence rather than relief.

Why “successful” changes fail silently

I open the talk with 3 stories, generalized from conversations with practitioners and stripped of identifying details. An upgrade that “completed successfully” and then took weeks to root-cause. A deny rule that sat unnoticed for months. And 2 outages from pairs that were supposed to be redundant: a load-balancer pair whose configuration sync had failed, and a VPN standby that had drifted away from its active peer.

None of those changes failed in the way a checklist looks for. The commands ran and the devices came back. What failed was an assumption, usually in one of 3 ways:

  • Silent failures. A line is rejected, a script skips a device, or an object that other rules reference now means something different. The syntax was valid, so nothing errors.
  • Behavior deltas. An upgrade changes a default, removes or rewrites a line, or changes how a feature behaves. The diff you planned is not the only diff you got.
  • Drift on the other half. The change lands on the active unit, and the standby quietly disagrees until the day it has to carry traffic.

“The command ran” is an execution fact. Whether the network still does everything it did before, plus the one thing you meant to change, needs its own evidence. I ask 3 questions of every change: did it do what we intended, did it do anything we did not intend, and will it still hold when something else fails?

The before-and-after state pack, by change type

Verification starts before the window. Write down the intent, the invariants (what must not change) and the rollback trigger. Then capture a state pack: the same commands, run the same way, from the same place, before and after, with timestamps. What goes into it depends on the change:

  • Routing: route counts per VRF, neighbor states and uptimes, the prefixes the change touches, and forward and reverse path traces for the flows that matter.
  • Interfaces and port-channels: status and error counters, bundle membership and LACP state, CDP or LLDP neighbors, MAC counts per VLAN.
  • ACL and firewall policy: hit counters on the rules you touch, read twice a few seconds apart; members of every object group involved; a policy test for flows that must still pass and flows that must now be denied.
  • VLAN and spanning tree: VLANs allowed on each trunk, the root bridge per VLAN, topology-change counts, MAC counts.
  • HA pairs: first-hop redundancy state, vPC or MLAG consistency, firewall HA state and configuration sync, captured on both members.
  • Software upgrades: version and boot settings, running versus startup, and everything above that the device runs, since an upgrade can touch any of it.

For the NYC-2 core pair, the pre-state can be as plain as this, run on both cores (syntax varies by platform and version):

show clock
show version
show ip route summary
show ip ospf neighbor
show standby brief
show etherchannel summary
show spanning-tree summary
show mac address-table count
show ip interface Vlan220 | include Helper

Use the platform’s own safety nets too: a checkpoint and rollback-patch diff on Cisco NX-OS, a configuration session with a commit timer on Arista EOS (it reverts unless you confirm), and a confirmed commit on Juniper Junos:

checkpoint CHG2107pre
show diff rollback-patch checkpoint CHG2107pre running-config

configure session CHG2107
show session-config diffs
commit timer 00:10:00
configure session CHG2107 commit

commit check
show | compare
commit confirmed 10

None of these replaces the state pack. A configuration diff tells you what the device was told; the state pack tells you what the network did. Run the post-checks twice: immediately, and again 30 minutes later or the next business morning. Silent failures tend to show up in the second read.

5 verdicts instead of “completed”

“Completed” is a workflow state. The outcome of a verified change is one of 5 verdicts, and each needs its evidence:

  • Verified. The intent is met, the invariants held, and the immediate and delayed post-checks match the pre-state everywhere the change was not meant to act.
  • Degraded. The change is in and the rollback trigger was not met, but something it was not supposed to touch got worse. The missing relay address on nyc2-core-2 is a Degraded change: reachability held, and the next failover would have exposed the regression.
  • Rolled back. The trigger was met, or the owner chose to reverse. The evidence has to show the network back at its pre-change state, so the rollback gets verified too.
  • Inconclusive. The baseline or the post-change evidence is missing. Say what is missing and what would close the gap: with no pre-change read of the ACL counters, the post-change counters prove nothing.
  • Blocked. The change did not go ahead because a pre-check failed or a precondition was not met, such as a standby that was already out of sync before anyone touched the active.

Inconclusive is the method being honest. Score your last 20 changes this way and expect a good share to land there; that is your baseline, not a judgment of your team.

Write the evidence into the change ticket

The next engineer, the change advisory board and the auditor will all look in the change ticket, so that is where the evidence belongs, not in a chat thread. For the NYC-2 upgrade, CHG-2041, the work note might read:

CHG-2041 verification · nyc2-core-1/2 · IOS XE 17.9.5 -> 17.12.4
Intent: release change only; one planned HSRP flip per core
Rollback trigger: probes below 46/48, or both cores Standby > 30 s
Pre-state 02:55 · post-checks 03:41, 04:11 (attached)
Diff: nyc2-core-2 lost ip helper-address 10.9.0.20 under Vlan220
Probes 48/48 · HSRP core-1 Active, core-2 Standby, max gap 4 s
Verdict: Degraded · remediation RB-12 awaiting approval
Implementer, approver: named, with timestamps

That note will still be useful a year from now if it follows 4 rules: the verdict is one of the 5 words, not free text; the diff separates what you intended from what the release or the script did; the pre-state and post-checks are attached with timestamps; and the people are named.

I close the talk with 3 questions for anyone who says their product verifies changes, including us: show me the evidence, show me time to answer at P95, and show me what happens when it is wrong.

Where REAP fits today: change impact analysis, which assesses a proposed change against topology, reachability, security policy and known CVEs before it runs, is Available. Post-change verification is a Pilot deliverable: every proof of value verifies one real change window with your team and records the outcome and the evidence in your change ticket. Automated state packs and verdicts are In development. The step-by-step version is in our guide to verifying a network change.

All postsRishit Lakhani