The method: capture, change, capture, compare
Every verification follows the same four steps, whatever the platform.
- Write the intent and the rollback trigger first. One sentence on what should change, a list of what must not change (the invariants), and the measurable condition that makes you roll back. Example: “Allow VLAN 230 on the uplink trunks of dist-1 and dist-2; no other VLAN, port-channel or spanning-tree root changes; roll back if any access switch loses its uplink for more than 60 seconds.”
- Capture the pre-state. Run the same command set on every device in scope, including both members of every redundant pair, as close to the window as you can. Save each output to a file named with the change number, the device, pre or post, and the time.
- Make the change, then capture the post-state twice. Once immediately after the window, and once after a delay: 30 minutes at least, longer if the change affects traffic that peaks later. ARP and MAC aging, BGP hold timers and DHCP leases mean some failures only appear later.
- Compare. Diff the configuration against the pre-change copy and read every changed line against the intent. Then compare state: neighbors, route counts, port-channel members, redundancy roles, MAC and ARP counts, counters and probe results.
Three habits make the comparison trustworthy:
- Complete captures. Disable paging first:
terminal length 0on Cisco (opens in a new tab) IOS XE, Cisco NX-OS (opens in a new tab) and Arista EOS (opens in a new tab),set cli screen-length 0on Juniper (opens in a new tab) Junos (opens in a new tab),set cli pager offon PAN-OS. - Two reads, not cleared counters. Read counters twice, a known interval apart. A counter that rises between two reads tells you what is happening now; a cumulative total does not, and clearing counters erases history someone else may need.
- The data plane, not only the control plane. A neighbor can be up while traffic is dropped. Keep a list of probes (ping, TCP connect, application checks) across the paths the change touches, and run the same list before and after.
Words in capitals in the commands below (PEER, NAME, INTERFACE, ADDRESS, CHG-NUMBER) are placeholders for your values.
What to check for each change type
Match the checks to the change. For each type, the list says what must be true afterward.
- Routing (BGP). Every session that was established is established again. Prefixes received and advertised per neighbor match the baseline, or differ exactly as intended. Best paths for your key prefixes are unchanged unless the change targeted them. No new flaps in the log.
- Routing (OSPF). The same neighbors in FULL state (on a broadcast segment, 2WAY between two routers that are neither DR nor BDR is normal). Route counts per area and source match the baseline. Interface costs are as intended.
- Interfaces and port-channels (LAGs). Every interface in scope is up, with no new input or CRC errors between two reads. Every port-channel has the same members bundled as before, and the LACP partner on each member is the device you expect.
- ACL or firewall policy. The traffic you meant to permit passes and the traffic you meant to block does not, tested with a policy lookup and a live probe. Hit counters rise on the new or changed entries. An explicit final deny does not start climbing.
- VLAN and spanning tree. The VLAN exists, is active, and is allowed on exactly the intended trunks; the rest of each allowed list is unchanged. The root bridge for each VLAN or instance is unchanged. The topology-change count stops rising after the window. MAC table counts return close to the baseline.
- First-hop redundancy (HSRP, VRRP). Each group is active (HSRP) or master (VRRP) on the intended router, with a standby or backup present. Preemption and tracking behave as designed.
- MLAG and vPC. Peer link and keepalive up, consistency checks passing, every MLAG or vPC interface active on both peers, no new orphan ports.
- HA pairs (firewalls, supervisors, stacks). Both members present, roles as expected, configuration and state synchronized, and no unplanned failover during the window.
- Software upgrades. The intended release is running, and committed where the platform asks for it, on every member. Modules, transceivers and licenses match the baseline; neighbor, route, MAC and ARP counts are back to the baseline; no new crash or error messages. Separate the lines a new release adds by default from lines that disappeared: a missing line after an upgrade is a regression until proven otherwise.
One trunk mistake is worth checking every time. On Cisco (opens in a new tab) IOS XE and Cisco NX-OS (opens in a new tab), switchport trunk allowed vlan 230 replaces the whole allowed list, while switchport trunk allowed vlan add 230 adds to it. Compare the allowed list before and after any trunk change.
Cisco IOS XE pre- and post-checks
Run each command before and after the change on every device in scope. On a Catalyst stack, run them on the active switch; the output covers every member.
Save a copy of the running configuration before you start. After the change, show archive config differences with that file name compares it with the running configuration. With no file name, the same command compares the startup configuration with the running configuration, which is the quickest way to find unsaved work before the window.
| Check | Command | What good looks like |
|---|---|---|
| Unsaved changes before you start | show archive config differences | No differences between the startup and running configuration, so the window starts from a saved, known state |
| Pre-change configuration copy | copy running-config flash:CHG-NUMBER-pre.cfg | File saved in flash before the first change |
| Configuration diff after the change | show archive config differences flash:CHG-NUMBER-pre.cfg | Only the lines the change intended: + marks lines added since the copy, - marks lines removed |
| BGP sessions | show ip bgp summary | Every neighbor that was established shows a prefix count in State/PfxRcd (not Idle or Active); counts match the baseline or the intended change |
| BGP prefixes advertised | show ip bgp neighbors PEER advertised-routes | Only the prefixes your policy intends; the total matches the baseline |
| OSPF neighbors | show ip ospf neighbor | Same neighbors as the baseline, in FULL (2WAY between two DROTHER routers is normal) |
| Routing table size | show ip route summary | Counts per source (connected, static, OSPF, BGP) match the baseline or the intended delta |
| Path to a key destination | show ip route ADDRESS | Same next hop and outgoing interface as the baseline, unless the change targeted this path |
| Interface state | show ip interface brief | Every interface in scope up/up; nothing that was up is now down |
| Interface errors | show interfaces INTERFACE | Input errors and CRC counters do not rise between two reads a minute apart |
| Port-channel members | show etherchannel summary | Each port-channel in use (SU or RU) with the same members bundled (P) as the baseline |
| LACP partner | show lacp neighbor | The partner system ID on every member is the device you expect |
| ACL hits | show ip access-lists NAME | Matches rise on the intended entries; an explicit final deny does not climb (the implicit deny keeps no counter) |
| VLANs | show vlan brief | VLAN present, active and assigned to the intended access ports |
| Trunks | show interfaces trunk | VLAN allowed and forwarding on the intended trunks; allowed lists otherwise unchanged |
| Spanning-tree root | show spanning-tree root | Root bridge for each VLAN unchanged |
| Spanning-tree topology changes | show spanning-tree detail | include ieee|occurr|from | Topology-change count stops rising after the window; the last change came from a port you touched, if any |
| MAC table (Catalyst switches) | show mac address-table count | Counts per VLAN close to the baseline once hosts have returned |
| HSRP | show standby brief | Active and Standby routers per group as in the baseline; no group stuck in Init |
| VRRP | show vrrp brief | Master and Backup per group as in the baseline |
| Supervisor, route processor or stack redundancy | show redundancy | Peer in STANDBY HOT; redundancy mode as in the baseline |
| Stack members (Catalyst) | show switch | Every member Ready, with the same roles as the baseline |
| Software release | show version | Intended release on every device; uptime shows only the planned reload |
| Install state (install mode) | show install summary | Target release activated and committed (C); nothing left activated but uncommitted (U) |
| Hardware inventory | show inventory | Same modules and transceivers as the baseline |
| Log since the window | show logging | No new tracebacks, err-disabled ports, neighbor flaps or duplicate addresses |
Syntax varies by platform and software version; test in a lab first.
Cisco NX-OS pre- and post-checks
Create a checkpoint before every change. checkpoint NAME saves the configuration in a form NX-OS (opens in a new tab) can diff and roll back to, and show diff rollback-patch checkpoint NAME running-config shows exactly what changed since. Checkpoint names must be unique and must not start with the word system.
Add vrf all, or vrf NAME, to routing commands when your design uses VRFs. ACL hit counts appear per entry only when the ACL includes statistics per-entry.
| Check | Command | What good looks like |
|---|---|---|
| Unsaved changes before you start | show running-config diff | No differences between the running and startup configuration |
| Checkpoint before the change | checkpoint NAME | Checkpoint listed in show checkpoint summary |
| Configuration diff after the change | show diff rollback-patch checkpoint NAME running-config | Only the lines the change intended |
| BGP sessions | show ip bgp summary vrf all | Every neighbor that was established shows a prefix count; counts match the baseline or the intended change |
| BGP prefixes advertised | show ip bgp neighbors PEER advertised-routes | Only the prefixes your policy intends |
| OSPF neighbors | show ip ospf neighbors vrf all | Same neighbors in FULL (2WAY between two DROTHER routers is normal) |
| Routing table size | show ip route summary vrf all | Counts per protocol match the baseline or the intended delta |
| Interface state | show interface brief | Every interface in scope up; nothing that was up is now down |
| Interface errors | show interface counters errors | No error counter rises between two reads |
| Port-channel members | show port-channel summary | Each port-channel SU or RU, with the same members (P) as the baseline |
| LACP partner | show lacp neighbor | The partner system ID on every member is the device you expect |
| ACL hits | show ip access-lists NAME | Matches rise on the intended entries (per-entry counts need statistics per-entry in the ACL) |
| VLANs | show vlan brief | VLAN present and active on the intended ports |
| Trunks | show interface trunk | VLAN allowed and forwarding on the intended trunks |
| Spanning tree | show spanning-tree summary | Mode, and the VLANs this switch is root for, match the baseline |
| Spanning-tree topology changes | show spanning-tree detail | Topology-change counts stop rising after the window |
| MAC table | show mac address-table count | Counts close to the baseline once hosts have returned |
| HSRP | show hsrp brief | Active and Standby per group as in the baseline |
| VRRP | show vrrp | Master and Backup per group as in the baseline |
| vPC status | show vpc | Peer adjacency formed, keepalive peer alive, consistency status success, every vPC up |
| vPC consistency | show vpc consistency-parameters global | Local and peer values match for every parameter |
| Orphan ports | show vpc orphan-ports | No new orphan ports compared with the baseline |
| Supervisor redundancy (modular chassis) | show system redundancy status | Other supervisor in HA standby |
| Software release | show version | Intended release on every switch |
| Modules | show module | Every module in the same state as the baseline (active, ha-standby or ok) |
| Upgrade impact, before the window | show install all impact nxos bootflash:IMAGE | Impact per module is what the plan assumed (disruptive or non-disruptive) |
| Log since the window | show logging logfile | No new errors, flaps or crashes |
| Who changed what | show accounting log | Configuration commands in the window match the change |
Syntax varies by platform and software version; test in a lab first.
Arista EOS pre- and post-checks
Make the change in a configuration session. configure session NAME holds the change until you commit it, show session-config diffs shows exactly what the commit will do, and that diff is the one you attach to the ticket. Inside the session, commit timer 00:10:00 commits with an automatic rollback unless you confirm with configure session NAME commit before the timer expires. Both commits are state-changing and belong in the approved window.
ACL hit counts appear per entry only when the ACL includes counters per-entry.
| Check | Command | What good looks like |
|---|---|---|
| Unsaved changes before you start | show running-config diffs | No differences between the running and startup configuration |
| Pre-change configuration copy | copy running-config flash:CHG-NUMBER-pre.cfg | File saved in flash before the first change |
| Review the change before commit (inside the session) | show session-config diffs | Only the lines the change intends |
| BGP sessions | show ip bgp summary vrf all | Every peer that was Estab is Estab again, with prefix counts as in the baseline |
| BGP prefixes advertised | show ip bgp neighbors PEER advertised-routes | Only the prefixes your policy intends |
| OSPF neighbors | show ip ospf neighbor | Same neighbors in FULL (2WAY between two DROTHER routers is normal) |
| Routing table size | show ip route summary | Counts per source match the baseline or the intended delta |
| Interface state | show interfaces status | Every interface in scope connected; nothing newly notconnect or errdisabled |
| Interface errors | show interfaces counters errors | No error counter rises between two reads |
| Port-channel members | show port-channel dense | Same active members per port-channel as the baseline |
| ACL hits | show ip access-lists NAME | Matches rise on the intended entries (per-entry counts need counters per-entry in the ACL) |
| VLANs | show vlan | VLAN present and active on the intended ports |
| Trunks | show interfaces trunk | VLAN allowed and forwarding on the intended trunks |
| Spanning tree | show spanning-tree | Root bridge and port roles as in the baseline |
| MAC table for a VLAN | show mac address-table vlan VLAN-ID | Entries for the VLAN present again, close to the baseline count |
| VRRP | show vrrp | Master and backup per group as in the baseline |
| VARP virtual router | show ip virtual-router | Virtual addresses active on both switches (a VARP gateway is active on both) |
| MLAG status | show mlag | State active, negotiation connected, peer-link up; no inactive or active-partial MLAG ports |
| MLAG consistency | show mlag config-sanity | No inconsistencies reported |
| MLAG interfaces | show mlag interfaces | Every MLAG active-full |
| Supervisor redundancy (modular systems) | show redundancy status | Peer supervisor in standby and ready for switchover |
| Software release | show version | Intended release on every switch |
| Boot image | show boot-config | Boot image points to the intended release |
| Extensions | show extensions | Same extensions installed as before the upgrade |
| Reload cause | show reload cause | Only the planned reload |
| Log since the window | show logging | No new errors, flaps or crashes |
Syntax varies by platform and software version; test in a lab first.
Juniper Junos pre- and post-checks
Junos (opens in a new tab) has no running and startup split: the committed configuration is active, and it is what the device boots. Use the commit tools instead. commit check validates the candidate. commit confirmed 10 commits with an automatic rollback after 10 minutes unless you confirm with commit; it is state-changing, so run it only in the approved window. commit comment "CHG-NUMBER" ties the commit to the ticket.
After the change, show configuration | compare rollback 1 shows what the last commit changed. If the change took several commits, find the right rollback number in show system commit. set cli timestamp prints the time above every output, and appending | save /var/tmp/FILE to a show command keeps the output on the device.
| Check | Command | What good looks like |
|---|---|---|
| Uncommitted changes (configuration mode) | show | compare | Nothing pending in the candidate before you start |
| What the last commit changed | show configuration | compare rollback 1 | Only the lines the change intended |
| Commit history | show system commit | Commits in the window match the change, with the change number in the comment |
| BGP sessions | show bgp summary | Every peer that was Established shows active/received/accepted counts as in the baseline |
| BGP prefixes advertised | show route advertising-protocol bgp PEER | Only the prefixes your policy intends |
| BGP prefixes received | show route receive-protocol bgp PEER | Same prefixes as the baseline, or the intended delta |
| OSPF neighbors | show ospf neighbor | Same neighbors in Full (2Way between two DROther routers is normal) |
| Routing table size | show route summary | Counts per table and protocol as in the baseline |
| Interface state | show interfaces terse | Admin and link up on every interface in scope |
| Interface errors | show interfaces INTERFACE extensive | Input errors and framing errors do not rise between two reads |
| LAG members | show lacp interfaces | Every member collecting and distributing; partner as expected |
| Firewall filter counters | show firewall | Counters on the intended terms rise; discard counters do not jump (terms need a count action) |
| VLANs | show vlans | VLAN present with the intended interfaces |
| Spanning-tree root | show spanning-tree bridge | Root bridge as in the baseline; topology-change count stops rising |
| Spanning-tree ports | show spanning-tree interface | Port roles and states as in the baseline |
| MAC table size | show ethernet-switching table | count | Line count close to the baseline once hosts have returned |
| VRRP | show vrrp summary | Master and backup per group as in the baseline |
| MC-LAG peering | show iccp | ICCP connection to the peer established and liveness detection up |
| MC-LAG interfaces | show interfaces mc-ae | Local and peer status as designed; no member down |
| Virtual Chassis | show virtual-chassis | Every member present, with the same roles as the baseline |
| Routing Engine redundancy | show chassis routing-engine | One Routing Engine in the primary (master) role and the other backup, as in the baseline |
| Graceful switchover readiness (run on the backup Routing Engine) | show system switchover | Graceful switchover on; configuration and kernel databases ready |
| Software release | show version | Intended release on every member |
| System alarms | show system alarms | No new alarms |
| Chassis alarms | show chassis alarms | No new alarms |
| Log since the window | show log messages | last 200 | No new errors, flaps or crashes |
Syntax varies by platform and software version; test in a lab first.
Palo Alto Networks PAN-OS pre- and post-checks
Save a copy of the configuration before you start, then review the pending change with show config diff, or with Preview Changes in the commit dialog of the web interface, before you commit. test security-policy-match tells you which rule a flow would hit without sending traffic: run the same test flows before and after. Run the HA checks on both firewalls of a pair.
The routing commands below apply to the legacy routing engine. If the Advanced Routing Engine is enabled, the routing show commands differ; use the CLI reference for your release.
| Check | Command | What good looks like |
|---|---|---|
| Disable paging | set cli pager off | Complete output in your capture |
| Release and content versions | show system info | PAN-OS and content versions as intended, and the same on both HA peers |
| Pending changes before commit | show config diff | Only the changes you intend to commit |
| Pre-change configuration copy (configuration mode) | save config to CHG-NUMBER-pre.xml | Saved before the first commit |
| Commit result | show jobs all | Commit job finished with result OK; any warnings read and understood |
| HA state (both firewalls) | show high-availability state | Roles as in the baseline; running configuration synchronized |
| Rule lookup | test security-policy-match from ZONE-A to ZONE-B source SRC-IP destination DST-IP protocol 6 destination-port PORT | The intended rule and action for each test flow; flows through rules you did not touch match the same rules as before |
| Rule hit counts | show rule-hit-count vsys vsys-name vsys1 rule-base security rules all | Hits rise on new or changed rules; the cleanup or default deny does not start climbing |
| Drop counters | show counter global filter delta yes severity drop | No new drop reasons, or only those the change explains (delta shows the change since the previous run) |
| Sessions | show session info | Active session count in line with the baseline for the time of day |
| Interfaces | show interface all | Every interface in scope up, in the intended zone and virtual router |
| Routes (legacy routing engine) | show routing route | Routes for key prefixes as in the baseline |
| BGP (legacy routing engine) | show routing protocol bgp summary | Peers established, with prefix counts as in the baseline |
| OSPF (legacy routing engine) | show routing protocol ospf neighbor | Same neighbors in full state |
| IPsec tunnels | show vpn ipsec-sa | Security associations present for every tunnel in scope |
| System log since the window | show log system direction equal backward | No new HA, routing or interface errors (newest entries first) |
Syntax varies by platform and software version; test in a lab first.
How to decide the verdict
“Completed” is a workflow state, not an outcome. Record one of five verdicts, and decide in this order:
- Blocked. The change did not start, or stopped before its first state-changing step, because a pre-check or the go/no-go gate failed. Example: the pre-check found the standby firewall out of sync. The evidence is the failed pre-check.
- Rolled back. The change was made and then reversed, because the rollback trigger fired or the team chose to back out, and the post-rollback checks show the baseline restored. If the rollback did not restore the baseline, you have an incident: open one and record Degraded.
- Degraded. The change is in place, but the intent was only partly met, or something outside the intent changed and stayed changed. Example: an upgrade completed and reachability held, but a DHCP relay line disappeared from one of two cores. Record the regression, its owner and the fix.
- Inconclusive. No regression found, but the evidence is incomplete: no baseline for a device in scope, a post-check not run, the delayed check skipped. Say what is missing and what would close the gap.
- Verified. The intent was met, every invariant held, every difference between pre and post is explained by the intent, and the delayed check passed.
Any difference you cannot explain blocks Verified. “Probably noise” goes in the ticket as a question with an owner, not as a pass. See change verdict and completed is not verified.
Where to store the evidence
The change record is the record. Months later it is where the next engineer, the problem manager or an auditor will look, so the evidence belongs there, not in a chat thread or on a laptop.
- Attach the raw files: pre, post and delayed outputs per device, the configuration diff and the probe results, named consistently, for example
CHG-NUMBER_DEVICE_post_TIMESTAMP.txt. - Write a short work note: the verdict, one sentence of reason, the checks run, who verified and when.
- Close with the verdict, not with “successful”. If your ITSM close codes only offer success or failure, put the verdict in a field of its own, or at the start of the closing note, so it can be reported.
- Remove secrets (keys, community strings, password hashes) before attaching configurations. On Arista EOS (opens in a new tab),
show running-config sanitizedmasks them. - Link related records: the incident a failed change caused, or the problem record a Degraded verdict opens.