How to test firewall failover without an outage

What a failover test proves, and what it does not

A controlled failover, where you tell the active unit to step down, proves that the standby can take the active role, that configuration and session state were synchronized, and that the network around the pair follows the move: switch ports, VLANs, routing neighbors and VPN peers. It is the test that finds a standby that stopped synchronizing, a VLAN missing from the standby’s switch port, a routing adjacency that takes too long to form again, or a software or content mismatch between the two units.

It does not prove failure detection. A unit told to step down does not exercise link monitoring, path monitoring or heartbeat timers. Test those separately, by taking down a monitored interface or path in a lab or a later window, once the controlled failover passes.

Treat the test as a change: an approved maintenance window, a written plan, a rollback path and named people. A test designed to cause no outage can still cause one if the standby is not ready. Finding that out is the point of the test.

Prerequisites

• An approved maintenance window and a change record, with application owners told what they might see.
• Console or out-of-band access to both units. If the failover goes wrong, in-band management may go with it.
• A configuration backup of both units taken the same day.
• Matching software on both units and, on platforms with content updates (application, threat and antivirus signatures), matching content versions.
• Known preemption behavior. With preemption enabled, the preferred unit takes the active role back when it returns: a second transition you must plan and measure. Without it, the pair stays on the other unit until you move it.
• Probes running before you start: one probe per second, with timestamps, across every path that matters (inside to outside, through NAT, through each VPN tunnel, to key applications). On Linux, ping -D -O ADDRESS prints a timestamp on every line and reports each missed reply.
• At least one long-lived TCP session through the pair, such as an SSH session running a continuous command, to show whether established sessions survive.
• Success criteria agreed in advance, for example: “no established TCP session lost, no more than 3 consecutive probes lost on any path, every BGP neighbor established again within 60 seconds.”
• Synchronized clocks (NTP) on both units and on the probe hosts, so the timestamps line up.

Pre-checks: HA state, configuration sync, session sync, link and path monitoring

Run these on both units and stop if any fails. A failed pre-check makes the test Blocked, and the finding alone is worth the window.

1. HA state. One unit active and the other in a ready standby state (passive, Standby Ready or secondary, depending on the platform), with no unexplained transitions in the recent HA history.
2. Configuration sync. The pair reports its configuration as synchronized, or its checksums match.
3. Session sync. The state-synchronization link is up and its counters rise between two reads; where the platform shows it, the standby’s session count is in line with the active unit’s.
4. Link and path monitoring. Every monitored interface and path target is up and reachable from both units. A monitored interface that is down on the standby can make it ineligible, and the failover will either not happen or bounce back.
5. The network around the pair. The switch ports facing the standby carry the same VLANs as those facing the active unit, and every routing neighbor and VPN peer that must form again after the move is known and watched.

The controlled failover and the fallback

1. Announce the start on the bridge or channel and note the time. Start the probes and the long-lived session.
2. Capture the pre-state on both units: HA status, session counts, routing neighbors, VPN tunnels.
3. Trigger the failover on the active unit with the platform command in the tables below. This is state-changing: run it only in the approved maintenance window, with the rollback decision owner on the call.
4. Watch both units change state, and watch the probes. Note the time of the first lost probe and of the first successful probe after the last loss, per path.
5. Hold on the new active unit for an agreed soak time, 15 to 30 minutes, and run the application checks while it carries real traffic: user access, NAT, VPN, authentication, logging to your collector, management access.
6. Fall back. Return the original unit to service and move the active role back, through preemption or with a second controlled failover. Measure the fallback exactly as you measured the failover; it can behave differently.
7. Run the post-checks on both units and record the results.

Write the abort rule before the test, for example: “if any critical application check fails and has not recovered 2 minutes after the failover, move the active role back immediately.” Name who makes that call. If the fallback also fails, the next step in the plan is console access to both units and the escalation path, not improvisation.

What to measure

• Convergence time: from the moment you triggered the failover to the first successful probe after the last lost one, per path. Report the slowest path, not the average.
• Dropped probes: lost probes per path. At one probe per second, the count is roughly the seconds of loss.
• Session loss: whether the long-lived TCP sessions survived, and the active session count on the new active unit compared with the count before. Stateful failover is designed to keep established TCP sessions; applications that reconnect on their own can hide losses, so check their logs too.
• Routing convergence: when each BGP or OSPF neighbor of the pair was established again, from the neighbor’s view as well as the firewall’s.
• VPN recovery: whether tunnels stayed up or had to negotiate again, and how long that took.
• The same numbers for the fallback, recorded separately.

Put the numbers next to the success criteria. A failover and fallback that met every criterion is Verified. One that worked with a gap, such as a tunnel that took 4 minutes to return, is Degraded, with an owner for the fix.

Post-checks and the record

• Both units in the roles you intended to end with, with configuration and session sync healthy again.
• No monitored interface or path left failed, and no new alarms or log errors on either unit.
• Routing tables, VPN tunnels and session counts back in line with the pre-test capture.
• Any manual-failover flag cleared (a Juniper (opens in a new tab) SRX chassis cluster keeps one until you reset it).
• The record in the change ticket: timestamps, probes lost per path, convergence times, sessions that survived, issues found and the verdict (Verified, Degraded, Rolled back, Inconclusive or Blocked).

Palo Alto Networks PAN-OS HA

In an active/passive pair, suspending the active firewall hands the active role to the passive one. request high-availability state functional returns the suspended firewall to service. It comes back passive, unless preemption is enabled on both firewalls and it has the higher priority; then it takes the active role back. On PAN-OS, the lower device priority value is preferred.

The routing command below applies to the legacy routing engine; with the Advanced Routing Engine enabled, use the routing commands for your release.

Cisco Secure Firewall ASA failover

Active/standby failover on the ASA does not preempt: after a failover, the new active unit stays active until you move the role back. The standby unit reaches Standby Ready only after it has synchronized the configuration from the active unit, so a unit stuck in an earlier state is itself a finding. failover exec mate, followed by a show command, runs that command on the peer, so you can read both units from one session.

Juniper SRX chassis cluster

Test the data-plane redundancy groups (1 and above) first. Redundancy group 0 holds the two Routing Engines, so failing it over moves the control plane as well; plan it as a separate step. A manual failover sets the new primary’s priority for that group to 255 and keeps it there until you reset it, and Juniper (opens in a new tab) requires request chassis cluster failover reset before another manual failover. After the reset, a group with preempt enabled can move back to the higher-priority node on its own.

Fortinet FortiGate HA (FortiOS)

The checks below confirm the cluster’s state and synchronization. How you move the primary role depends on the FortiOS release and the HA settings. With override disabled, which is the default, FortiOS weighs HA uptime before priority when it chooses the primary, so raising a priority alone may not move the role; with override enabled, priority is weighed before uptime. FortiOS 6.4 and later also document a forced failover for testing: follow Fortinet (opens in a new tab)’s article “Force HA failover for testing and demonstrations” for your release, and treat it as state-changing.