Buyer’s guide
Evaluating agentic AI for network operations
Ten questions, 26 criteria, a 45-day test design and a scorecard. It describes practices and tests, not products; every criterion can be met by more than one vendor, or by a well-run internal team. Free to copy and adapt.
Why evaluating an agent is different from evaluating a tool
A monitoring tool that is wrong shows you a bad chart. An agent that is wrong can escalate the wrong team, close the wrong ticket or, if it is allowed to act, change a production device. Three things follow. Confidence is not correctness: a fluent explanation of the wrong root cause is worse than no explanation, because it is believed. A demo proves nothing about your network: an agent’s value depends on the context it has, so the only meaningful test runs on your environment, with your incidents, against your baseline. Governance is a feature, not paperwork: what the agent may do, how it asks, how it proves what it did and how it is stopped deserve the same rigor as accuracy.
Start with your own baseline
Without a baseline, every claim (yours and the vendor’s) is an opinion. A week is enough.
| Measure | How to get it in a week | Why it matters |
|---|---|---|
| Time to detect a high-impact incident | For the last 10 P1/P2s: incident start vs. ticket open | The 2026 New Relic (opens in a new tab) Observability Forecast (opens in a new tab) reports 41 minutes on average; many teams are surprised by their own number |
| Time to an evidence-backed root cause | Ticket open to the moment someone wrote down the cause with evidence, not to restore | Restore time hides the diagnosis problem |
| Engineer-minutes per investigation; people involved | Ask the engineers who handled the last five incidents | This is the capacity an agent can return |
| Tools opened per investigation | Count them for one recent incident | The count is usually higher than teams expect |
| Repeat rate | Share of incidents that recurred within 90 days with the same cause | The “restored, not fixed” number |
| Change verification practice | For the last 20 changes, how many had documented post-change verification with evidence? | The “completed, not verified” number |
| Cost per hour of a high-impact outage | Revenue at risk × share lost + productivity of affected staff + direct costs | 57% of organizations said their most recent major outage cost more than $100,000 (Uptime Institute (opens in a new tab), 2026) |
Ten questions every vendor should answer
The first five follow the requirements analysts have set out for production-grade agentic NetOps software (Gartner (opens in a new tab), Market Guide for Agentic NetOps Software, May 2026); the second five are practical tests for a time-boxed evaluation.
| Question | What a good answer sounds like | Red flag |
|---|---|---|
| 1. Which of our vendors, platforms and versions do you support, and how do you discover them? | A specific list; discovery in days; an honest statement of what is not supported | “Any vendor” with no list |
| 2. How do you use our organizational context: tickets, runbooks, design documents, change records? | Named sources, how they are ingested and refreshed, an example only your documents could answer | Context means “we read the configs” |
| 3. What autonomy modes exist, per action and per role? | Recommend only; approve then act; act and notify. Configurable; the default is recommend only | One mode; “you can turn it off” |
| 4. What is validated before an action or change executes? | Redundancy, reachability, security policy and dependencies checked with reasons shown; blast-radius limits | Syntax checking only |
| 5. Can we trace every conclusion to the exact evidence? | Yes: outputs, logs, diffs and ticket text linked from each conclusion; shown live | A confidence score with no link to sources |
| 6. What happens when the evidence is insufficient? | It says so and proposes the next diagnostic step | It always has an answer |
| 7. How long does an investigation take: P50 and P95, measured? | Instrumented numbers from pilots; known patterns vs. novel investigations distinguished | A single demo timing; no P95 |
| 8. How do you verify a change your platform did not execute? | Pre/post state captured and compared for any change window; a verdict written to the change record | Verification only for changes run through the platform |
| 9. How does the pilot end? | A baseline in week 0, written success criteria, a business review with a decision date, BUY / EXTEND / STOP | An open-ended evaluation or a free production period |
| 10. Where does our data go, and who can see it? | Deployment options that fit your policy; connection direction, ports and encryption documented; retention, deletion and model-training policy in writing | “It is secure” without documents |
The 26 criteria and how to test them in 45 days
Score each vendor 0–5: 0 not available · 1 claimed, not shown · 2 shown on vendor data · 3 shown on your data · 4 measured in your pilot · 5 measured and meets your target. Suggested must-haves are marked; weights are yours to set. Any must-have scored below 3 removes a vendor from the shortlist regardless of the total, unless the vendor commits in writing to a delivery date before your production decision; record those as R (roadmap, committed) and re-score them at the business review. Criteria marked must-have for production must reach 3 or better before the production decision, not before the pilot.
Multivendor coverage and context
| # | Criterion | What to look for | How to test it in 45 days |
|---|---|---|---|
| 1 | Works across our actual estate without per-site hardware must-have | Discovery of at least 95% of in-scope devices across two or more vendors, cloud controllers and firewalls; an explicit list of unsupported platforms and versions | Run discovery on the pilot domain in week 1; compare with the CMDB and the site list |
| 2 | Builds and maintains topology and dependencies automatically must-have | Device roles, HA pairs, paths and service dependencies discovered, not drawn by hand; kept current | Topology accuracy of at least 90% validated by the architect; a deliberate change reflected within the agreed time |
| 3 | Uses our organizational context | Tickets, runbooks, SOPs, design documents, change records and naming conventions inform investigations | Ask a question only answerable from your runbook or design document |
| 4 | Coexists with the tools we own and writes back to them must-have | Monitoring, ITSM, PAM and on-call remain; evidence and outcomes written to the ticket or change record automatically | Incident record enriched in the ITSM without manual steps |
| 5 | Read-only by default with least-privilege credentials through our PAM must-have | No standing write access; credentials brokered and revocable by us | Security review of the credential model; revoke a credential and confirm loss of access |
Reasoning quality and evidence
| # | Criterion | What to look for | How to test it in 45 days |
|---|---|---|---|
| 6 | Finds the root cause on incidents with known answers, without being told the answer must-have | Correct or materially useful root cause; blast radius and impacted services identified | Replay 5–10 historical incidents; score at least 80% correct or useful |
| 7 | Every conclusion traceable to the exact evidence must-have | Command outputs, logs, configuration diffs and ticket text linked from each conclusion | Click through from conclusion to source for ten random findings; at least 90% traceable |
| 8 | States uncertainty and the next-best diagnostic step | No confident guessing when evidence is insufficient; proposes what to collect next | Remove a data source and repeat an investigation |
| 9 | Time to an evidence-backed answer reported as P50 and P95 | Instrumented across all investigations; consistent, fast answers for known patterns; no hidden tail | Instrumentation report for every pilot investigation |
| 10 | Handles novel incidents, not only known patterns | Dynamic investigation that adapts to what it finds, versus fixed workflows only | Include at least two incidents with no existing runbook in the replay set |
| 11 | Correlates across domains and answers "what changed" | Network, firewall, WAN and cloud correlated in one investigation with change history | One cross-domain incident in the replay set |
Autonomy, governance and safety
| # | Criterion | What to look for | How to test it in 45 days |
|---|---|---|---|
| 12 | Selectable autonomy modes per action type must-have for production | Recommend only; approve then act; act and notify, configurable per runbook and per role | Run the pilot in recommend-only mode; demonstrate one approval-gated runbook |
| 13 | Pre-execution validation of any action or change must-have | Checked against redundancy, reachability, security policy and dependencies before execution, with reasons | Submit a deliberately risky change; expect it to be flagged with reasons |
| 14 | Full, exportable audit trail must-have | Who, what, when and why for every query and action | Export the audit log for the pilot period |
| 15 | Rollback and blast-radius limits for actions | Reversible actions; limits on scope per execution | Demonstrate rollback on a runbook in a lab or maintenance window |
| 16 | Role-based access, SSO and tenant isolation appropriate to the deployment model must-have | RBAC tiers, SAML/OIDC, isolation controls documented | Security questionnaire and a live RBAC check |
Change and resilience assurance
| # | Criterion | What to look for | How to test it in 45 days |
|---|---|---|---|
| 17 | Verifies changes made anywhere (pre/post) with the outcome recorded in the change record must-have for production | Not only changes executed through the tool; state compared before and after; one outcome (Verified, Degraded, Rolled back, Inconclusive or Blocked) written back with the evidence | Verify one real change window executed by your own team |
| 18 | Proves redundancy continuously | Standby health, failover readiness and backup-path capacity checked with evidence | For each redundant pair and backup path in the pilot domain, the platform states whether readiness is proven, with the evidence (sync state, standby reachability, capacity versus observed load); where proof needs a failover test, it says so and proposes the test for a maintenance window |
| 19 | Prioritizes drift, posture and CVE (opens in a new tab) exposure by topology and business impact | Findings ranked by where the device sits and what it carries; features in use, not just versions | Findings review with the architect; false-positive rate |
| 20 | Turns root causes into prevention | Proposes the verified fix, updated alert or runbook that stops the repeat | For each replayed incident, does the tool propose the prevention? |
Deployment, security and data
| # | Criterion | What to look for | How to test it in 45 days |
|---|---|---|---|
| 21 | Deployment options that fit our policy must-have for production | SaaS, our cloud account, or on-prem / air-gapped; documented connection direction, ports and encryption | Security packet review with a data-flow diagram |
| 22 | Clear data handling must-have | What is collected, retention, deletion, whether our data trains models, LLM endpoint options | Written answers in the security packet; contractual commitments |
| 23 | Security program evidence | SOC 2 (opens in a new tab) or equivalent status, penetration-test summary, subprocessor list | Documents provided within a week of the request |
Proof, economics and partnership
| # | Criterion | What to look for | How to test it in 45 days |
|---|---|---|---|
| 24 | Measured before/after results on our baseline in a time-boxed pilot must-have | Baseline in week 0; success criteria in writing; business review with a decision date; BUY / EXTEND / STOP | Insist on the structure; decline open-ended evaluations |
| 25 | Value model built from our numbers, with hard-dollar and risk-adjusted value shown separately | Uses our baseline; states the realization factor; vendor targets labeled as targets until measured | Review the model with finance; require the measured version at the business review |
| 26 | References, roadmap transparency and responsiveness | Two reference calls; roadmap items labeled as such; defect turnaround during the pilot. For an early-stage vendor without production references: founder access, design-partner calls under NDA once consented, and a paid pilot with a real STOP option | Reference calls in week 2; issue log reviewed at the business review |
Compare against the real alternatives
Most evaluations are lost to three alternatives, not to another vendor. Test them the same way.
- Doing nothing. Count last quarter’s P1s without a verified root cause and last month’s changes closed without post-change evidence.
- Building it. Price the lifecycle, not the demo: collectors per vendor, model evaluations, guardrails, integrations, security review and on-call for the agent.
- Your vendors’ and your ITSM’s own AI. Run one incident that crosses vendors and one change made outside the vendor’s tool, and score both with the criteria above.
Designing the evaluation
| Element | Standard |
|---|---|
| Entry criteria | A named technical champion; the economic decision-maker committed to the business review; the problem in your words; a baseline; written success criteria; your purchase process mapped; a commercial hypothesis; a decision date |
| Scope | One production domain with enough complexity that context matters (50–250 devices, 2–3 vendors, 1–2 sites); ITSM plus one telemetry source plus configuration access; read-only credentials via your PAM |
| Weeks 0–6 | Baseline and criteria → connect and discover → incident replay with known answers → findings review → one real change window verified → measurement and economics → business review with a decision |
| Commercial | A paid proof of value credited against year one, or a fee exchanged for reference rights: either way both sides are committed to a decision |
Reading the economics honestly
- Two numbers, not one. Hard-dollar value (engineer capacity at loaded cost, plus anything you would actually stop paying for) and business impact (outage minutes avoided × your cost per minute). Ask for them separately.
- A visible realization factor. Agree the share of modeled outage cost you will count with finance before the pilot.
- Targets are not results. Vendor improvement figures belong in a column labeled “target” until your pilot measures them.
- Capacity is not headcount. Hours returned are FTE-equivalents redeployed, deferred hiring or reduced contractor spend: say which.
- Break-even framing. Divide the annual subscription by the cost of one high-impact outage. If the answer is less than one outage a year, the pricing conversation is about confidence, not value.
- Pricing model. Understand how the price scales with adoption and avoid models that penalize the wider use you want.
Red flags
- A demo on the vendor’s data as the proof point, with no path to your environment
- No baseline and no decision date: an evaluation designed to continue
- A single response-time figure and no P95
- Root causes without links to evidence; confidence scores instead of sources
- “Autonomous” without selectable modes, approval gates and an audit trail
- Verification only for changes the platform itself executed
- Vague answers on data residency, retention or model training
- A pricing model that penalizes wider adoption, or that cannot be explained in a sentence
- Statistics without a source and a year
- A vendor who criticizes other vendors instead of describing tests you can run
Sources: Gartner (opens in a new tab), Market Guide for Agentic NetOps Software (May 19, 2026). New Relic (opens in a new tab), 2026 Observability Forecast (opens in a new tab) (September 2026). Uptime Institute (opens in a new tab), Annual Outage Analysis 2026 (opens in a new tab) (May 2026). Gartner does not endorse any vendor, product or service depicted in its research publications.
Score REAP like any other vendor.
Request the guide and the weighted scorecard as files, then run the 45-day test on your network. We will also send REAP’s own score on these criteria (available today, in pilots, or roadmap with a date) with the evidence behind each.