Updated: 2026-01-05 By: Ari Heljakka

Short answer

An agent monitoring tool is a runtime contract: it watches the live system, raises an alert when something is wrong, and gets paged engineers to the right span fast enough to matter. The selection criteria for monitoring are different from the criteria for observability platforms, which are oriented around investigative reach and lineage. Monitoring is oriented around alert quality, SLO discipline, page routing, on-call workflow, and mean time to detect. What the buyer is paying for is recovery time, not feature breadth. A tool that surfaces the right alert at 3 AM with the right context attached, against an SLO the team can defend, is worth more than a tool with a wider feature catalogue that pages on noise. The ranking that has held up in practice is alert quality first, then integration with the existing on-call rotation, then contract terms.

Key facts

Definition: A monitoring buyer's guide is a structured comparison of agent monitoring tools against runtime selection criteria: alert quality, SLO support, on-call integration, page routing, mean time to detect, and the contract that holds when the tool fails.
When to use: Whenever a team is selecting a monitoring tool for production agent traffic and the choice will sit in the on-call rotation for years.
Limitations: Demos rarely expose alert quality at scale. Most of the criteria below require a representative load test, not a slide deck.
Example: A team picks the monitoring tool that wins the demo, then spends six months tuning out false positives. The on-call rotation routes around the tool. The actual selection criterion, page-worthy alerts per week, was never measured.

Key takeaways

Monitoring is a runtime contract. The deliverable is a page that gets the right engineer to the right span fast.
Alert quality is the first and largest variable. A tool with high false-positive rate trains the team to ignore it.
SLO discipline is what separates a dashboard from a monitoring tool. Without service-level objectives and a defensible error budget, every alert is opinion.
On-call integration matters more than alert features. A monitoring tool that does not route into the team's pager and escalation path is not in the rotation.
Mean time to detect is the operational metric the buyer should ask the vendor to guarantee. Mean time to resolve depends on the team; mean time to detect depends on the tool.
The contract terms that pay off are the ones that hold during a real incident, not the ones that look good on the pricing page.

Definition

An agent monitoring tool is the runtime watch on a production agent system. It ingests live signal (traces, evaluator scores, error rates, latency, tool outcomes), evaluates it against thresholds or SLOs, raises an alert when a threshold is breached, and routes the alert into the team's on-call workflow. Its primary deliverable is timely, actionable notification.

A buyer's guide for agent monitoring is the set of selection criteria a team uses to choose this tool, weighted by their operational setup (traffic volume, on-call structure, SLO maturity). The criteria are runtime-oriented: how good are the alerts, how quickly does it detect a real failure, how cleanly does it integrate with the pager, and what does the contract look like when the tool itself has an outage.

This is distinct from observability buyer criteria, which are oriented around investigation, lineage, and retention. The two tools may live in the same platform; the buying decisions have different shapes.

When this matters

Monitoring selection criteria become decisive when:

The agent system is in production and a page is the difference between a bad five minutes and a bad incident report.
The team has an on-call rotation and the monitoring tool is going to sit inside it.
The product has explicit SLAs to customers or implicit service-level expectations from the business.
The agent traffic is high enough that noisy alerts will train the team to ignore them.
A regulator, a board, or a customer contract will ask "how long did it take you to know."

For an early prototype with no users, monitoring is premature; observability suffices. Once the agent is serving paying traffic, the monitoring contract is the difference between an outage that is detected at 3 AM and an outage that surfaces in a Monday morning support queue.

How it works

Alert quality

Alert quality is precision and recall on real incidents. The criteria that matter:

Precision at the page. When the tool pages, is the underlying signal a real problem. A tool with thirty percent false-positive rate trains the on-call to suppress.
Recall on real incidents. When something real happens, does the tool page at all. Missed incidents are worse than noisy ones in the long run.
Signal composition. Can multiple weak signals (latency creep, evaluator drift, tool-error rate) be combined into a single, high-precision alert. Most real failures are visible in two or three signals before any single threshold breaches.
Suppression and grouping. When fifty spans share a failure cause, the tool should page once with the cause attached, not fifty times with the symptom.
Anomaly versus threshold. Static thresholds are tunable and predictable. Anomaly detection is opaque and harder to defend in an incident review. Most teams use both; the buyer should ask which is the default.

SLO discipline

A monitoring tool without first-class SLOs is a dashboard with alerts on top. The criteria:

SLO authoring. Can the team define an SLO (for example, 99 percent of agent responses scored above 0.8 on the relevance evaluator, over a 28-day window) in versioned configuration.
Error budget tracking. Does the tool track the remaining budget over the window and surface burn rate, not just a binary pass-fail.
Burn-rate alerts. Does the tool support fast-burn and slow-burn alerts so the team is paged on a meaningful drop, not on every flicker.
Composite SLOs. Can SLOs be defined across multiple dimensions (latency, evaluator score, success rate) so that the operational target is a vector, not a single number.

On-call integration

The monitoring tool is only in the rotation if it integrates with the rotation. The criteria:

Pager integration. Native integration with the team's incident management platform, with proper severity mapping.
Escalation policies. Does an unacknowledged page escalate. Does the tool respect rotation handovers and quiet hours.
Runbook attachment. Can a runbook be attached to an alert so the on-call has next steps without context-switching.
Acknowledgement and resolution. Does the acknowledgement flow back to the monitoring tool, or does the alert keep firing.
Postmortem hooks. Can an incident report be generated from the alert, including the relevant spans and evaluator scores, for the postmortem.

Mean time to detect

Mean time to detect (MTTD) is the operational metric the monitoring tool can affect most directly. The buyer should ask:

Detection latency. How quickly after a real failure starts does the tool fire its first true-positive alert. Sub-minute on hot paths is normal; minute-scale on long-window SLO burns is acceptable.
Trace-to-page time. From the moment a trace is ingested to the moment a page lands, what is the budget. Some tools batch traces in five-minute windows; that is invisible until an incident.
Sampling implications. A tool that samples 1 in 100 traces will miss low-volume failures entirely. The buyer should ask about the relationship between sampling rate and MTTD on rare failure modes.

Operational fit and contract terms

The contract terms that survive a real incident:

SLA on the monitoring tool itself. A monitoring tool with a weaker SLA than the system it watches is a bad bet. Ask about the SLA, the credit policy, and the past 12 months of incidents on the vendor status page.
Status page transparency. Does the vendor publish a real status page, with incident postmortems. Tools that hide their incidents will not partner during yours.
Data residency and retention. Trace data and evaluator scores are sensitive. The buyer should ask about data residency, retention defaults, and deletion semantics.
Pricing predictability. Per-trace, per-alert, per-seat, and per-evaluator pricing axes; the buyer should model a realistic workload and ask for a quote, not a list price.
Egress and lock-in. Can the team export their alert history, their SLO definitions, and their span store. OpenTelemetry GenAI semantic conventions are the shared schema that makes export meaningful.
Termination assistance. A written plan for what happens when the contract ends.

Example

A team running a customer support agent at moderate scale runs the monitoring selection against three tools.

Operational profile:

1.5 million agent conversations per month.
Five-person on-call rotation, integrated with an incident platform.
Two SLOs: 99 percent latency under 4 seconds; 95 percent of responses scored above 0.8 on the policy-adherence evaluator over a 7-day window.
Mean time to detect target: under 2 minutes for hot-path failures, under 10 minutes for evaluator drift.

Selection results:

Criterion	Tool A	Tool B	Tool C
Alert precision in trial	Low (noisy)	High	Medium
SLO support	Static thresholds only	First-class SLOs, burn-rate alerts	SLOs, no burn-rate
On-call integration	Native + escalation	Native + escalation + runbook attach	Native only
MTTD on hot path	30 seconds	45 seconds	90 seconds
MTTD on evaluator drift	Not detected	8 minutes	15 minutes
Vendor status page	Sparse	Transparent + postmortems	Transparent
Export and OTLP	Proprietary store	OTLP export, alert history export	OTLP read, no alert history export

Outcome: Tool B is selected. Tool A wins the demo but loses the trial on alert noise. Tool C is acceptable but lacks burn-rate alerting and runbook attachment, both of which the team values for the on-call workflow. The buyer's guide justifies the procurement conversation that follows.

Monitoring versus observability buyer's guides

The criteria above are runtime. An observability buyer's guide weights different criteria more heavily: lineage queries, retention horizons, span attribute schema, replay, and investigative search. A team may select different vendors for monitoring and observability, or one vendor for both if the trade-offs work out.

Who should not buy a dedicated monitoring tool

Teams without an on-call rotation, without explicit SLOs, or without enough production traffic for noise to matter usually get more from the alerting features of their observability platform than from a dedicated monitoring tool. A dedicated tool pays back more once the team has the on-call structure and SLO discipline to act on the signal it provides.

Where each category is stronger

Dedicated monitoring tools win on alert quality, SLO sophistication, and on-call workflow integration. They are weakest at investigation.
Observability platforms with monitoring features win when the team is small enough that one tool is preferable to two, and when the alerting needs are simple enough that thresholds suffice.
In-house monitoring stacks (a metric store plus a paging system) win on control and cost. They are weakest at agent-specific signals like evaluator drift, which require integration the team will have to build.

Limitations

Demos do not reveal alert quality. The only reliable test is a trial under representative load with the team's existing on-call rotation.
Vendor MTTD numbers are not yours. A vendor's reported MTTD averages across customers and workloads. Estimate yours against your traffic.
SLOs depend on calibrated evaluators. A monitoring tool's SLO on an evaluator score is only as good as the evaluator's calibration. Evaluator drift is a monitoring concern in its own right.
Anomaly detection is hard to defend in an incident review. Static thresholds are slower to evolve but easier to explain to a board.
The contract that matters is the one during an incident. Vendor responsiveness during a real outage is the line item the buyer cannot evaluate before signing.

Evidence and sources

Google SRE Workbook chapter on alerting and SLOs, https://sre.google/workbook/alerting-on-slos/, for the burn-rate alert design that good monitoring tools support natively.
OpenTelemetry GenAI semantic conventions, https://opentelemetry.io/docs/specs/semconv/gen-ai/, the shared schema that makes export and tool replacement meaningful.
"Practical Alerting from Time-Series Data," Google SRE Book, https://sre.google/sre-book/practical-alerting/, for the precision-versus-recall trade-offs at the heart of alert quality.

FAQ

What is the difference between monitoring and observability? Monitoring watches the live system and pages on a breach. Observability lets the team investigate after the page. A team typically needs both; the criteria for selecting each differ.

Is OpenTelemetry support enough? For instrumentation, yes. For monitoring, OpenTelemetry plus a metric store plus an alerting layer is the underlying stack. The monitoring tool's value over a raw OTLP pipeline is alert quality, SLO discipline, and on-call integration.

How do I trial a monitoring tool in a way that actually predicts on-call performance? Run it in parallel with the existing alerting for 30 days. Compare its true-positive and false-positive rates on real incidents. Confirm the on-call workflow integration in a non-emergency drill. The trial budget is short next to the contract length.

What about cost? Model the workload. Most teams underestimate per-trace and per-alert costs and overestimate per-seat. A predictable per-trace tier with caps is preferable to opaque enterprise pricing.

Should monitoring run on every trace or sampled traffic? Hot-path latency and error metrics should be unsampled. Evaluator scoring is typically sampled for cost. The monitoring tool should make the sampling visible in the SLO, not hide it.

Who owns the monitoring tool inside the team? Most commonly the platform team or the on-call team. Engineers consume alerts; the platform team owns SLO definitions and tool configuration.

Agent Monitoring Buyer's Guide: Selection Criteria