Scorable logo

Developer's Guide to Agent Observability: What Matters

Developer's Guide to Agent Observability: What Matters

Updated: 2026-02-02 By: Ari Heljakka

Short answer

Most agent observability evaluations ignore the part that decides whether the platform gets used: the developer experience. The SDK that takes ten lines to instrument a chain gets adopted; the one that needs a context manager around every call does not, even if no one says so. The platform that mirrors production traces in local dev gets debugged against; the one that requires a deploy to see anything gets bypassed. The CI hook that exposes a scorecard delta in the PR view gets respected; the dashboard link in a Slack message does not. A developer's guide to agent observability is about the surfaces engineers touch every day: SDK ergonomics, framework integration, local-dev replay, trace search, evaluator authoring, CI integration. The platform that wins on these is the one that survives the next six months.

Key facts

  • Definition: Developer-facing agent observability is the set of interfaces (SDK, CLI, local-dev tools, CI hooks, trace and evaluator UIs) that engineers use to instrument, debug, evaluate, and gate agent code. It is the operational shell around the platform's data model.
  • When to use: Whenever the platform is meant to be used by application engineers, not only by an ops team or a compliance team. Adoption follows ergonomics; ergonomics is rarely on the demo slide.
  • Limitations: A good SDK does not fix a weak data model. A clean local-dev experience does not fix bad calibration. Ergonomics is necessary, not sufficient.
  • Example: Two platforms ship the same evaluator catalogue. One requires importing six modules and wrapping every call. The other ships a decorator and an auto-instrumenter. Six months later, one is the team's default; the other is a story about why the team rolled their own.

Key takeaways

  • Adoption is the dominant signal. A platform that lands the daily edit-debug loop wins every other comparison by default.
  • Auto-instrumentation for the team's actual orchestration framework, not just for popular ones, decides how much of the codebase shows up in traces.
  • Local-dev replay (the ability to re-run a production trace against a new prompt locally) is the single most under-marketed feature.
  • Evaluator authoring should feel like writing a function, with versioning, calibration data, and a test harness. Anything heavier becomes a separate codebase nobody owns.
  • CI integration that surfaces a scorecard delta in the PR view changes how prompts are reviewed. CI integration that emits a JSON blob into a log does not.

Definition

Developer-facing agent observability is the part of an observability platform that engineers interact with during their normal work: writing application code, debugging a failing trace, authoring an evaluator, running a local check, opening a PR, reviewing a colleague's PR. The touchpoints are SDKs, CLIs, IDE plugins, local-dev tools, the trace viewer, the evaluator catalogue, and the CI hooks.

A platform can have an exemplary data model and a weak developer surface. The data model decides what is possible; the developer surface decides what gets done.

When this matters

The developer experience becomes the dominant factor whenever:

  • The platform is supposed to be used by every engineer on the team, not by a centralised ops or eval group.
  • The team's iteration loop is fast (multiple prompt or agent changes per day).
  • The agent is built in a framework that does not have a first-class integration.
  • Debugging requires touching production traces, not just local logs.
  • The team's confidence in any deploy depends on a scorecard the engineer can see in the PR view.

If the platform is used only by a small ops or eval team, ergonomics still matters but is less critical. If every engineer is expected to author evaluators, instrument new code paths, and gate their own PRs, ergonomics is the platform.

How it works

SDK ergonomics

The SDK is the daily contact surface. Useful properties:

  • One-line instrumentation. A decorator or context manager covers the common case (LLM call, tool call, chain step). Auto-instrumentation for the team's framework is the default.
  • Idiomatic to the language. Pythonic in Python, typed in TypeScript, no surprising globals. The SDK should look like the codebase it lives in.
  • Quiet by default. No noisy logs, no surprising network calls, no global state mutation. A failing platform should not break the application.
  • Composable, not opinionated. The SDK should let the application combine spans, attach attributes, and emit custom events without fighting the framework's primitives.
  • Local-first. The SDK works against a local collector for development. Going to production is a configuration change, not a rewrite.
  • Versioned. Breaking changes are rare and signposted. The team upgrades on its own schedule.

A small example test: how long does it take a new engineer to add observability to a new agent route. A platform whose answer is "ten minutes" gets adopted; one whose answer is "a half-day with the docs open" does not.

Framework integration

Most agents live inside an orchestration library or a thin internal framework. The integration choices a platform supports:

  • Auto-instrumentation. The platform's SDK ships first-class integrations for the popular orchestration libraries. Spans, runs, and traces appear without manual wrapping.
  • OpenTelemetry GenAI conventions. The platform reads OTLP and uses the GenAI semantic conventions. Any framework that emits OTLP works without a custom integration.
  • Custom orchestration hooks. For internal frameworks, the SDK exposes primitives to wrap spans, retries, tool calls, and decision points. Documentation includes a worked example.

A useful adoption test is whether a non-trivial custom agent can be instrumented in under a day. A platform that can only handle the popular frameworks is brittle the day the team builds something new.

Local-dev experience

The local edit-debug loop decides whether the platform stays in the daily workflow:

  • Local collector. The SDK targets a local OTLP collector by default in dev. Traces appear immediately in a local UI or CLI.
  • Production trace replay. A specific production trace can be downloaded, edited (new prompt, new model, new retrieved context), and re-run locally. The diff against the original trace is queryable.
  • Evaluator dry-run. Evaluators can be run locally against a single trace or a calibration sample. Authoring a new evaluator does not require a deploy.
  • CLI parity. The same things the UI shows are scriptable from a CLI. Reproducing an issue means sharing a command, not a screen share.

Replay is the feature that separates platforms that engineers actually use from platforms they tolerate. Without it, every prompt change is a leap of faith.

Trace search and trace UI

Once traces exist, the question is how fast a developer can find the trace they need:

  • Trace ID propagation. A request that fails in production produces a trace ID that lands in the application log. The developer pastes it into the platform and gets the trace.
  • Attribute search. Traces are searchable by attributes the team chose to expose (user ID, agent version, tool name, score). The schema is open enough to add new attributes without a platform release.
  • Span tree quality. The span tree shows the agent's decisions in a shape the engineer recognises: which tool was called, which retrieved context fed which call, which decision came from which LLM. Anything more abstract is harder to debug.
  • Diff view. Two traces can be compared side by side. A regression is "this span changed, here is the diff."

A trace UI that an engineer can navigate in 30 seconds is the difference between debugging and guessing.

Evaluator authoring

If the platform expects engineers to write evaluators, the authoring path has to look like writing a function:

  • Local file, version-controlled. Evaluator code lives in the application repo, not in a vendor UI. The version is the git SHA.
  • Calibration data alongside. A small labelled dataset lives next to the evaluator file. Running the evaluator against the calibration set is a single command.
  • Pinned model and prompt. When the evaluator uses an LLM judge, the model and prompt are explicitly pinned in code. Drift is visible as a code diff.
  • Test harness. A unit-test style harness runs the evaluator against the calibration set and reports agreement with labels. CI runs the harness.
  • Composable. Evaluators can be combined into scorecards without leaving the codebase.

If the only way to author an evaluator is a vendor UI with a YAML editor, the team will stop authoring evaluators.

CI integration

The PR view is where evaluation becomes part of how code is shipped:

  • Scorecard delta in the PR. The CI run produces a scorecard, the platform comments the delta against the base branch (per dimension, per dataset), and the diff is visible in the PR view.
  • Blocking on regression. A regression beyond a configured budget blocks merge. The block names the dimension, the dataset, and the rubric version.
  • Replay link. Each scored sample in the scorecard has a link back to a replay view (the same trace replay the developer uses locally).
  • Manual override is signed. A developer who overrides a block has to leave an explicit reason, captured for audit.

CI integration that lives in a Slack channel is informational. CI integration that lives in the PR view is operational.

Documentation and examples

The smallest test: a junior engineer can instrument a new agent, author a new evaluator, and add a CI gate, from the docs alone, in one sitting. If they cannot, no other feature matters.

Example

A team is evaluating two platforms for a new agent.

Platform A:

  • SDK requires importing four modules and wrapping every LLM call in a context manager.
  • Framework integration is documented for two popular libraries; the team's internal framework requires a custom integration the docs gesture at.
  • Local dev requires hitting a hosted endpoint; no local collector.
  • Trace UI is search-heavy and shows abstract "runs" rather than the team's agent structure.
  • Evaluator authoring is done in the vendor UI with a YAML editor.
  • CI integration emits a JSON file to a build log.

Platform B:

  • SDK has one decorator that auto-instruments the framework, and an OTLP fallback for everything else.
  • Framework integrations include the popular libraries and a worked example for custom orchestration.
  • Local dev runs against a local OTLP collector; traces appear immediately in a local UI.
  • Trace UI shows the agent's decision tree directly; trace search supports attribute filters.
  • Evaluators live in the application repo as small Python files with a calibration dataset alongside.
  • CI integration comments a scorecard delta in the PR view; blocked merges name the dimension and rubric version.

Both platforms claim "agent observability with LLM evaluators." Six months in, Platform B is used by every engineer and the scorecard is part of code review. Platform A is used by the eval team and engineers route around it.

The data model differences may or may not be similar. The adoption difference dominated.

Comparison

Where each category is stronger

Framework-bundled observability is stronger when

  • The team uses one framework and the bundled tool's primitives match the team's mental model.
  • Auto-instrumentation is invisible and the team never has to think about it.
  • Local dev is supported by the framework's own development server.

Independent agent observability is stronger when

  • More than one framework is in play, or custom orchestration is significant.
  • Evaluators are first-class code authored by engineers, gated in CI, and surviving framework choice.
  • Local-dev replay against production traces is part of the daily loop.

Who should not use a hosted eval-first platform

Solo developers and small teams whose iteration loop is "edit, run locally, eyeball the output" do not need a hosted eval-first platform. A local trace viewer and a few inline assertions cover the case. The hosted platform's developer experience becomes valuable once there is more than one engineer, more than one agent, and a deploy cadence fast enough that a CI gate pays off.

Limitations

  • Ergonomics moves fast and roadmaps lie. A platform whose SDK is awkward today may be exemplary in six months. Evaluations should weight current state and recent release cadence, not promises.
  • Auto-instrumentation has a maintenance cost. Framework integrations break when the framework releases. A platform that promises N integrations is signing up to maintain N integrations. Quality of the few that matter beats breadth.
  • Trace UIs reward what is easy to render. Token usage and latency are easy. Agent decision trees with nested tool calls are not. A platform with a beautiful token chart and an opaque agent view fails the daily workflow.
  • CI integration is a security surface. Comments in PRs from a third-party platform are a supply-chain consideration. A well-engineered comment bot is also a webhook with write access.
  • Evaluator authoring in code has a learning curve. Engineers used to dashboard authoring will resist code-first evaluators until the test harness shows the payoff.

Evidence and sources

FAQ

What is the single most important developer-experience criterion? Local-dev replay against production traces. It changes debugging from guessing to inspection, and it is the feature most platforms skip without announcing it.

Should evaluators live in the application repo or in the platform? In the application repo. Version control, code review, CI integration, and refactor tooling all work as expected. Vendor UIs for evaluator authoring become a separate codebase nobody owns.

Is auto-instrumentation always good? It is good for the common case and bad when it gets in the way of custom abstractions. The best SDKs offer auto-instrumentation by default and explicit wrapping for anything custom.

How important is the trace UI versus the CLI? Both. The UI matters for debugging and review; the CLI matters for reproducibility and scripting. A platform with only one of these forces the team to build the other.

What is the right way to surface a scorecard regression to engineers? In the PR view, with the dimension, the dataset, the rubric version, and a replay link. Anywhere else is informational and gets ignored.

Can a great developer experience compensate for a weak data model? For a while. Engineers will tolerate a lot if the daily loop is smooth. The data-model debt shows up later, usually when the team tries to add audit lineage or framework portability and discovers the platform cannot back the promises the demo made.

Related reading