Updated: 2026-03-09 By: Ari Heljakka
Short answer
The LLM observability landscape is not a flat list of competitors; it is three structurally different categories with different data models, lifecycles, and operational roles. Evaluation-first platforms version objectives and gate changes. Framework-coupled platforms attach deeply to a specific LLM framework and trade portability for ease of integration. Open-source tracing tools store raw traces and stay deliberately neutral on what to do with them. Most teams need at least two of the three for different jobs. A category-first map prevents both the "pick the most-funded option" mistake and the "pick the most-OSS option" mistake.
Key facts
- Definition: A category is defined by the platform's primitive, its target lifecycle, and the role it plays in the engineering workflow, not by its feature list or branding.
- When to use: Before reading any vendor comparison; before any platform selection; during quarterly stack reviews.
- Limitations: Several products straddle categories; the boundary is set by the data model, not the marketing. Some hybrid options now span two categories in earnest, not just by feature flag.
- Example: A team uses an open-source tracing layer for raw capture, an evaluation-first platform for gating, and a framework-coupled tool only inside the niche where its framework dependency is acceptable.
Key takeaways
- Three categories cover most of the field: evaluation-first, framework-coupled, and open-source tracing.
- Each category is shaped for a different job. Stretching one outside its job works for a while and stops scaling.
- Framework-coupled platforms trade portability for integration depth; the trade is fine if your framework choice is stable for years.
- Open-source tracing wins on raw capture and self-hosting; it loses on scored signals and gating unless paired with an evaluation layer.
- Evaluation-first platforms produce the gate that the other two categories do not natively produce.
Definition
LLM observability is the set of capabilities that lets a team see, score, and act on the behavior of LLM-powered systems in production: trace capture, scoring, alerting, gating, root-cause clustering. The category divisions below are about which of those capabilities each platform optimizes for.
- Evaluation-first platforms version success criteria as first-class artifacts (objectives, calibration datasets, judges, rubrics) and use them to gate changes pre-deploy and to monitor scores post-deploy.
- Framework-coupled platforms attach to a specific LLM framework (chains, agent graphs, prompt graphs) and produce observability that is tightly synchronized with the framework's runtime.
- Open-source tracing tools capture and store traces in a vendor-neutral format, typically built on (or compatible with) OpenTelemetry, with optional add-ons.
The categories share vocabulary; their data models and lifecycles are different.
When this matters
The category-first view matters whenever a single platform is being asked to do all three jobs. Most teams discover the seams when:
- A framework-coupled platform makes a framework switch hard.
- An open-source tracing tool does not gate a PR, so regressions ship.
- An evaluation-first platform's tracing layer is shallower than the production triage workflow needs.
In each case, the failure mode is asking the wrong category to do a job it was not shaped for. A category map prevents the category error before the platform selection.
How it works: the three categories
Category 1, evaluation-first platforms
The core abstraction is the evaluator, which carries a versioned objective, a calibration dataset, a pinned judge model, and a rubric. These platforms run continuously, scoring every PR, every merge, every canary, and a sample of production traffic against the same evaluators.
What they are good at is versioned objectives that compose into per-dimension scorecards, calibration against ground-truth datasets with judge agreement tracked as its own first-class metric, per-dimension gating in CI/CD (a prompt change cannot ship if faithfulness drops by more than N), and 0 to 1 normalized scoring that lets dimensions combine without double-counting.
They stop being the right fit when the workload is pure trace capture at very high volume (the evaluation layer is unused and the tracing layer is shallower than a dedicated tracing tool) or when there is no quality bar to enforce in the first place, as in very early prototypes and throwaway scripts.
Category 2, framework-coupled platforms
The core abstraction is the framework runtime object (chain, graph, node, edge), and the platform lifecycle is synchronized with the framework's own execution so that traces and spans map one-to-one to the framework's primitives.
What they are good at is minimal integration cost when the team is already on the framework, out-of-the-box visualization of framework-native objects (chains, agent graphs), and serving as the primary debugging surface for framework-specific features such as node-level retries.
They stop being the right fit when the team's framework choice is unstable or expected to change, when multiple frameworks coexist in production and the platform only deeply supports one, or when the workload includes non-framework systems (custom orchestration, raw API calls) that the platform does not see.
Category 3, open-source tracing tools
The core abstraction is the span and trace, typically aligned with OpenTelemetry, and the lifecycle is continuous trace capture with reactive search and slicing on top.
What they are good at is vendor neutrality (portable across hosted and self-hosted deployments), a rich ecosystem of exporters, processors, and visualization tools, self-hostability for data-residency and compliance constraints, and zero coupling to any specific LLM framework.
They stop being the right fit when scored signals and gating are required and the team has to add an evaluation layer on top, or when the operational ergonomics of clustering, version diffs, and replay end up depending on add-ons or homegrown tooling rather than coming from the platform itself.
How the categories compose
Most production teams end up with at least two of the three categories, used for different things. A common composite:
- Open-source tracing for raw capture, self-hosted to control cost and data residency.
- Evaluation-first for versioned objectives, calibration, gating, and per-dimension scoring on a sampled subset of the traces.
- Framework-coupled only inside the surface where the framework is the dominant runtime and the integration depth pays off.
The categories compose because their primitives are orthogonal: a trace and a versioned objective can coexist without conflict; a framework's runtime view and a generic trace view are different layers on the same call.
Comparison: the three categories on orthogonal axes
This is not a vendor list; it is a category comparison.
| Axis | Evaluation-first | Framework-coupled | Open-source tracing |
|---|---|---|---|
| Primitive | Versioned objective and judge | Framework runtime object | Span and trace |
| Native lifecycle | Continuous; gates and monitors | Synchronized with framework execution | Continuous capture; reactive search |
| CI/CD role | Hard gate on PR, merge, canary | Indirect; surfaces failures the framework reports | Source of post-hoc triage |
| Portability | High; evaluators are framework-agnostic | Low; coupled to one framework | High; OpenTelemetry-aligned |
| Self-host | Varies | Varies | Native |
| Calibration discipline | First-class | Out of scope | Out of scope |
| On-call workflows | Clustering, diffs, replay against candidates | Framework-native traces, sometimes shallow at edges | Search and slice; clustering via add-ons |
| Cost model | Per-event plus evaluator calls | Per-event, framework-tied | Self-hosted cost on the team |
| Center of mass | Engineer shipping an LLM feature | Engineer on a specific framework | Platform engineer or SRE |
The pattern is consistent: each category is excellent at one shape of problem and awkward at the others.
Where evaluation-first stops being the right answer
For pure trace capture at very high volume, or workloads with no quality bar (early prototypes), the evaluation layer is unused and the tracing layer is shallower than a dedicated tracing tool.
Where framework-coupled stops being the right answer
The moment the framework choice is unstable, multiple frameworks coexist, or non-framework systems join the production surface, the framework-coupled tool stops covering the production reality.
Where open-source tracing stops being the right answer
The moment a hard gate is needed on a PR (block the merge if faithfulness drops), or scored signals are needed on production traffic, the tracing tool's neutrality becomes a gap. The team either builds the evaluation layer or adopts one.
Example: a team's three-category stack
A team running a tool-using agent in production:
- Open-source tracing captures every span at the head, tail-samples failures at full fidelity, and exports to a self-hosted store. Cost is operational time plus storage.
- Evaluation-first platform versions three objectives (faithfulness, tool-call quality, policy adherence), runs them on every PR and on a 10 percent sample of production traces, with per-dimension floors enforced in CI/CD. Calibration is against a labeled dataset of 800 sessions; judge agreement is tracked weekly.
- Framework-coupled platform is used only for a separate workflow agent built on a single framework, where the runtime-native trace is the primary debugging artifact.
The team ends up with three categories doing three different jobs in a single composite stack, each one doing the work it was designed for and none asked to substitute for another.
Limitations
- Categories blur. Several products span two categories; the boundary is set by the data model, not the marketing.
- The map is approximate. A few products cover all three jobs at moderate depth in earnest; they are valid for teams that want one bill and accept the trade-offs.
- Self-hosted is not free. Open-source tracing trades vendor cost for operational cost. Run the math on platform engineering time before assuming it is cheaper.
- Framework coupling can be temporary. Some platforms began framework-coupled and added generic OpenTelemetry support; the category lines move.
- The composite has integration cost. Three categories is also three integrations and three on-call surfaces. The win has to be worth it.
Evidence and sources
- OpenTelemetry GenAI semantic conventions, https://opentelemetry.io/docs/specs/semconv/gen-ai/, for the trace-and-span model that anchors open-source tracing.
- Anthropic, Test and evaluate, https://docs.anthropic.com/en/docs/test-and-evaluate/, for the rubric-and-judge model behind evaluation-first platforms.
- OpenAI, Function calling and tool use, https://platform.openai.com/docs/guides/function-calling, for the framework-runtime patterns that framework-coupled platforms exploit.
FAQ
Can one platform do all three jobs? A few claim to. Read the data model. If the primitive maps to all three (versioned objective, framework runtime object, span and trace) and each is a first-class artifact, the claim holds. Otherwise the platform is one category with adapters.
Is framework coupling always bad? No. If the team's framework choice is stable for years and the platform's integration depth saves real time, the trade is fine. The risk is locking the platform decision to the framework decision.
Should I always use open-source tracing? For raw capture and portability, often yes. For scored signals and gating, you need an evaluation layer on top, either built in-house or adopted alongside.
How do I avoid stacking three platforms? Start with the category that maps to your most expensive failure mode. Add the next category only when the second-most-expensive failure compounds. Do not adopt all three on day one.
Where do guardrails fit? Guardrails are a runtime concern, not a category. Most observability stacks integrate with whatever guardrail layer the team uses; the evaluation-first platform is usually where guardrail effectiveness is measured.