Updated: 2026-02-21 By: Ari Heljakka

Short answer

Logs are the first tool teams reach for when an LLM-powered system misbehaves, and they remain the most efficient way to root-cause specific incidents fast. The discipline is the same as for any distributed system: structured JSON logs with stable field names, correlation IDs linking the user request to every downstream operation (LLM call, tool invocation, retrieval, post-processing), PII redaction at the source, and a per-call payload that captures enough context to reconstruct what happened. The LLM-specific additions are full prompt and completion content (subject to redaction), token usage and model version on every call, and an explicit handoff to the evaluator layer for the failure classes logs cannot detect on their own (semantic drift, goal-level failures, judgement quality). Logs answer "what happened in this session"; evaluators answer "is this kind of session getting worse over time." Both are required; neither substitutes for the other.

Key facts

Definition: Log-based LLM debugging captures structured per-call records (input, output, model, tokens, latency, error, correlation ID) and uses them to reconstruct sessions, trace failures to their first wrong step, and feed candidate examples into the evaluator and annotation pipeline.
When to use: From day one. Even pre-launch, structured logs make integration tests debuggable; post-launch, they are the primary forensic surface for incidents.
Limitations: Logs do not detect semantic failures by themselves; a 200 OK with a hallucinated answer is indistinguishable from a 200 OK with a correct one until an evaluator scores the content. PII redaction at the source is mandatory.
Example: A team uses correlation IDs to link a user complaint to the originating prompt, the retrieval that returned bad context, and the LLM call that produced the wrong answer; root cause is found in 10 minutes instead of a multi-day forensic exercise.

Key takeaways

Structured JSON logging with stable field names is the floor. Plain-text logs do not survive contact with multi-step LLM systems.
Correlation IDs (and session IDs for multi-turn agents) are the single highest-leverage primitive. They turn forensic exercises into queries.
The three log classes worth capturing: request/response, error/exception, and performance metrics. Each answers different questions.
Logs answer "what happened in this session." Evaluators answer "is this kind of failure increasing." Use logs for forensics; use evaluators for monitoring and gates.
PII redaction belongs at the source, not downstream. Trying to redact later is a security debt that compounds.
Sample, do not capture everything. Full-fidelity logging on every call at scale is expensive and rarely necessary; sample by surface, by anomaly, and by signal value.

Definition

A production-grade LLM logging discipline has four properties that have to coexist:

A structured format, meaning JSON logs with stable field names, versioned field schemas, and documented field semantics.
Correlation across the stack, where every log line carries a correlation ID linking it to the originating user request, plus a session ID linking multi-turn interactions.
Comprehensive coverage per failure class, with three log classes (request and response logs, error and exception logs, performance metrics) ingested in parallel rather than treated as separate systems.
A bridge into evaluation, where anomalous or user-flagged log records are routed into the annotation queue and from there feed the calibration set for evaluators. The logging surface is the upstream of the reliability loop, not its endpoint.

The objective (root-cause incidents fast, surface patterns for evaluators to monitor) is independent of the implementation (which logging stack, which storage backend, which forwarder). Standardising on the OpenTelemetry GenAI schema is the cheapest way to keep that independence.

When this matters

The case for treating LLM logging as first-class infrastructure sharpens under four conditions.

First production incident. A user reports "the answer was wrong yesterday around 3pm." Without correlation IDs and full content, the investigation is a forensic exercise; with them, it is a query.
Multi-turn agents in production. Without session IDs, multi-step failures look like unrelated single-call anomalies; the actual failure pattern is invisible.
Regulated surfaces. Audit replay requires reproducible session reconstruction; structured logs with full content (subject to redaction) are what the replay runs against.
Cost or latency anomalies. Performance log analysis catches infinite reasoning loops, token-cost spikes, and tool-call timeouts that user-facing monitoring misses until they propagate.

How it works

Why logs matter for LLM debugging

LLMs amplify the standard distributed-system debugging problem. A single user-facing request often expands into:

1 or more LLM calls (often with different prompts and models)
0 to several retrieval calls
0 to several tool invocations
0 to several post-processing or formatting steps

Without structured logs and correlation, reconstructing what happened in a specific failing session is materially harder than for a generic microservice request. With them, the reconstruction is a query.

The three log types worth capturing

Each answers a different class of question; all three are necessary.

Request and response logs. Per-LLM-call records with: input prompt (subject to redaction), output completion, model identifier, model version, request timestamp, latency, token usage, correlation ID, session ID, and tool-call metadata if applicable. These records are what forensic reconstruction works from.

Error and exception logs. API failures (rate limits, timeouts, provider errors), tool-call failures (invalid arguments, downstream errors), parse failures (malformed JSON output), and validation failures (schema violations). LLM systems have a larger surface area of probabilistic failures than typical services; capturing every exception with full context is non-negotiable.

Performance metrics. Time-to-first-token (TTFT), per-percentile latency, token usage distributions, throughput by model and endpoint, cache hit rates. Used both for cost and latency monitoring and as anomaly signals into the evaluator pipeline.

Common failure patterns visible in logs

A trained reader can spot several recurring patterns directly in log analysis.

Hallucinations and incorrect outputs. Visible only by reviewing response content (often surfaced first by user complaints). Logs make the review possible; evaluators make detection systematic.
Latency spikes and timeout errors. Performance log analysis surfaces them quickly; the harder question is which downstream change caused the spike. Correlation IDs and per-call model versions answer that.
Tool call failures and post-processing errors. Tool invocation logs and parse error logs together show the upstream cause; without correlation, the user-facing error and the underlying tool failure look like unrelated events.
Infinite reasoning loops. Step-count spikes in agent traces (10x normal step count for a session) signal a loop. Logging the step counter on every agent iteration is the cheapest control.
Cost spikes. Per-call token usage logs aggregated by model and surface reveal unexpected token growth before the monthly bill does.

Step-by-step workflow for log-based debugging

The pattern most teams converge on for incident investigation.

Start from the user-reported correlation ID (or session ID, or timestamp). The user complaint comes in; the support log entry is linked to a correlation ID; the correlation ID is the entry point.
Reconstruct the session. Query all logs with the correlation ID or session ID; order by timestamp; reconstruct the full step sequence.
Identify the first wrong step. Walk the session forward; the failure mode is "the first step whose output was not what it should have been." Multi-step failures usually have a clear first cause once reconstructed.
Check for environment context. Was there a recent release? A model version change? A retrieval index update? A tool API change? Correlate the first wrong step's timestamp with deploy history.
Reproduce. Where possible, replay the exact input against the current system to confirm the failure is still live, or against a pinned version to confirm a regression.
Open an issue. The failure example feeds the annotation queue. If a calibrated evaluator already exists for this failure mode, score it; if not, this is a candidate for a new evaluator.
Ship the fix. The fix is gated by the evaluator (if one exists) or by manual review (if not). The next regression is caught at CI.

The pattern is forensic for the specific incident and feeds the reliability loop for the failure class.

Correlation IDs as the highest-leverage primitive

Three implementation requirements that pay back fast.

Generated at the edge. The correlation ID is created at the user-facing API boundary, not deeper in the stack, and every downstream call carries it.
Propagated across all boundaries. Async tool calls, parallel branches, downstream services, and evaluator runs all need to forward the ID. Most propagation bugs are async-boundary bugs in product code, and lint rules and helper libraries are the standard mitigations.
Indexed in storage. Querying by correlation ID is an O(1) operation, and without an index the log records are not queryable on the time scale a forensic investigation requires.

A real-world case: a fintech team observed null-value hallucinations after a schema change to a downstream API. The user complaint included a correlation ID; the engineer queried by correlation ID; the LLM call's input contained nulls the schema had not previously emitted; the prompt did not handle nulls; total investigation time, on the order of 10 minutes. The same investigation without correlation IDs would have started with "what changed yesterday" and taken days.

Where logs hand off to evaluators

Structured logs answer "what happened in this session" with high fidelity. They do not by themselves answer "is the rate of this kind of failure changing over time." That is the evaluator layer's job.

The handoff:

Anomaly-flagged log records (high latency, error responses, unusual step counts) route to the annotation queue.
User-flagged sessions route to the annotation queue.
A stratified sample of nominal sessions also routes to the queue, weighted to under-represented failure modes.
Annotation labels feed the calibration set for evaluators.
Evaluators score the sampled stream continuously; per-dimension drift surfaces as monitoring signal.

Both surfaces share the same underlying data (OpenTelemetry GenAI traces, correlation IDs, session reconstruction). Logs are the forensic surface, evaluators are the operational surface, and the same trace records flow into both.

Best practices

A short, opinionated list that survives most production environments.

JSON-structured logs with stable field names. Standardise on a schema (OpenTelemetry GenAI is the cheapest standard); document fields; version the schema.
Correlation IDs everywhere, propagated to every async boundary. Provide a helper library; do not let each team reinvent context propagation.
PII redaction at the source. Sensitive fields redacted in the collector before the trace lands in storage. Allowlist or denylist as policy; never trust downstream redaction.
Sample, do not capture everything at full fidelity. 100% of anomaly-flagged sessions, 100% of safety-relevant events, 5 to 30% of nominal sessions, 100% of an adversarial canary slice.
Storage tiering by use case. Hot for the active investigation window, warm for regression backstop, cold for audit replay. Retention policies per surface.
Weekly regression tests using golden datasets. Replay the calibration set against the current system; surface deltas before users do.
Sample 5 to 10% of production traffic for evaluator scoring. Per-surface and per-dimension; tuned for cost and signal value.

Example

A team operates a customer-support agent on a regulated platform. The setup:

Structured JSON logs. OpenTelemetry GenAI schema; correlation ID and session ID propagated through every span; PII redaction in the collector; storage tiered hot/warm/cold.
Three log classes ingested in parallel. Request/response, errors, performance metrics. Per-model and per-surface dashboards as defaults.
Anomaly-flagged routing. Sessions with high latency, parse failures, or user thumbs-down route to the annotation queue automatically.
Calibrated evaluators on the sampled stream. Five judges across faithfulness, on-policy, escalation, format, and tone. Each calibrated to MCC ≥ 0.6.

Then an incident: users report that the support copilot started giving wrong refund-window answers after a schedule change last week.

The first user complaint comes in with a correlation ID. An engineer queries by ID.
Reconstruction: the LLM call's retrieved context contained an outdated refund policy paragraph. The retrieval index was updated last week; one document was missed in the migration.
The fix: re-index the missed document. The investigation time, on the order of 30 minutes including the fix.
The systemic followup: the failure mode
is added to the annotation queue. Domain experts label 80 sampled sessions; a new judge is calibrated against the labels; the judge is promoted to gate-eligible after MCC clears threshold.
The next refund-window incident does not happen, because the gate catches the regression at CI before it ships.

Logs solved the specific incident in minutes. The evaluator pipeline turned the specific incident into a class that cannot silently recur.

Limitations

Logs do not detect semantic failures. A correct-looking, well-formatted, wrong answer leaves no log signature. Evaluators are required for that class.
Full-fidelity logging is expensive. Without sampling and storage tiering, content-heavy traces dominate the storage bill. Cost controls are platform features.
PII redaction has to be tested. False negatives leak PII; false positives destroy traces. Test the redaction pipeline against a labelled adversarial set; treat redaction as a calibrated evaluator with its own quality metric.
Correlation discipline is product-side work. If async context propagation is broken in the product code, the platform cannot reconstruct sessions. Health attestations and helper libraries are the mitigations.
Log analysis does not scale to monitoring. Reading logs is for forensics; for ongoing monitoring, evaluators and dashboards are the right surface.
Replay is only as faithful as the pinned versions. Reproducing a 6-month-old failure requires pinned dataset snapshots, evaluator versions, and judge model versions; without lineage, replay drifts.

Evidence and sources

OpenTelemetry GenAI semantic conventions, https://opentelemetry.io/docs/specs/semconv/gen-ai/, for the trace schema that LLM-aware logging stacks should adopt by default.
OpenTelemetry collector design, https://opentelemetry.io/docs/collector/, for the edge-redaction and routing primitives a multi-backend logging pipeline depends on.
Matthews correlation coefficient, https://en.wikipedia.org/wiki/Phi_coefficient, the alignment metric used to qualify evaluators that the log pipeline feeds.

Evidence cap reached at three links. Additional reading:

FAQ

What is the single most useful logging primitive? The correlation ID, propagated end-to-end and indexed in storage. It turns the forensic problem from "what changed yesterday" into "what happened in this specific session."

Should we log full prompt and completion content? Yes, subject to PII redaction at the collector. Without content, log-based debugging cannot reconstruct semantic failures. Without redaction, you accumulate compliance and liability risk.

How do we keep log storage from exploding? Stratified sampling (not uniform), storage tiering (hot/warm/cold with retention by use case), per-tenant cost dashboards with weekly review.

What is the difference between a log and a trace? A log is a discrete record of an event; a trace is a structured collection of spans across a request lifecycle. In LLM debugging, traces are the right primitive; "logs" colloquially often means "trace records" in this context.

How do we link a user complaint to the failing session? Show the correlation ID in user-facing surfaces where appropriate (support tickets, error pages, footer of debug builds). Make it easy for users to copy. The few seconds of user effort saves hours of engineering investigation.

When do we move from logs to evaluators? For incident-specific root cause, logs are the right tool. For trend detection ("is this class of failure increasing"), evaluators and dashboards are the right tool. Most teams use both, and both run on the same underlying trace records.

How do we handle PII in log records? Edge redaction in the collector; allowlist or denylist as policy; test the redaction pipeline against a labelled adversarial set; treat redaction itself as a calibrated component with its own quality metric. Do not rely on downstream redaction.

How Teams Use Logs to Debug LLM Failures: Structured Logging, Correlation IDs, and the Trace-to-Eval Bridge