Observability for LLM Systems
When a user reports a bad answer next week, you need to reconstruct what happened. What to log, and the one field everyone omits.
On this page
A user says the assistant gave them wrong information yesterday. You have their question. You cannot reproduce the problem.
Without logs, that investigation is over before it starts. LLM systems are nondeterministic, depend on retrieved content that changes, and run against models that change under stable version names. Reproduction from the input alone is not possible.
What to log
Per request:
The complete prompt as sent. Not the template — the assembled text including system prompt, history, and retrieved content.
The raw response, before any parsing or post-processing.
Model identifier and version, plus the parameters used — temperature, max tokens.
Token counts, input and output separately. This is your cost data.
Latency, split into time-to-first-token and total where you stream.
Retries and errors, including what failed and what the retry produced.
A correlation ID linking related calls in a multi-step flow.
The field everyone omits
Retrieved context. For RAG systems, log which chunks were retrieved, their scores, and their source identifiers.
This is the single most valuable field and the most commonly missing one, because most RAG failures are retrieval failures. The diagnostic question is nearly always “was the correct passage retrieved?” — and you cannot answer it after the fact without this log.
Logging the question and the answer but not the retrieved passages means every RAG investigation stalls at the first step.
Tracing multi-step flows
A single user request may produce several model calls: query rewriting, retrieval, reranking, generation. Or a dozen agent loop iterations.
Log these as a trace — one parent span per user request, child spans per model call and tool call, with timing and tokens attached to each.
Two things this gives you that flat logs do not: which stage consumed the latency, and which stage introduced the error. For agents it is essential, since a failure at step nine is incomprehensible without steps one through eight.
Sensitive data
Full prompt logging means your logs contain whatever users sent. That is frequently a larger data exposure than the API call itself, because logs persist longer and more people can read them.
Practical reconciliation: redact identifiable fields before writing rather than after, bound retention to your actual policy rather than the default, restrict log access as you would database access, and sample full prompts at a low rate while keeping metadata for everything — plus all failures, since those are what you investigate.
What to alert on
Error rate, by type. A rise in 429s means capacity; a rise in parse failures means the model changed or your prompt drifted.
Latency percentiles, not averages. The tail is what users notice.
Cost per hour. The classic unmonitored failure is a loop or a runaway prompt discovered on the monthly bill. Hourly spend with a threshold catches it in an hour.
Refusal and fallback rate. A rise here often means a model update changed behaviour — and it is otherwise invisible.
Validation failure rate. Leading indicator of a prompt or model problem.
Retrieval quality proxies. Rising rates of low-scoring retrievals or empty result sets, which predict bad answers before users report them.
Feedback as data
A thumbs-down control is the cheapest source of real eval cases available.
Capture the rating with the correlation ID so you can pull the full trace. A negative rating plus its trace — prompt, retrieved context, response — is exactly a test case, and it costs nothing to collect.
This closes the loop: production failures become the eval set that prevents them recurring.
What to remember
- Nondeterminism, changing retrieval, and silent model updates make reproduction from input alone impossible.
- Log the assembled prompt, raw response, model version, token counts, latency, retries, and a correlation ID.
- Log retrieved context — the most valuable and most commonly missing field, since most RAG failures are retrieval failures.
- Trace multi-step flows as parent and child spans; agents are undiagnosable otherwise.
- Logs are a data exposure: redact before writing, bound retention, restrict access, sample — but keep all failures.
- Alert on error rate, latency percentiles, hourly cost, refusal rate, and retrieval quality.