TL;DR: LLM observability is watching an AI system in production: logging every request, tracing what happened inside it, and scoring a sample of live outputs. Offline evals tell you a change looked good before it shipped; observability tells you whether it is still good now, with real users typing things no test set predicted.
How it works
The foundation is capture. Every production request is logged with its inputs, outputs, model and prompt versions, token counts, latency, and any errors - usually structured as a trace so multi-step requests stay legible. On top of that raw stream sit aggregate metrics: p95 latency, error rate, tokens per request, cost per feature, refusal rate. This is classic application monitoring extended with the dimensions unique to LLMs.
What makes it more than logging is quality signal. Language models fail silently - a wrong-but-fluent answer returns HTTP 200, so no traditional alert will ever fire. Observability platforms close that gap by scoring a sample of live traffic with judge models or heuristic checks, and by collecting explicit feedback like thumbs-down clicks, retries, and human-escalation rates. A dashboard that shows only latency and cost is watching the plumbing while ignoring the water.
A concrete example: a provider quietly updates the model behind your chat feature. Latency and error rate do not move, but over two days the thumbs-down rate on billing questions doubles and judge-scored faithfulness slips. With observability in place that is an alert and a diff between two model versions; without it, the first signal is a frustrated customer email weeks later.
Getting value out of it is mostly a tagging discipline. Every logged request should carry the prompt version, model version, and feature that produced it, because "quality dropped" only becomes actionable when you can slice by what changed. Sample judge scoring on a percentage of traffic to keep the scoring bill sane, and redact personal data before it lands in the trace store - production logs of user conversations are sensitive by default.
Online monitoring and offline evals form one loop. Production surfaces the failures your test set never imagined; the best of those get folded into the golden dataset so the next regression run catches them before shipping. Observability data also feeds cost tracking - the same token counts that explain quality explain the bill.
Where it sits in the AI stack
Observability wraps the production application. Instrumentation in the app emits traces and metrics; the platform aggregates, scores, and alerts:
Key tools and implementations
-
Langfuse
Open-source LLM observability - tracing, cost breakdowns, and online scoring, self-hostable.
-
LangSmith
Tracing and monitoring paired with datasets and evals, from the LangChain ecosystem.
-
Arize Phoenix
Open-source tracing and evaluation built on OpenTelemetry conventions for LLM apps.
-
APM suites with LLM views
General observability platforms - Datadog and peers - that add LLM traces beside existing infra dashboards.
Related entries
- Tracing Recording every step of an AI request as a tree of spans so you can see exactly what the system did.
- Token cost tracking Counting tokens per request and pricing them per model so AI spend is a metric, not a surprise bill.
- Evals (AI evaluations) Structured tests that score an AI system's outputs so teams can measure quality and catch regressions.
- Guardrails Programmatic checks on an AI system's inputs, outputs, and actions that block or correct behavior outside defined limits.