TL;DR: A trace is the complete record of one request through an AI system, broken into spans - one per step, with its inputs, outputs, timing, and token counts. When a user reports "the bot gave a weird answer," the trace is how you replay exactly what happened instead of guessing.
How it works
Tracing borrows its vocabulary from distributed systems. A span is one timed operation - an LLM call, a retrieval query, a tool invocation - recording what went in, what came out, how long it took, and what it cost. A trace is the tree of spans for a single request, held together by a shared ID. For LLM work each span carries extra fields: the exact rendered prompt, the model and parameters, token counts, and the raw completion.
Take one question to a RAG app. The trace shows a parent span for the request, a child for embedding the query, a child for the vector search with the ten chunks it returned, and a child for the model call with the full assembled prompt. When the answer is wrong, the trace localizes the failure in seconds: retrieval fetched the wrong chunks, or retrieval was fine and the model ignored them. Those are different bugs with different fixes, and without the trace they look identical. The span tree also shows where the time went - the user experienced one slow answer, but the trace shows whether retrieval or generation owns the latency.
Traces matter even more for agentic workflows, where a single user request can fan out into dozens of model calls and tool invocations chosen at runtime. There is no fixed code path to read - the trace is the only complete account of what the agent actually decided to do, which loop it got stuck in, and where the tokens went.
In practice you instrument once with an SDK or proxy and every request is captured automatically; OpenTelemetry's generative-AI conventions give the span fields standard names so tools can interoperate. Two habits keep tracing useful: scrub or restrict sensitive prompt content, since traces store exactly what users typed, and sample high-volume routes so storage does not balloon. Individual traces answer "what happened here"; aggregated across traffic, they become the raw material of LLM observability. They also make excellent eval fodder - a trace that ended badly is a ready-made test case, carrying the exact inputs needed to reproduce the failure.
Where it sits in the AI stack
Tracing is the capture layer between the running application and everything that analyzes it:
Key tools and implementations
-
OpenTelemetry
The vendor-neutral tracing standard, with semantic conventions for generative-AI spans.
-
Langfuse
Open-source trace capture and a UI built for inspecting prompts, completions, and costs.
-
LangSmith
Deep tracing for LangChain and LangGraph apps, with traces feeding datasets and evals.
-
W&B Weave
Call-level tracing from Weights and Biases - decorate a function and every invocation is recorded.
Related entries
- LLM observability Collecting traces, metrics, and quality signals from a live AI system so problems surface before users complain.
- Token cost tracking Counting tokens per request and pricing them per model so AI spend is a metric, not a surprise bill.
- Agentic workflow A multi-step process that chains, branches, or loops model calls so an AI system can complete tasks too big for one prompt.
- Tool use (function calling) Letting a language model request actions by emitting structured calls that your code executes and whose results feed back in.