TL;DR: LLM providers bill per token, so cost is a per-request property you can measure, attribute, and optimize - not a monthly surprise. Token cost tracking records input and output counts on every call, prices them per model, and rolls them up by feature, user, and version so someone notices the day spend doubles.
How it works
Every model API response reports usage: how many tokens came in and how many went out. Multiply each by the model's per-token rate - output tokens typically cost several times more than input - and you have an exact price for that one call. Capture it on every request, tagged with the feature, user, model, and prompt version that produced it, and spend becomes a queryable metric like latency.
The classic gotcha is conversation growth. Most chat apps resend the entire history with every turn, so input tokens grow with each message: turn one costs 500, turn thirty costs 15,000 - for the same size answer. Whether a long session even fits is a context window question, but the bill hurts long before the limit does. This is why per-request tracking beats a monthly invoice: the invoice says spend tripled, the per-request data says power users' sessions now average 60 turns, which is a fact you can act on.
Attribution is where tracking earns its keep. Rolled up by dimension, the data answers the questions that actually change behavior: which feature burns the budget, whether cost per conversation is trending up, what the expensive one percent of requests have in common. Teams then set budgets and alerts on those rollups - a monthly cap per feature, a page when cost per request jumps after a deploy - the same way they already alert on error rate as part of observability.
The data also drives the fixes. Trim a bloated system prompt that is resent on every call and the savings multiply across all traffic. Prompt caching makes repeated context dramatically cheaper - cached input tokens are billed at a fraction of the normal rate, and usage reports show cached counts separately so you can verify the discount is real. Routing simple requests to a smaller model, capping output length, and summarizing old conversation turns are the other standard levers - each visible in the numbers the day it ships.
Where it sits in the AI stack
Cost tracking taps the usage data on every model response and turns it into rollups, budgets, and alerts:
Key tools and implementations
-
Helicone
Proxy-based logging that adds per-request cost, caching stats, and user-level breakdowns.
-
LiteLLM
Open-source gateway across providers with spend tracking, budgets, and per-key rate limits.
-
Langfuse
Cost rolled up from traces, so spend sits beside quality and latency for every request.
-
Provider usage dashboards
The billing consoles from model vendors - coarse but authoritative, the baseline to reconcile against.
Related entries
- Token The chunk of text - roughly three-quarters of a word - that a language model reads and writes one at a time.
- Prompt caching Reusing the computed state of a repeated prompt prefix so later requests skip that work.
- LLM observability Collecting traces, metrics, and quality signals from a live AI system so problems surface before users complain.
- Tracing Recording every step of an AI request as a tree of spans so you can see exactly what the system did.