TL;DR: An inference provider runs AI models on its own hardware and sells access through an API, usually billed per token. You send a request over HTTPS, the provider's GPUs do the work, and tokens come back - no servers to buy, no models to deploy. The trade is convenience and scale for per-call cost and less control over where your data goes.
How it works
Running a large language model well is an infrastructure problem: you need GPUs with enough memory to hold the weights, a serving engine that keeps those GPUs busy, and capacity planning for spiky traffic. Inference providers absorb all of that. Your application makes an API call - a prompt in, a completion out - and everything between the request and the response is the provider's problem. A chatbot that serves ten users and one that serves ten million can hit the same endpoint.
Providers come in three broad shapes. Model-maker APIs are run by the labs themselves - Anthropic, OpenAI, and Google each serve their own models, and their frontier models are available nowhere else. Open-weight hosts such as Together AI, Fireworks, and Groq serve open-weight models like Llama and Qwen, competing on speed and price for the same model. Cloud platforms - AWS Bedrock, Azure, Google Cloud - resell many models inside an existing cloud account, or rent you raw GPUs to run your own serving stack on.
Choosing between them comes down to a handful of levers: which models are offered, latency (both time to first token and tokens per second), price per million tokens, rate limits, data-handling and retention policies, and reliability. Techniques like batching and prompt caching are how providers hit their price and speed numbers, and most expose them to you as discounts.
The alternative is running models yourself - on rented GPUs with a serving engine, or fully on your own machines as a local LLM. Self-hosting wins when privacy rules forbid sending data out, when token volume is high enough that rented hardware beats per-token pricing, or when you need a fine-tuned model no provider hosts. For most teams, though, a provider is the right default: the serving problem is genuinely hard, and the people selling it as a service are very good at it.
Where it sits in the AI stack
The provider is the layer between your application and the hardware. Everything below the API line is theirs:
Key tools and implementations
-
Model-maker APIs
Anthropic, OpenAI, and Google serve their own models first-party - the only way to run their frontier models.
-
Open-weight hosts
Together AI, Fireworks, and Groq serve open-weight models, competing on latency and price per token.
-
Cloud platforms
AWS Bedrock, Azure, and Google Cloud resell many models inside your existing cloud account and billing.
-
API routers
Gateways like OpenRouter put one API in front of many providers, making it easy to switch or fail over.
Related entries
- Choosing a model A framework for weighing capability, latency, cost, privacy, and openness when picking a model for a task.
- Local LLM Running a language model entirely on hardware you control instead of calling a hosted API.
- Batching Grouping multiple requests into one GPU pass so serving hardware stays fully utilized.
- Inference Running a trained model to produce output - the phase where every user request is actually served.