TL;DR: RAG (retrieval augmented generation) fetches the most relevant pieces of your data at question time and hands them to a language model along with the question. The model answers from those sources instead of from memory alone - which means current information, private data, and citations, without retraining anything.
How it works
A RAG system has two phases. At indexing time, your documents are split into chunks, each chunk is converted into an embedding (a list of numbers capturing its meaning), and the embeddings are stored in a vector database. At query time, the user's question is embedded the same way, the database returns the chunks whose embeddings sit closest to the question's, and those chunks are pasted into the model's prompt as context. The model then writes its answer grounded in what it just read.
The quality levers live in the details: how documents are chunked, which embedding model is used, how many chunks are retrieved, and whether a reranker re-sorts the candidates before the model sees them. A RAG pipeline that answers badly is almost always retrieving badly - the model can only be as good as the context it is given.
RAG exists because of two hard limits: models know nothing after their training cutoff, and context windows cannot hold your entire knowledge base. Retrieval turns an open-book exam into a short one - find the right page first, then read it.
Where it sits in the AI stack
RAG is the bridge between your data layer and the model. It consumes embeddings, lives alongside the vector database, and feeds the model's context window:
Key tools and implementations
-
pgvector
Vector search inside Postgres - the boring-but-proven default when your data already lives there.
-
LlamaIndex
A framework built around ingestion, indexing, and retrieval pipelines for RAG applications.
-
LangChain
General LLM orchestration with a large ecosystem of retrievers, loaders, and RAG chains.
-
Managed retrieval APIs
File-search tools from model providers that bundle chunking, embedding, and retrieval behind one API.
Related entries
- Vector database A database that stores embeddings and finds nearest neighbors fast, powering semantic search and RAG.
- Chunking Splitting documents into smaller pieces so each one can be embedded and retrieved on its own.
- Reranking A second scoring pass that reorders retrieved candidates so the most relevant results come first.
- Embedding models A model that converts text into a vector of numbers so similar meanings land near each other.
- Context window The maximum amount of text, measured in tokens, that a model can consider in a single request.