TL;DR: Chunking splits documents into smaller pieces before they are embedded and indexed. It exists because you retrieve chunks, not documents: a 200-page manual is useless as a single search hit, but the half-page section that answers the question is exactly right. Chunk size and boundaries quietly decide how good retrieval can ever be.
How it works
During ingestion, each document is cut into pieces - typically a few hundred tokens each - and every piece is embedded and stored individually. When a query arrives, retrieval returns the closest chunks, and only those chunks enter the model's context window. The document as a whole never travels; the chunk is the unit of everything downstream.
That makes chunking a balancing act. Chunks that are too small lose context: a lone sentence saying "this setting defaults to off" embeds fine but retrieves uselessly, because nothing says which setting or which product. Chunks that are too large blur together: an embedding of three unrelated sections points at the average of their meanings and matches none of them well. Most teams start somewhere between 200 and 800 tokens with a modest overlap between neighboring chunks, so a sentence falling on a boundary appears whole in at least one of them.
Strategy matters as much as size. The naive approach cuts every N characters regardless of meaning. Recursive splitting improves on it by preferring natural boundaries - paragraphs, then sentences - and only cutting mid-sentence as a last resort. Structure-aware splitting uses the document's own shape, keeping a Markdown section or an HTML table together as one chunk. Semantic chunking goes further, using embeddings themselves to find where the topic shifts. A useful refinement is attaching context to each chunk - the document title, the section heading, sometimes a generated one-line summary - so the chunk still makes sense when it is retrieved alone.
When a RAG system answers badly, chunking is one of the first places to look. If the right document was indexed but the wrong text comes back, the boundaries - not the model - are usually to blame.
Where it sits in the AI stack
Chunking happens at indexing time, after documents are parsed and before anything is embedded. Every layer after it inherits its decisions:
Changing a chunking strategy means re-embedding and re-indexing the affected documents, which is why it belongs in a repeatable pipeline rather than a one-off script.
Key tools and implementations
-
LangChain text splitters
Recursive, token-based, and language-aware splitters used across many RAG stacks.
-
LlamaIndex node parsers
Chunkers that keep document structure and metadata attached to every node they produce.
-
Unstructured
Parses PDFs, slides, and HTML into clean elements that make sensible chunk boundaries possible.
-
Docling
An open-source document converter that preserves layout, tables, and reading order for chunkers.
Related entries
- RAG (retrieval augmented generation) Fetching relevant documents at query time so a language model can answer from your data instead of memory alone.
- Embedding models A model that converts text into a vector of numbers so similar meanings land near each other.
- Data pipelines for AI The ingestion, cleaning, chunking, and embedding steps that turn raw data into a searchable index.
- Context window The maximum amount of text, measured in tokens, that a model can consider in a single request.