TL;DR: Reranking is a second, more careful scoring pass over search results. Fast retrieval pulls in, say, 50 plausible chunks; a reranker then reads each one together with the query and reorders them so the genuinely relevant ones land on top. It is often the cheapest single upgrade to a retrieval pipeline's quality.
How it works
First-pass retrieval is built for speed. Semantic search embeds the query and the documents separately, so all the document math is done ahead of time and a query only needs one nearest-neighbor lookup. The price of that speed is precision: two texts can sit close in embedding space while differing on exactly the detail the question hinges on.
A reranker spends more compute to recover that precision. The standard design is a cross-encoder: instead of comparing two precomputed vectors, it feeds the query and a candidate passage through the model together, letting every word of one attend to every word of the other, and outputs a single relevance score. Score each of the top candidates this way, sort by the new scores, and keep the best handful. Because the reranker must run once per query-candidate pair, it is far too slow to scan a whole corpus - which is precisely why it runs second, on a shortlist the fast pass already produced.
The pattern is retrieve-wide, rerank-narrow. Ask the index for 50 or 100 candidates - casting a wide net so the right answer is probably somewhere in the pile - then let the reranker find it and promote it. In a RAG pipeline this matters doubly: context windows are finite and models weigh early context heavily, so whether the correct passage arrives first or fifteenth visibly changes the answer. Rerankers also give hybrid search a clean final stage, scoring the merged keyword and vector candidates on one consistent scale.
The trade-off is latency and cost: one extra model call per query, tens to hundreds of milliseconds. For most knowledge-base and support workloads that is a bargain; for autocomplete-style search it may not be. Measure on your own queries - reranking is also one of the easiest components to A/B test, since it bolts onto an existing pipeline without touching the index.
Where it sits in the AI stack
Reranking is the last step of retrieval, sitting between the candidate list and the model's context window:
Nothing upstream changes when you add one, which is why it is usually the first optimization teams try after basic retrieval works.
Key tools and implementations
-
Cohere Rerank
A hosted reranking API: send a query plus candidates, get back relevance-sorted results.
-
Sentence-transformers cross-encoders
Open-source cross-encoder models you can run locally for self-hosted reranking.
-
BGE rerankers
A widely used family of open-weight rerankers with strong multilingual coverage.
-
LLM-as-reranker
Prompting a general language model to score or order candidates - flexible, but slower and pricier.
Related entries
- Semantic search Search that ranks results by meaning, using embeddings, so matches do not require the exact words.
- Hybrid search Running keyword and vector search together and merging results, so exact terms and meaning both count.
- RAG (retrieval augmented generation) Fetching relevant documents at query time so a language model can answer from your data instead of memory alone.
- Embedding A list of numbers capturing a piece of content's meaning, so similar things sit close together in vector space.