Reranking for RAG: Cross-Encoders, ColBERT and Hosted Rerankers (2026)

What you need to know

• Reranking is a two-stage funnel. Retrieve a wide, cheap candidate set with fast retrievers, then re-score only that shortlist with an expensive-but-accurate reranker and keep the top few for the model. You get cross-encoder precision without running it over your whole corpus.
• The durable pattern in 2026 is lexical + dense → RRF → cross-encoder. BM25 catches exact terms, a dense bi-encoder catches meaning, Reciprocal Rank Fusion merges them into one pool of about 100 candidates, and a cross-encoder reranks to the top five to ten that reach the LLM.
• Bi-encoders are fast; cross-encoders are precise. A bi-encoder embeds query and document separately so vectors can be pre-indexed and searched over millions of documents. A cross-encoder reads query and document together for far higher precision, but cannot be pre-indexed, so it only runs over a shortlist.
• Three ways to ship the reranker. A hosted reranker (Cohere Rerank) for lowest friction, a self-hosted cross-encoder (BGE or Jina) for the best accuracy-per-dollar at scale, or ColBERT late interaction for high-precision retrieval over medium corpora where storage is affordable.

Every team that ships retrieval-augmented generation hits the same wall in the same order. You stand up a vector store, embed your documents, run top-k similarity search, and the demo is magic. Then real questions arrive. An Indian fintech building a support assistant asks "which charges apply to a failed UPI mandate?" and the chunk that actually answers it sits at rank eleven, below ten chunks that merely mention charges. A UK legal-search startup asks "does clause 7.2 survive termination?" and the embedding model, which never saw clause 7.2 as a distinct token, returns the wrong clause with total confidence. The right document was in the corpus. It was even in the top fifty. It just was not near the top — and the model only reads the top five.

That gap between "retrieved the right document" and "ranked the right document first" is exactly what reranking closes. As of June 2026, the reranking layer is the most reliable single quality upgrade available to a RAG system, and it is the one most teams running plain vector search have never wired in. This guide is the practical version: the funnel pattern, why bi-encoders and cross-encoders differ at the architecture level, how Reciprocal Rank Fusion merges your lexical and dense lists, working Python for the whole pipeline, and an honest comparison of the three ways to deploy a reranker — hosted, self-hosted and ColBERT — so you can pick the one that fits your corpus, your latency budget and your bill.

The production pattern: retrieve wide, rerank narrow

The architecture that has settled into production — it appears in the Cohere and OpenAI RAG cookbooks and in most serious 2026 retrieval stacks — is a funnel with four stages. Read it once and the rest of this guide is just detail on each stage.

1. Lexical retrieval (BM25). Pull the top ~50 documents by keyword overlap. BM25 is unbeatable at exact terms — product codes, statute numbers, error strings, the literal token "UPI" or "clause 7.2".
2. Dense retrieval (bi-encoder). In parallel, pull the top ~50 by semantic similarity from your vector store. This catches paraphrase and meaning that keywords miss.
3. Fuse with Reciprocal Rank Fusion. Merge the two ranked lists into a single candidate pool of roughly 100 documents using RRF, which combines ranks without needing to reconcile incompatible scores.
4. Rerank with a cross-encoder. Score every (query, document) pair in the pool with a cross-encoder, sort, and keep the top ~5 to 10 for the LLM context.

The economics of the funnel are the whole point. Stages 1 and 2 are cheap and run over your entire index because they are pre-indexable. Stage 4 is expensive per pair but only ever sees about a hundred candidates, so the total cost stays bounded no matter how large your corpus grows. You are buying cross-encoder precision and paying only bi-encoder-scale retrieval cost. Our production hybrid retrieval guide covers the dense-plus-sparse retrieval half in depth; this article is about the rerank that sits on top of it.

Bi-encoder versus cross-encoder: why two models

The funnel uses two fundamentally different kinds of model, and understanding the difference is what makes every later decision obvious. It comes down to when the model gets to see the query and the document together.

A bi-encoder — the model behind your vector store — encodes the query and each document into vectors separately. Because the document side never depends on the query, you can embed every document once, at index time, and store the vectors. At query time you embed only the question and run approximate nearest-neighbour search over millions of pre-computed vectors in milliseconds. That separation is exactly what makes vector search scale. The cost is precision: the model judges relevance from two independently computed vectors and never actually reads the query and the document side by side, so it misses relevance that depends on how the two interact.

A cross-encoder does the opposite. It concatenates the query and one document and feeds them through the transformer together, running full cross-attention between every query token and every document token, and outputs a single relevance score. This is far more precise — it judges relevance the way an attentive human skim-reader would. But the document representation now depends on the query, so nothing can be pre-computed. You must run a forward pass for every (query, document) pair at query time. Over a million-document corpus that is hopeless; over a hundred-candidate shortlist it is fast and entirely affordable. Hence the funnel: retrieve cheaply with the bi-encoder, rerank a shortlist with the cross-encoder.

Property	Bi-encoder (retriever)	Cross-encoder (reranker)
Query and document seen	Separately	Together (full cross-attention)
Pre-indexable?	Yes — embed documents once at index time	No — must run at query time
Scales to	Millions of documents (ANN search)	A shortlist of ~50–100 candidates
Relative precision	Lower — independent scoring	Higher — interaction-aware scoring
Role in the funnel	Stage 2: cast a wide net	Stage 4: re-score the survivors

Reciprocal Rank Fusion: merging lexical and dense

Before the reranker can do its job, you have to hand it a single candidate pool, and you have two lists to merge: the BM25 list and the dense list. Their scores are not comparable — BM25 produces an unbounded term-frequency score, a bi-encoder produces a cosine similarity in a different range — so averaging or thresholding raw scores is fragile. Reciprocal Rank Fusion sidesteps the problem entirely by using only ranks.

For each document d, RRF sums a small contribution from every list it appears in. The contribution from list i is 1 / (k + rank_i(d)), where rank_i(d) is the document's position in that list and k is a constant, conventionally about 60. Formally:

score(d) = Σ  1 / (k + rank_i(d))     # summed over each ranked list i, k ≈ 60

The intuition is clean. A document near the top of either list earns a large contribution; a document buried deep earns almost nothing. A document that ranks decently in both lists earns from both and rises to the top of the fused pool — which is exactly the consensus signal you want. The constant k dampens the influence of the very top ranks so that a single first place does not dominate. RRF needs no score normalisation, no per-corpus tuning and no training; it is robust precisely because it throws away the raw scores and trusts only the orderings. That simplicity is why it is the default fusion step in nearly every hybrid stack.

def reciprocal_rank_fusion(ranked_lists, k=60, top_n=100):
    """Merge several ranked lists (each a list of doc ids, best first) into one pool."""
    fused = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked):        # rank is 0-based here
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    ordered = sorted(fused.items(), key=lambda kv: kv[1], reverse=True)
    return [doc_id for doc_id, _ in ordered[:top_n]]  # candidate pool for the reranker

The full pipeline in Python

Here is the whole funnel wired together: BM25 and dense retrieval each return ~50, RRF fuses them into a pool of ~100, and a cross-encoder reranks to the top five the model will actually see. The example uses a self-hosted sentence-transformers cross-encoder; further down, the same final stage is shown as a single Cohere API call so you can compare.

from sentence_transformers import CrossEncoder

# Self-hosted cross-encoder reranker (BGE family runs the same way).
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

def retrieve_candidates(query: str, k: int = 50) -> list[dict]:
    """Stage 1 + 2: lexical (BM25) and dense (bi-encoder) retrieval, then RRF."""
    bm25_hits  = bm25_search(query, top_k=k)          # exact-term match
    dense_hits = vector_store.search(query, top_k=k)  # semantic match

    by_id = {h["id"]: h for h in bm25_hits + dense_hits}
    pool_ids = reciprocal_rank_fusion(
        [[h["id"] for h in bm25_hits],
         [h["id"] for h in dense_hits]],
        k=60, top_n=100,
    )
    return [by_id[i] for i in pool_ids]               # ~100-doc candidate pool

def rerank(query: str, top_n: int = 5) -> list[dict]:
    """Stage 4: cross-encoder scores every (query, doc) pair; keep the best."""
    candidates = retrieve_candidates(query, k=50)
    # Truncate over-long docs before scoring to protect the latency budget.
    pairs  = [(query, c["text"][:2000]) for c in candidates]
    scores = reranker.predict(pairs)                  # full cross-attention per pair
    for c, s in zip(candidates, scores):
        c["rerank_score"] = float(s)
    ranked = sorted(candidates, key=lambda c: c["rerank_score"], reverse=True)
    return ranked[:top_n]                             # the 5 you pass to the LLM

The hosted variant collapses the entire final stage into one call. You still run stages 1 to 3 yourself, then hand the candidate pool to the reranker endpoint:

import cohere
co = cohere.Client()  # COHERE_API_KEY in the environment

def rerank_hosted(query: str, top_n: int = 5) -> list[dict]:
    candidates = retrieve_candidates(query, k=50)
    docs = [c["text"][:2000] for c in candidates]
    resp = co.rerank(
        model="rerank-3",                  # or the faster "rerank-3-nimble"
        query=query,
        documents=docs,
        top_n=top_n,
    )
    return [candidates[r.index] for r in resp.results]

Three ways to deploy the reranker

The funnel is the same regardless of how you run the final stage; what changes is the operational trade-off. There are three credible options in 2026, and the right one depends on your scale and your appetite for infrastructure.

Hosted: Cohere Rerank

The lowest-friction option by a wide margin. Cohere Rerank 3 — with a faster production variant marketed as "Rerank 3 Nimble" — is one API call with no infrastructure to run. Pricing is roughly one US dollar per thousand searches, priced per search rather than per document, and it adds on the order of 200 milliseconds including the API round trip. At small-to-mid scale that is predictable to budget and almost always the right place to start: you ship the quality gain this week and worry about per-query cost only if and when volume makes it the dominant line item. An independent 2026 evaluation of Cohere Rerank walks through measuring its impact on a real pipeline.

Self-hosted: BGE and Jina cross-encoders

When query volume climbs, the per-search API cost eventually dominates, and self-hosting flips the economics. The BGE reranker family (such as bge-reranker-v2-m3) and the Jina rerankers give the best accuracy-per-dollar when you run them on your own GPU — in an AWS Mumbai or London region close to your data. You trade infrastructure and operations work, model serving, autoscaling, GPU babysitting, for a much lower marginal cost per query at high volume. The 2025 survey of top rerankers is a good map of the open-weight options and how they stack up.

ColBERT and late interaction

ColBERT takes a different route to precision. Instead of one vector per document, it stores a vector per token, and at query time it scores a query against a document with MaxSim — for each query token it takes the maximum similarity over all document tokens and sums those, a mechanism called late interaction. This recovers much of a cross-encoder's token-level discrimination while remaining partly indexable. The cost is storage: a multi-vector index is far larger than a single-vector one. In 2026 ColBERT is more of a niche than a default — for most teams the bi-encoder plus cross-encoder pipeline is simpler to operate and usually delivers equivalent quality. Where it shines is high-precision retrieval over medium corpora of roughly one thousand to ten thousand documents, where the storage cost is affordable; jina-colbert-v2 reduces that storage footprint and adds multilingual support.

Approach	Setup friction	Latency	Cost model	When to use
Cohere Rerank (hosted)	Lowest — one API call	~200 ms incl. round trip	~$1 per 1,000 searches	Shipping fast at small-to-mid scale; predictable budget
BGE / Jina cross-encoder (self-hosted)	Higher — serve on your own GPU	Depends on GPU and pool size	Infra cost; lowest marginal cost per query	High volume where best accuracy-per-dollar wins
ColBERT (late interaction)	Medium — multi-vector index to build	Low at query time; storage-heavy index	High storage cost (per-token vectors)	High-precision retrieval over ~1k–10k docs

What the gains actually look like

Reranking has a reputation as a quality lever, and it earns it — but the numbers deserve care. Guides consistently report double-digit improvements in ranking metrics such as nDCG and recall@k from adding a cross-encoder reranker over an un-reranked baseline, and one 2025 guide cites up to roughly a 40 percent accuracy gain on its own benchmark corpus. The direction is real and repeatable; the magnitude is not a promise you can carry to your own data.

The reason the gain varies so much is intuitive once you see it. A reranker improves the ordering of documents that were already retrieved. If your initial retrieval is noisy — the right answer is in the pool but scattered among near-misses — there is a lot of ordering for the reranker to fix, and the lift is large. If your retrieval is already clean and the right answer is usually at rank one or two, there is little to reorder and the lift is modest. So the published figures are best read as the answer to "how much can reranking help when retrieval is messy?", which is exactly the situation reranking exists to address.

Candidate pool reranked	Added latency (intuition)	Notes
~10 candidates	Lowest	Fine when retrieval recall is already high
~50 candidates	Moderate	Common default; good recall-vs-latency balance
~100 candidates	Higher	Maximises recall; watch long-document cost

The table is an intuition, not a benchmark: cost and latency scale with candidate count multiplied by average document length, so a hundred long candidates can cost far more than a hundred short ones. The discipline is to measure the rerank step in isolation on your own stack — and to attribute any published nDCG or recall figure to its source rather than presenting it as your own result.

Failure modes and how to avoid them

Reranking is reliable, but it has a small set of characteristic failures, and every one of them is avoidable if you know to look for it.

The first and most important: reranking cannot fix bad retrieval recall. The reranker only ever sees the candidate pool. If the document that answers the question never made it into the BM25 or dense top-50, the reranker has nothing to surface — it will faithfully rank the wrong documents in a better order. This is why the build order matters: get chunking and retrieval recall right first, then add the reranker on top. Our chunking and embedding strategies guide covers the upstream half that sets the ceiling for everything the reranker can do.

The second: reranker latency adds to every query. Unlike retrieval, which you can cache aggressively, the rerank score is a function of the live query, so it runs every time. On a tight latency SLA, rerank fewer candidates, truncate documents harder, or rerank only when retrieval confidence is low. Measure the rerank step on its own so it never becomes a mystery 200 milliseconds nobody can account for.

The third: over-long candidate documents blow the latency budget. A cross-encoder's cost grows with the combined length of the query and document, so a handful of very long chunks in the pool can dominate the entire rerank time. Truncate documents to a sensible cap before scoring, and keep your chunks reasonably sized at index time so the reranker is never asked to read a whole page to score a paragraph.

Conclusion: order is the last mile

Retrieval and reranking solve two different problems, and conflating them is what trips most teams up. Retrieval answers "is the right document anywhere in reach?" — and you make that answer yes with hybrid search, good chunking and high recall. Reranking answers "is the right document at the front?" — and you make that answer yes with a cross-encoder over a fused candidate pool. The LLM only reads the front, so the second question is the one your users actually feel. Reranking is the last mile between a document that was technically retrieved and an answer that is actually right.

As of June 2026 the build order is settled. Retrieve wide with BM25 and a dense bi-encoder; fuse the two lists with Reciprocal Rank Fusion into a pool of about a hundred; rerank that pool with a cross-encoder and pass the top five to the model. Start with a hosted reranker to ship the gain this week, move to a self-hosted BGE or Jina cross-encoder when volume makes the per-query cost worth owning, and reach for ColBERT only where its precision-over-medium-corpora trade-off genuinely fits. Whichever you choose, fix retrieval recall first, measure the lift on your own data, and treat every published nDCG number as someone else's benchmark, not your guarantee. Do that and reranking becomes what it should be: the cheapest, highest-leverage upgrade in your whole RAG stack.

For the foundations underneath the rerank — fusion mechanics, chunking and the routing that decides when to even bother — start with our hybrid retrieval guide and the companion on adaptive RAG query routing. Retrieval gets the document into the room; reranking gets it to the front; routing decides whether you needed the heavy machinery at all.