Adaptive RAG: Route Each Query From Naive to Agentic

What you need to know

• One pipeline for every query is a mistake. A single retrieval pattern either over-serves simple lookups, burning money and seconds you did not need to spend, or under-serves hard questions and returns a thin, wrong answer. Adaptive RAG matches the query to the right amount of machinery.
• There are three tiers worth building. Naive RAG (embed, top-k vector search, stuff, generate) for simple facts; hybrid retrieval plus a cross-encoder reranker for harder questions; and an agentic loop that plans, retrieves, judges sufficiency and re-retrieves for synthesise-and-cite tasks.
• Reranking is the highest-ROI single change you can make. Retrieve a wide candidate set, then re-score the top results against the query with a cross-encoder. It needs no re-indexing and commonly lifts answer quality by roughly 15 percent — an illustrative figure you should confirm on your own eval set.
• A lightweight classifier does the routing. A small model or a cheap LLM call labels each query simple, moderate or complex, and a router sends it to the matching pipeline. A faithfulness or confidence check provides the escalation path when the classifier guesses wrong.

If you have run a retrieval-augmented generation system in production for more than a quarter, you have watched the same tension play out. Your simplest questions — "what is the refund window?", "which region hosts the Mumbai cluster?" — get answered instantly and cheaply by a plain vector search. Then a user asks something genuinely hard — "compare how our India and UK contracts handle data deletion, and cite the clauses" — and the same plain vector search returns three loosely related chunks and a confident, shallow answer that is subtly wrong. So you upgrade the whole system to handle the hard question, and now every "what is the refund window?" costs ten times as much and takes ten times as long. You have optimised for your worst case at the expense of your common case.

Adaptive RAG is the resolution to that tension, and it is where the state of the art sits as of mid-2026. Rather than picking one pipeline and forcing every query through it, you build a small ladder of pipelines and put a router at the front that decides, per query, which rung to climb to. Cheap questions stay cheap. Expensive questions get the expensive treatment they need. This guide walks the full ladder — naive, hybrid-plus-rerank, agentic — shows you the classifier and router that tie them together, gives you working Python for the query classifier and the reranking pipeline, and lays out the cost and latency trade-offs you need to defend the design to whoever signs off on your inference bill.

The three tiers, from naive to agentic

Before you can route between pipelines, you need to understand what each one buys you and where it stops paying. Think of these as three rungs on a ladder, each more capable, more expensive and slower than the last.

Tier 1 — Naive RAG

The original recipe, and still the right tool for a large share of real queries. The query is embedded, you run a top-k similarity search against your vector store, you stuff the retrieved chunks into the prompt, and the model generates. It is a single round trip with no reranking, no query rewriting and no second pass. For simple factual lookups where the answer lives in one chunk and the wording of the question closely matches the wording of the source, naive RAG is fast, almost free, and entirely adequate.

Its ceiling is real, though. On anything more complex than a direct lookup — questions that need information spread across several chunks, that use vocabulary different from the source, or that require any reasoning over the retrieved material — naive RAG plateaus somewhere around 70 to 80 percent precision. That last 20 to 30 percent is exactly the set of questions your users remember, because those are the ones it gets wrong.

Tier 2 — Hybrid retrieval plus reranking

The workhorse middle tier, and the one most teams under-invest in. Two upgrades stack here. First, hybrid retrieval: you run both dense vector search and sparse keyword search (BM25) and merge the results, so you catch both semantic matches and exact-term matches — model numbers, error codes, proper nouns and clause references that embeddings alone often miss. Our production guide to hybrid retrieval covers the fusion mechanics in depth.

Second, and this is the part with the highest return on effort, reranking. Initial retrieval — dense, sparse or hybrid — scores the query and each document independently and then sorts by similarity. That independence is the weakness: a chunk that is genuinely the best answer can rank fifteenth because its standalone embedding sits a little far from the query's. A cross-encoder reranker fixes this by re-scoring the query and each candidate together, with full cross-attention between them, so it judges relevance the way a human skim-reader would. The standard pattern is to retrieve a wide net — top-20 or top-50, ideally from hybrid search — and rerank down to the top-5 you actually pass to the model.

Tier 3 — Agentic RAG

The top rung, for the questions the first two tiers cannot reach. Agentic RAG replaces the linear retrieve-then-generate pipeline with an autonomous agent that runs a loop. It plans an approach, retrieves, evaluates whether the context it has gathered is sufficient to answer, and if it is not, reformulates the query or fans out new sub-queries and retrieves again — for as many passes as the question needs. Some implementations route between a vector store and a knowledge graph, or call tools mid-loop. The defining feature is the self-assessment: the agent decides for itself whether it has enough to answer, rather than answering from whatever the first retrieval happened to return.

That capability is not free. An agentic answer can run several model calls deep, take on the order of 10 to 15 seconds, and cost meaningfully more per query than the other tiers. It is the right choice for hard, multi-hop, synthesise-and-cite questions where the user is asking for analysis and will wait for a good answer. It is straightforwardly the wrong choice for "find the chunk that answers this", where you are paying agent prices and agent latency to do a job a vector search finishes in a fraction of a second.

Why reranking is the highest-ROI change you can make

If you take one practical step away from this article, make it this one: put a cross-encoder reranker into your pipeline. It is the single change with the best ratio of answer-quality gain to engineering effort, and most teams running naive RAG have simply never tried it.

The reason it works comes down to architecture. A bi-encoder — the model behind your vector store — encodes the query and each document into vectors separately and then compares them with a cheap distance metric. That separation is what makes vector search fast and scalable: you embed every document once, at index time, and at query time you only embed the question. But it means the model never actually looks at the query and the document at the same time. A cross-encoder does exactly that — it feeds the query and one candidate document into the model together and runs full attention across both, producing a single relevance score that reflects how the two genuinely relate. It is far too slow to run over your whole corpus, but perfectly fast over the 20 or 50 candidates that initial retrieval already narrowed down to.

So the production pattern is a funnel: cast a wide, cheap net with hybrid retrieval, then apply the expensive-but-accurate cross-encoder only to the survivors. Hybrid retrieve top-50, rerank to top-5, pass those five to the model. In practice this commonly improves answer quality by roughly 15 percent over the un-reranked baseline — treat that as an illustrative order of magnitude and measure it on your own data, because the gain depends heavily on how noisy your initial retrieval is. The managed options here are mature: Cohere's Rerank exposes this as a single API call, and on the open-weight side a BGE cross-encoder reranker runs comfortably on your own GPUs in AWS Mumbai or London.

from FlagEmbedding import FlagReranker
from rank_bm25 import BM25Okapi

# --- Tier 2: hybrid retrieve, then cross-encoder rerank ---------------------

reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

def hybrid_retrieve(query: str, k: int = 50) -> list[dict]:
    """Merge dense vector hits and sparse BM25 hits into one candidate set."""
    dense_hits  = vector_store.search(query, top_k=k)        # semantic match
    sparse_hits = bm25_search(query, top_k=k)                # exact-term match
    # De-duplicate by chunk id, keeping the best-ranked occurrence.
    merged = {h["id"]: h for h in (sparse_hits + dense_hits)}
    return list(merged.values())

def retrieve_and_rerank(query: str, top_n: int = 5) -> list[dict]:
    candidates = hybrid_retrieve(query, k=50)               # wide, cheap net
    pairs = [[query, c["text"]] for c in candidates]
    scores = reranker.compute_score(pairs, normalize=True)  # full cross-attention
    for c, s in zip(candidates, scores):
        c["rerank_score"] = s
    ranked = sorted(candidates, key=lambda c: c["rerank_score"], reverse=True)
    return ranked[:top_n]                                   # the 5 you actually pass on

Adaptive RAG: match the query to the pipeline

Now we assemble the tiers into a single system. The core idea of adaptive RAG is almost embarrassingly simple to state: do not pick one pipeline, pick a pipeline per query. A lightweight classifier reads the incoming question and decides how hard it is, and a router sends it to the cheapest tier that can answer it. Simple lookups go to naive RAG. Harder, more specific questions go to hybrid-plus-rerank. Genuinely complex, multi-hop, synthesise-and-cite questions go to the agentic loop. The expensive machinery only runs when the question has earned it.

The classifier is the heart of the system, and it should itself be cheap — running an expensive model to decide whether to run an expensive pipeline defeats the purpose. Two implementations work well. The first is a small fine-tuned classifier trained on a few hundred labelled queries, which is fast and nearly free per call. The second, and the one most teams start with, is a single low-latency LLM call with a tight prompt and a constrained label set. Either way the output is one of a small number of complexity classes that map to your tiers.

from openai import OpenAI

client = OpenAI()

# --- Adaptive RAG: classify the query, then route to the right tier ---------

CLASSIFY_PROMPT = """You are a query router for a retrieval system.
Classify the user's question into exactly one complexity class:

- "simple"   : a single factual lookup answerable from one passage.
- "moderate" : needs specific terms, comparison, or a few related passages.
- "complex"  : multi-hop synthesis across many sources, or a cited analysis.

Respond with only the class word."""

def classify_query(question: str) -> str:
    """Cheap, fast routing decision — a small model is ideal here."""
    resp = client.chat.completions.create(
        model="gpt-5.2-mini",                  # use the cheapest capable model
        messages=[
            {"role": "system", "content": CLASSIFY_PROMPT},
            {"role": "user", "content": question},
        ],
        max_tokens=1,
        temperature=0,
    )
    label = resp.choices[0].message.content.strip().lower()
    return label if label in {"simple", "moderate", "complex"} else "moderate"

def answer(question: str) -> dict:
    """Route the query to the cheapest pipeline that should answer it."""
    tier = classify_query(question)

    if tier == "simple":
        context = naive_retrieve(question, k=4)          # Tier 1: one vector pass
    elif tier == "moderate":
        context = retrieve_and_rerank(question, top_n=5)  # Tier 2: hybrid + rerank
    else:
        return agentic_answer(question)                   # Tier 3: plan + re-retrieve loop

    result = generate(question, context)

    # Escalation safety net: if a cheap path's answer is not well grounded,
    # re-route the same query one tier up rather than shipping a weak answer.
    if tier != "complex" and not passes_faithfulness_check(result, context):
        return agentic_answer(question)

    return result

Notice the escalation at the bottom of answer(). The classifier will sometimes guess wrong — that is not a bug to eliminate but a behaviour to plan for. By running a grounding or faithfulness check on the answer from a cheap tier and escalating to the agentic path when it fails, you turn mis-routing from a quality failure into, at worst, a latency cost on a small fraction of queries. The check itself is where retrieval evaluation meets routing; our sibling guide on RAGAS faithfulness and context precision covers exactly the metrics you would wire in here as the escalation trigger.

The routing architecture, end to end

It helps to see the whole flow laid out as a pipeline rather than as scattered functions. Here is what a request does from the moment it arrives, with the decision points called out.

Stage	What happens	Decision / output
1 · Classify	Small model or cheap LLM call labels the query complexity	simple · moderate · complex
2a · Route → naive	Embed, top-k vector search, stuff, generate	One pass, sub-second answer
2b · Route → hybrid + rerank	Dense + BM25 retrieve top-50, cross-encoder rerank to top-5, generate	Two-stage funnel, low-single-digit seconds
2c · Route → agentic	Agent plans, retrieves, judges sufficiency, re-retrieves in a loop	Multi-pass cited answer, 10–15s
3 · Ground-check	Faithfulness / confidence check on a cheap-tier answer	Pass → return · Fail → escalate one tier up
4 · Observe	Log tier chosen, latency, cost, ground-check result per query	Feeds classifier tuning and tier mix review

Stage 4 is not optional. You cannot tune a router you cannot see. Log the tier each query took, the latency and cost it incurred, and whether the ground-check passed, and you can answer the questions that actually matter: is the classifier sending too much traffic to the agentic tier, are simple questions getting escalated too often, is the reranker latency eating your budget? Our guide on hybrid retrieval and agent observability goes deep on the instrumentation; the short version is that every routed query should leave a trace you can aggregate.

Cost and latency: the case for routing

The argument for adaptive RAG is, in the end, an economic one, so it is worth putting numbers on it — with a heavy caveat. The figures below are illustrative orders of magnitude, not benchmarks. Your real numbers depend on your models, your token counts, your hosting and your corpus, and you must measure them on your own stack before you quote them to anyone. Treat the table as a shape, not a price list.

Pipeline	Cost per query (indicative)	Latency (indicative)	Best for
Naive RAG	~ $0.001	Sub-second	Simple factual lookups, single-chunk answers
Hybrid + rerank	~ $0.005	~ 1–3 seconds	Specific, comparative or few-passage questions
Agentic RAG	~ $0.02–0.10	~ 10–15 seconds	Multi-hop synthesis, cited analysis, hard questions

The leverage is in the spread. The agentic tier can cost twenty to a hundred times what a naive query costs and take an order of magnitude longer. If your traffic looks like most real systems — a fat head of simple questions and a thin tail of hard ones — then forcing everything through the agentic pipeline means paying the top-tier price on the eighty or ninety percent of queries that never needed it. Routing inverts that: you pay top-tier prices only on the tail, and the head stays cheap and instant. The result is the pattern Rishi describes above — average cost per query falls sharply while hard-question quality rises, because the saved budget is spent where it matters.

There is a broader cost discipline here that goes beyond retrieval. Routing by complexity is one instance of a general principle — send each request to the cheapest resource that can serve it — that also governs model selection, caching and prompt compression. If adaptive RAG is saving you money on retrieval, the same mindset applied across your whole inference path compounds the win; our cache, route, compress cost playbook covers the rest of that surface.

Practical guidance for shipping it

Start with two tiers, not three

You do not have to build the whole ladder on day one, and you probably should not. The fastest path to value is a two-tier system: naive RAG for simple queries and hybrid-plus-rerank for everything else. That alone captures most of the quality gain and most of the cost discipline, because the reranker does so much heavy lifting. Add the agentic tier only once your evals show a real, identifiable slice of questions that the second tier genuinely cannot answer — multi-hop synthesis, cited cross-document analysis — rather than building it speculatively. Over-engineering when naive-plus-rerank already clears your eval bar is the most common way teams waste a quarter on this.

Tune the classifier against a labelled eval set

The router is only as good as the classifier, and the classifier is only as good as the data you tune it on. Pull a few hundred real queries from your logs, label them by the tier that actually answered them best, and treat that as a routing eval set. Measure not just classification accuracy but the downstream cost of errors: a simple-labelled-as-complex mistake costs you money and latency, while a complex-labelled-as-simple mistake costs you a wrong answer unless the escalation catches it. Tune the thresholds to make the cheap errors common and the expensive ones rare.

Treat the agentic tier as an evaluable system, not a black box

When you do build the agentic tier, instrument its loop. An agent that plans, retrieves and re-retrieves has a trajectory — a sequence of decisions and tool calls — and that trajectory is what you debug and improve, not just the final answer. The same evaluation discipline you apply to any agent applies here: score the trajectory, the tool calls and the outcome separately. Our guide on evaluating AI agents by trajectory, tool call and outcome lays out how, and it is the difference between an agentic tier you can trust in production and one that occasionally spirals into a ten-call loop nobody can explain.

Keep contextual retrieval and hybrid search underneath everything

Routing decides which pipeline runs, but every pipeline retrieves from the same index, so the quality of that index sets the ceiling for all three tiers at once. Contextual retrieval — situating each chunk in its document before embedding — reduces failed retrievals across the board, and hybrid dense-plus-sparse search widens the net every tier draws from. Invest in the index before you invest in elaborate routing, because a clever router over a poor index just chooses efficiently between bad answers. For Bengaluru and London teams alike serving regulated data, the index also lives in your residency region — AWS Mumbai or London — and contextual retrieval plus hybrid search are the techniques that let a self-hosted, in-region index match the retrieval quality of anything hosted elsewhere.

Conclusion: complexity on demand, not by default

The mistake adaptive RAG corrects is treating retrieval as a fixed pipeline rather than a decision. For two years the field climbed a ladder — naive RAG plateaued, so teams added hybrid search and reranking; hard questions still slipped through, so teams added agentic loops. Each rung was a real improvement, but applying the top rung to every query is its own kind of failure: you pay the cost of complexity on questions that never needed it. The 2026 answer is not a better single pipeline. It is a router that spends complexity only where complexity pays.

The build order is clear. Get your chunking and contextual retrieval right so every tier draws from a strong index. Add a cross-encoder reranker over hybrid retrieval — the single highest-ROI change, and the one most teams have not made. Put a cheap classifier in front to route simple questions to the cheap path, and wire a faithfulness check as the escalation net for when it guesses wrong. Add the agentic tier last, only for the thin slice of questions that earn it, and instrument its trajectory so you can trust it. Do that and you get the thing every team actually wants: cheap, instant answers to the easy questions, genuinely good answers to the hard ones, and an inference bill that reflects the work each question really required.

For the retrieval foundations underneath all of this — fusion, chunking and the metrics that tell you whether any of it is working — start with our hybrid retrieval guide and today's sibling on RAGAS faithfulness and context precision. Routing decides which pipeline runs; evaluation tells you whether it should have.