When Your Agent Has 200 Tools: Retrieval and Deferred Schemas

What you need to know

• Tool count grows by accretion, not design. Each integration adds ten to forty tools and nobody removes one.
• Three separate costs — tokens, selection accuracy and latency — and they do not respond to the same fix.
• Prompt caching addresses tokens only. The model still reads every definition, so accuracy is untouched.
• Retrieval controls breadth: embed descriptions and example invocations, retrieve top-k per task, pin a small must-have set.
• Deferred schemas control depth: names and one-line summaries first, full parameter shapes only when needed.
• Adding a tool is a migration. Re-run the selection eval before it ships, as you would for a schema change.

How an agent quietly ends up with 200 tools

The arithmetic is unremarkable and that is precisely why it catches teams out. A support agent at a Bengaluru SaaS company starts with a handful of internal functions — look up an account, fetch a subscription, open a ticket. Six tools, hand-written and well described. Then someone connects the ticketing system's MCP server, which brings twenty-two tools because it exposes every CRUD operation on every object type. Then the CRM server, thirty-one. Then a documents server, fourteen. Then an internal billing wrapper with eighteen endpoints, because exposing all of them was easier than deciding which mattered.

Nine months later the tool block has two hundred and six entries and nobody on the team can name thirty of them from memory. The Model Context Protocol, an open standard for connecting agents to external tools and data, made each of those additions trivially cheap — you point the client at a server and its whole tool surface arrives. That is the point of the standard. The consequence is that the marginal cost of adding a tool has dropped to near zero while the marginal cost of having one has not.

The same story plays out at a London insurance platform with a different cast: policy administration, a claims workflow engine, document generation, a fraud-scoring API and two regulatory-reporting integrations. Different domain, identical curve.

The symptom curve

What is worth learning is how this presents, because it rarely presents as "too many tools". It shows up in three stages.

Stage one — near-miss selection. The agent starts choosing tools that are almost right. It calls search_tickets when it should have called search_ticket_comments. It calls the CRM's get_contact when the user asked about an account, because both descriptions mention "customer". Individually these look like prompt problems, so the team rewrites descriptions — which helps a little and then stops helping.

Stage two — cost on empty turns. Someone notices that turns where the agent calls nothing — a clarifying question, a greeting, a refusal — still cost real money. That is the tell. If a turn producing forty output tokens costs the same as a turn that does substantive work, the cost is in the fixed prefix, and the tool block is usually the largest part of it.

Stage three — latency drift. Time-to-first-token creeps upward without correlating with anything the team changed. Larger prompts take longer to process, and a tool block that quietly tripled shows up here first.

What 200 schemas actually cost you

Three costs, and they are genuinely separate. Conflating them is why teams reach for the wrong fix.

Cost one: tokens

Tool definitions are part of the prompt, re-sent on every turn unless caching intervenes. Their size depends entirely on how verbose your schemas are, so any table has to state its assumption openly.

Assumption for the table below: roughly 150 tokens per tool definition, including the name, the description and a small parameter schema. Measure yours — it will differ, probably by a lot. A terse tool with two string parameters might cost 60 tokens; a generated wrapper around a REST endpoint with fifteen optional query parameters and enum documentation can exceed 500. Run a token counter over your actual serialised tool block rather than trusting this figure.

Illustrative prompt overhead by tool count, assuming ~150 tokens per tool definition. Your figures will differ; measure your own serialised tool block.

Tools exposed	Est. tokens in the tool block	Share of a 200k context window	Uncached tokens across a 12-turn conversation
10	~1,500	0.8%	~18,000
25	~3,750	1.9%	~45,000
50	~7,500	3.8%	~90,000
100	~15,000	7.5%	~180,000
200	~30,000	15%	~360,000
400	~60,000	30%	~720,000

The final column is the one that stings. At two hundred tools, a twelve-turn conversation pays for the tool block twelve times over, and most of those tokens describe capabilities that were never relevant.

Cost two: selection accuracy

This is the expensive one and it appears on no invoice. As the list grows, near-duplicate tools crowd each other. Two tools whose descriptions both begin "Retrieve information about a customer" compete for the same intent, and the model must disambiguate them from text written at different times, in different registers, for different audiences.

The research line on tool-augmented models has been circling this for years. Toolformer established that models can learn to call APIs; Gorilla and ToolLLM both address the case where the API surface is far too large to fit in a prompt, and both converge on the same structural answer: retrieve a relevant subset before asking the model to choose. The details differ; the shape of the solution does not.

Cost three: latency

Larger prompts take longer to process before the first token appears. This is a smaller effect than the other two on most providers, and caching mitigates much of it, but it is felt most on the interactive surfaces where users watch a cursor blink.

What prompt caching does and does not fix

Prompt caching reduces the repeated cost of a stable prefix — you pay to write the block into the cache once, then a much reduced rate to read it on later turns. If your tool block is genuinely stable across a conversation, caching is the highest-return change available and you should make it before anything else in this guide. Our guide to prompt caching across Claude, GPT and Gemini covers the mechanics; the provider documentation on prompt caching is the authority on current behaviour.

Note carefully what caching leaves untouched. The model still reads every definition. Nothing about a cache hit makes two hundred schemas easier to disambiguate. Selection accuracy is a function of what is in the context window, not of how the tokens got there.

Four strategies, compared

Four broad approaches are in use as of August 2026, and most mature systems combine two.

Static full list is the baseline: every tool, every turn. It is the most accurate approach at small counts, because nothing was wrongly filtered out. It fails by degrees rather than suddenly.

Role or context gating exposes only the toolset relevant to the current task phase or the caller's permission scope. A claims agent in the triage phase does not need document-generation tools; a read-only analyst account should never see the write tools at all. It is deterministic, trivially debuggable and needs no embedding infrastructure. Its weakness is that somebody maintains the mapping, and a task spanning phases needs an escape hatch.

Semantic tool retrieval embeds each tool's description and retrieves the top-k against the user's request. It scales to very large surfaces without anyone maintaining a mapping. Its failure mode is silent and specific: a rarely-used but critical tool never ranks highly enough to be seen, so the agent behaves as though the capability does not exist.

Hierarchical or namespace routing exposes a few domain-level meta-tools — crm, billing, documents — which expand into their namespace on demand. The model reasons about domains first and individual tools second, which matches how people think about capability. It costs an extra round trip and requires namespaces that are genuinely distinct, which they often are not.

The four strategies scored on the dimensions that decide the choice.

Strategy	Selection accuracy	Latency	Implementation cost	Characteristic failure mode
Static full list	Best at small counts, degrades steadily above it	Worst — full block every turn	None	Near-miss selection among crowded duplicates
Role / context gating	High within a phase; blind across phases	Good — small stable block, cache-friendly	Low, but ongoing mapping maintenance	Task spans phases and the needed tool is not exposed
Semantic retrieval	Good, and roughly flat as the catalogue grows	Good, plus an embedding call per retrieval	Moderate — index, evals, refresh pipeline	A rare but critical tool never surfaces
Hierarchical routing	High when namespaces are genuinely distinct	One extra round trip per namespace opened	Moderate — needs a clean namespace design	Wrong namespace chosen; agent never recovers

A recommendation ladder by tool count

What to do at each scale. Thresholds are judgement calls, not measured constants — validate against your own selection eval.

Tools in the catalogue	Recommended approach	Where to spend the effort
Under ~20	Static full list	Writing better descriptions and error messages
~20–60	Role / scope gating, static within a phase	Defining phases and the permission scope map
~60–200	Gating plus semantic retrieval, sticky per task	Tool index quality and a selection eval set
Above ~200	Retrieval plus namespace routing and deferred schemas	Namespace design and pinned-tool policy

Before implementing any of it, do the unglamorous thing: audit the catalogue and delete. Removing a tool is strictly better than retrieving around it.

Building a tool index

Now the practical build. The most consequential decision is not which embedding model you use — it is what text you embed.

Embed the description, example invocations and the questions the tool answers. Do not embed the raw JSON schema. Schemas are dominated by type names, enum values, required-field markers and structural boilerplate. Two entirely unrelated tools that both take {id: string, limit: integer} land as near neighbours in embedding space, while the user's actual query — "how many open claims does this policyholder have" — looks like neither. The parameter schema tells you how to call a tool; it says almost nothing about when to.

For each tool, build a small document: the name, a one-line summary, the full description, three or four example user utterances that should route to it, and a short list of the tools it is most often confused with (used for hard-negative checks later, not for embedding). If your descriptions are not good enough to embed usefully, that is the real finding, and designing tools for AI agents covers how to write them properly.

import math
from dataclasses import dataclass, field


def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    return dot / (na * nb) if na and nb else 0.0


@dataclass
class ToolDoc:
    name: str
    namespace: str          # "crm", "billing", "documents"
    summary: str            # one line, shown in compact mode
    description: str        # full natural-language description
    examples: list[str] = field(default_factory=list)
    scopes: list[str] = field(default_factory=list)   # required permissions
    pinned: bool = False    # always exposed, never filtered out

    def index_text(self) -> str:
        # Embed intent, NOT the JSON schema.
        parts = [self.name.replace("_", " "), self.summary, self.description]
        parts.extend(f"Example request: {e}" for e in self.examples)
        return "\n".join(parts)


class ToolIndex:
    def __init__(self, embed_fn, schemas=None):
        self.embed = embed_fn
        self.schemas = schemas or {}      # name -> raw JSON parameter schema
        self.docs: list[ToolDoc] = []
        self.vectors = []
        self._by_name: dict[str, ToolDoc] = {}

    def build(self, docs: list[ToolDoc]) -> None:
        self.docs = list(docs)
        self._by_name = {d.name: d for d in self.docs}
        self.vectors = self.embed([d.index_text() for d in self.docs])

    def get(self, name: str):
        return self._by_name.get(name)

    def raw_schema(self, name: str):
        return self.schemas.get(name, {"type": "object", "properties": {}})

    def select(self, query, caller_scopes, k=8, floor=0.30, namespace=None):
        """Return the tools to expose for this turn."""
        # 1. SCOPE FILTER FIRST. Never rank then filter.
        allowed = [
            (d, v) for d, v in zip(self.docs, self.vectors)
            if all(s in caller_scopes for s in d.scopes)
            and (namespace is None or d.namespace == namespace)
        ]

        # 2. Rank what remains.
        qv = self.embed([query])[0]
        scored = [(cosine(qv, v), d) for d, v in allowed]
        scored.sort(key=lambda t: -t[0])

        # 3. Top-k above a floor score. A weak match is worse than no match.
        picked = [d for s, d in scored[:k] if s >= floor]

        # 4. Pinned tools are always present, regardless of retrieval.
        pins = [d for d, _ in allowed if d.pinned]
        seen = {d.name for d in picked}
        return picked + [p for p in pins if p.name not in seen]

Four details in that snippet carry most of the value. The scope filter runs before ranking — a security property, not an optimisation, and the section on failure modes returns to it.

The floor score matters as much as k. Without it, a query about the weather retrieves your eight least-irrelevant billing tools and presents them as plausible. A weak match is worse than no match: it invites a wrong call. With a floor, an off-topic query yields only the pinned tools and the agent answers in natural language as it should.

Pin a small must-have set. At minimum a finish or answer tool, so the agent can always terminate cleanly, and an escalate or hand-off tool, so it can always route to a human. Keep the list short — every pin is a permanent tax — but never let retrieval strand the agent with no way out.

Re-retrieval policy is a real design decision. Per-turn retrieval adapts when a conversation changes subject; sticky-per-task preserves prompt caching and gives the model a stable world. Default to sticky with an expansion trigger: keep the task's tool set, and re-retrieve only when the new user message scores below the floor against every currently exposed tool. That check is cheap and it catches genuine topic changes without churning the cache every turn.

Embedding model choice matters less than description quality, but it is not nothing, and the honest way to decide is to benchmark on your own tool queries rather than a public leaderboard — see choosing an embedding model by benchmarking on your own data.

Deferred schemas: load the name, not the whole shape

Retrieval controls how many tools the model sees. Deferred schemas control how much of each one it sees, and the two compose cleanly.

The pattern: the base prompt carries only compact entries — a tool name and a one-line summary, perhaps fifteen to twenty-five tokens each rather than a hundred and fifty. When the model decides it wants to call something, it first fetches that tool's full parameter schema through a meta-tool, then makes the real call with correct arguments. One extra round trip buys a base prompt a fraction of the size.

The trade is favourable in exactly the cases you would expect. If a typical turn calls one or two tools out of a hundred and fifty, you are paying full price for a hundred and forty-eight schemas the model never used. If a typical turn calls twelve tools out of fifteen, deferral is pure overhead. Know which shape your workload has before adopting it.

# Two meta-tools replace the full schema block.
#
# The base prompt now carries a compact catalogue:
#   crm_get_account          - Fetch an account by ID or registered email
#   crm_search_contacts      - Find contacts by name, company or phone
#   billing_list_invoices    - List invoices for an account, newest first
#   ... 150 more lines, ~20 tokens each

META_TOOLS = [
    {
        "name": "search_tools",
        "description": (
            "Find tools that can accomplish a described task. "
            "Use this when no compact catalogue entry is an obvious match."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "task": {"type": "string",
                         "description": "What you are trying to do, in plain English"},
                "limit": {"type": "integer", "default": 5},
            },
            "required": ["task"],
        },
    },
    {
        "name": "load_tool_schema",
        "description": (
            "Load the full parameter schema for one or more tools. "
            "Call this before invoking any tool whose schema you have not yet loaded."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "names": {"type": "array", "items": {"type": "string"},
                          "description": "Tool names from the catalogue"},
            },
            "required": ["names"],
        },
    },
]


def handle_meta_tool(call, index, caller_scopes, loaded: set):
    if call.name == "search_tools":
        hits = index.select(call.args["task"], caller_scopes,
                            k=call.args.get("limit", 5))
        return [{"name": d.name, "summary": d.summary,
                 "namespace": d.namespace} for d in hits]

    if call.name == "load_tool_schema":
        out, denied = [], []
        for name in call.args["names"]:
            doc = index.get(name)
            if doc is None or not all(s in caller_scopes for s in doc.scopes):
                # Same response for "missing" and "not permitted":
                # do not let schema loading enumerate tools the caller cannot use.
                denied.append(name)
                continue
            loaded.add(name)
            out.append({"name": doc.name, "description": doc.description,
                        "input_schema": index.raw_schema(name)})
        if denied:
            return {"loaded": out,
                    "unavailable": denied,
                    "hint": "These tools are not available to you. Do not retry."}
        return {"loaded": out}

    raise ValueError(call.name)

Two implementation notes. Keep the compact summary genuinely one line and genuinely useful — it is the only text the model has, so "Fetch an account by ID or registered email" earns its place where "Account operations" does not. And return the same shape for "does not exist" and "exists but you cannot use it", so schema loading cannot enumerate the catalogue.

As of August 2026 several agent runtimes and SDKs ship variants of this natively, under names like deferred, lazy or searchable tool definitions, and the MCP ecosystem has converged on similar ideas for large servers. Treat this as a pattern rather than any one product's feature: check the Model Context Protocol documentation and your provider's tool use documentation for what exists natively before building it yourself, and prefer the native path — it will be better integrated with caching than anything you assemble.

A third lever is worth knowing: having the model write code that calls tools programmatically rather than emitting one tool call per turn. That collapses multi-tool sequences into a single generated script and cuts token traffic sharply for workflows that chain many calls; code execution with MCP covers it. It complements everything here rather than replacing it.

Measuring selection accuracy

You cannot tune what you do not measure, and "the agent feels better" is not a measurement. Build a labelled set of request-to-tool pairs and track it like any other eval.

Source the pairs from production traces, not imagination. Take real user messages and the tool call the agent made, and have someone label whether that call was correct — and if not, which tool was. A hundred to three hundred pairs is enough to be useful, and labelling is quick because most cases are obvious. Include a deliberate slice of no-tool-needed examples: greetings, clarifications, questions the agent should answer from context. Without them you will never detect the failure where retrieval hands over eight plausible tools and the model dutifully calls one.

The four metrics worth tracking, and what a regression in each one means.

Metric	Definition	What a drop tells you
Top-1 recall	Correct tool ranked first by retrieval	Descriptions are crowding; two tools compete for one intent
Top-k recall	Correct tool appears anywhere in the exposed set	k or the floor score is too aggressive — the model never had a chance
Wrong-tool rate	Agent called a tool other than the labelled correct one	Retrieval was fine but disambiguation failed; a description problem
No-tool false positives	Agent called a tool on a no-tool-needed case	Floor score too low; weak matches are being presented as plausible

def evaluate_selection(index, labelled, caller_scopes, k=8, floor=0.30):
    """Measure the RETRIEVAL layer only: what the model was shown.

    Each labelled example is {"query": str, "tool": str | None},
    where None means "this turn needed no tool at all".
    """
    top1 = topk = missed = fp = 0

    for ex in labelled:
        picked = index.select(ex["query"], caller_scopes, k=k, floor=floor)
        names = [d.name for d in picked]
        gold = ex["tool"]            # None means "no tool needed"

        if gold is None:
            # Only pinned tools should survive the floor here.
            if any(not d.pinned for d in picked):
                fp += 1
            continue

        if names and names[0] == gold:
            top1 += 1
        if gold in names:
            topk += 1
        else:
            missed += 1              # never exposed: the model had no chance

    graded = sum(1 for e in labelled if e["tool"] is not None)
    no_tool = len(labelled) - graded
    return {
        "top1_recall": top1 / max(graded, 1),
        "topk_recall": topk / max(graded, 1),
        "missed_entirely": missed / max(graded, 1),
        "no_tool_false_positive": fp / max(no_tool, 1),
        "k": k, "floor": floor, "catalogue_size": len(index.docs),
    }


# Run this in CI on every change to the tool registry.
# Fail the build if top1_recall drops more than a fixed margin
# below the committed baseline.

One boundary worth being explicit about: that function scores the retrieval layer, not the agent. It answers "was the right tool in front of the model", and missed_entirely is simply the complement of top-k recall, kept separate because it is the number you argue about in a review. The wrong-tool rate in the table above needs a second pass — run the agent on the same labelled set and compare the tool it actually called against the label. Keep the two measurements apart, because they have different fixes: a retrieval miss is a k, floor or index problem, while a wrong call from a correctly retrieved set is almost always a description problem.

The discipline that makes this pay off: treat the tool registry like a schema migration. Adding a tool is not risk-free — a new tool can steal top-1 from an existing one whose queries it partially matches, and the regression surfaces in a workflow nobody was thinking about. Wire the selection eval into CI, gate on top-1 recall against a committed baseline, and require a review when a new tool moves any existing tool's ranking. Evaluating agents on trajectory, tool calls and outcome covers the layer above this one.

Failure modes and how they present

Five failure modes recur, each with a distinctive signature.

Near-duplicate tools. Two tools whose descriptions overlap materially split the retrieval score and neither ranks cleanly. This presents as an agent that alternates between them for the same request on different runs. The fix is editorial, not technical: rewrite both descriptions so each states explicitly what it is not for. "Use this for account-level billing history. For individual line items on a single invoice, use billing_get_invoice_lines instead."

The tool that never surfaces. A capability used twice a month by one team — a regulatory export, an emergency policy override — has few traces, thin descriptions and no example utterances, so it ranks poorly forever. It presents as the agent claiming it cannot do something it demonstrably can. Detect it by tracking retrieval rate across the catalogue and investigating anything at zero; fix it by pinning the tool within the relevant role, or by writing the example utterances by hand because production will never supply them.

Stale descriptions. A tool's behaviour changes; its description does not. Retrieval keeps ranking it for queries it no longer serves, and the agent keeps calling it and getting results that are subtly wrong. This is the hardest to detect because nothing errors. The structural defence is to keep the description inside the tool's own definition rather than a separate registry entry, so it travels with the code, and to rebuild the index automatically whenever a definition's hash changes.

The silent cost of the tool nobody picks. A tool that exists but is never selected costs tokens on every uncached turn, occupies embedding space where it can crowd out a neighbour, and carries maintenance and security surface. It is the clearest win available and almost nobody takes it. Once a quarter, join the catalogue against your call logs and delete or archive anything with zero production calls. If deletion is politically difficult, move it behind a namespace so it costs one catalogue line rather than a full schema.

Retrieval hiding a design problem. Named last because it is the most common. Sometimes two hundred tools is the symptom, and the disease is that one MCP server exposes every CRUD operation on every object when the agent needs four of them. Writing a thin server that exposes those four well-described operations is often less work than building a retrieval layer, and produces a better agent. If you have never written one, building your first MCP server is a two-hour exercise.

Conclusion: what to do next

Do these in order. Each is worth doing alone, and each makes the next easier to evaluate.

First, measure. Count your tools, serialise the tool block, token-count it, and work out what fraction of a typical conversation's cost it represents. Half the teams reading this will find the number is fine and can stop here, which is a genuinely useful result.

Second, delete. Join the catalogue against production call logs and remove what is never used. No engineering required, and it improves every metric in this guide.

Third, cache. If the tool block is stable, prompt caching is the cheapest large saving available and it changes the economics downstream.

Fourth, build the eval before the retrieval. A hundred labelled request-to-tool pairs takes an afternoon, and without them you cannot tell whether retrieval helped or quietly broke a workflow that was working.

Fifth, gate, then retrieve, then defer. Scope gating is deterministic and cheap. Semantic retrieval comes next, sticky-per-task so it does not fight your cache, with a floor score and a short pinned set. Deferred schemas last, and only if a typical turn uses a small fraction of the catalogue.

One closing thought. Tool architecture is invisible in a CV and obvious in a conversation — anyone who can explain why they filter by scope before ranking, or why they chose sticky retrieval over per-turn, is showing judgement no certification captures. If you have built one, the write-up is worth as much as the system.

Reference material: the Model Context Protocol documentation, provider documentation on tool use and prompt caching, and the research line running through Toolformer, Gorilla and ToolLLM. Product behaviour described is as of August 2026 and moves quickly; verify against current documentation before relying on it.