Making Small Models Reliable at Tool-Calling

What actually breaks when a small model calls a tool

Run a capable open-weight model in the 27 to 32 billion parameter range against a realistic tool suite and the transcript is rarely stupid. It reads the request, works out that a database lookup is needed rather than a calculation, and emits something that looks very much like a function call. The decision layer is fine. The failures cluster elsewhere: a date formatted as 11/08/2026 when the schema wanted 2026-08-11, a currency field carrying GBP_STERLING when the enum only accepts GBP, a nested object flattened into a string, a required field quietly omitted because the model could not infer it and did not want to say so.

This is why "it picked the right tool" is the wrong success metric. Tool-selection accuracy is the easy half of the problem and largely the half small models have solved. Argument correctness and strict schema adherence are the hard half, and they matter more: a call to the right endpoint with the wrong parameters does not error, it succeeds — returning the wrong customer, charging the wrong amount, filing to the wrong region. The system reports a green tick. The finding reported in arXiv 2510.07248 in the current literature is exactly this: tool-use accuracy depends far more critically on argument correctness and strict schema adherence than on raw parameter count, and paired with explicit tool schemas and robust validators, small models frequently match or surpass larger models on function-calling reliability and speed.

The second property that makes this hard is intermittency. A model that fails half the time is easy to reject. A model that fails four per cent of the time, on a task with twelve tool calls in it, fails the task roughly forty per cent of the time and does so non-deterministically — the worst debugging surface there is. Reliability engineering for small models is mostly about turning a stochastic per-call error rate into a bounded, observable, terminating process.

As of August 2026 the shortlist of open-weight models cited as dependable tool-callers — models that emit well-formed function-calling JSON and survive strict MCP schema validation — includes Gemma 4 27B, GLM-4.7 32B, Qwen3 32B, Qwen3-Coder 30B and Llama 3.3 70B. Meta added another option with Muse Glimmer, a 30B model released under Apache 2.0 on 10 August 2026 and built for agent workloads. Any of these can anchor a production agent. None will do it unaided.

Anatomy of a tool-call failure

You need a taxonomy first, because the eight distinct things people call "the model got the tool call wrong" have eight different detection methods and wildly different blast radii. Lumping them together is how teams spend a fortnight on JSON parsing while the expensive failure goes unmeasured.

Notice the shape of that table. Rows four to seven are formatting problems your code can solve without asking the model anything. Rows one, three and eight are semantic problems that no amount of schema tightening will touch. Most of the industry conversation about structured outputs concentrates on the rows that matter least — understandably, since they are the ones you can fix in an afternoon — which produces a systematic illusion of progress. Our guide to structured output prompting patterns in production covers that formatting layer properly; this article is about what happens once it is already in place.

Design the schema for the model you have

The most cost-effective intervention available is the one teams reach for last, because it feels like cheating: change the schema instead of the model. Recent arXiv work makes the argument directly — "Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Models" (arXiv 2510.07248) treats the schema as a design surface and finds that reshaping tool schemas to suit the model is often more effective than adapting the model to the schema. Your schema was probably written for a REST API by someone optimising for expressiveness and normalisation. The model is not a REST client. It is a next-token predictor that must serialise a nested structure left to right with no ability to backtrack.

The rules that follow are mechanical. Prefer flat objects to nested ones — every level is another closing brace the model must keep on a mental stack. Prefer enums to free text wherever the value space is finite, because an enum turns a generation problem into a selection problem. Keep required fields few and make the rest genuinely optional with sane defaults. Put units in the field name, so amount_gbp_minor carries its own contract. Avoid oneOf, anyOf and discriminated unions, which ask the model to commit to a branch before generating the evidence for it. And keep the tools visible in any single turn few — for a large catalogue, retrieve a candidate subset first, a pattern we worked through in retrieval and deferred schemas for agents with 200 tools.

{
  "name": "create_refund",
  "parameters": {
    "type": "object",
    "required": ["order", "amount", "reason", "requester", "channel"],
    "properties": {
      "order": {
        "type": "object",
        "properties": {
          "identifier": {
            "oneOf": [
              {"type": "string"},
              {"type": "object",
               "properties": {"legacy_id": {"type": "integer"}}}
            ]
          },
          "market": {"type": "object",
                     "properties": {"country": {"type": "string"}}}
        }
      },
      "amount": {"type": "object",
                 "properties": {"value": {"type": "number"},
                                "currency": {"type": "string"}}},
      "reason": {"type": "string"},
      "requester": {"type": "string"},
      "channel": {"type": "string"}
    }
  }
}

{
  "name": "create_refund",
  "parameters": {
    "type": "object",
    "required": ["order_id", "amount_minor_units"],
    "properties": {
      "order_id":          {"type": "string",
                            "pattern": "^ORD-[0-9]{8}$"},
      "amount_minor_units":{"type": "integer", "minimum": 1},
      "currency":          {"type": "string",
                            "enum": ["GBP", "INR", "EUR", "USD"],
                            "default": "GBP"},
      "reason_code":       {"type": "string",
                            "enum": ["damaged", "not_received",
                                     "wrong_item", "duplicate",
                                     "customer_changed_mind"]},
      "requested_by_email":{"type": "string", "format": "email"}
    }
  }
}

The second schema is easier to emit correctly for four specific reasons, not aesthetic ones. The oneOf is gone, so there is no branch commitment. Currency and reason are enums, so the model selects from a closed set instead of generating a string it might decorate. Amount is a single integer in minor units, which removes the whole class of decimal and rounding errors. And only two fields are required, so the most common failure — a missing required key — has two chances to occur instead of five. The pattern on order_id costs nothing at generation time and gives your validator a cheap, precise rejection rule.

The constraint tax: valid is not correct

Here is where most reliability projects go wrong. Having found that the model emits malformed structures, the obvious move is to make malformed structures impossible: constrain the decoder with a grammar or a JSON Schema so only conforming tokens can be sampled. Validity goes to 100 per cent, the parse-failure alerts stop firing, and the team declares the problem solved.

Recent arXiv work measuring exactly this trade-off — a study of validity-correctness trade-offs in structured outputs for small language models (arXiv 2605.26128), across roughly 15,000 generations — reports the following.

Read the third row again, because it is the whole argument of this article. Under hard constraints, the overwhelming majority of outputs are well-formed and wrong. The mechanism is not mysterious: constraining the decoder removes the model's escape hatch of failing visibly. An unconstrained model that does not know the answer will hedge, ramble, produce broken JSON — all of which your pipeline catches and routes to a retry or a human. Force it into a grammar and it must produce a legal value whether or not it has one, so uncertainty is laundered into confident, parseable, incorrect output. You have not reduced the error rate. You have converted loud errors into silent ones and destroyed the signal you were using to detect them.

"When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models" (arXiv 2605.02363) reinforces this from the other direction: high task accuracy does not imply high output accuracy when format requirements are strict, and while constrained decoding addresses syntax it adds latency overhead and can degrade task performance. Its more useful result for practitioners is what worked instead — an iterative black-box prompt optimiser that substantially closed the output-accuracy gap without any fine-tuning, at close to baseline inference cost. The format problem is frequently a prompt problem wearing a decoder costume.

None of this means constrained decoding is never right. It is right under two conditions held together. First, the downstream consumer genuinely cannot tolerate malformed input — a hard parse boundary, a strict MCP server, a batch job with no human in the loop. Second, and non-negotiably, you have an independent correctness check downstream of the constraint: a business-rule validator, a verifier pass, or an execution result you can assert against. Constraints without an independent check are a way of making your error rate invisible. Our guide to reliable JSON from any LLM with constrained decoding goes deeper on the implementation and latency profile; treat this section as the warning label that belongs on it.

Repair instead of constrain

The alternative that preserves the failure signal is to let the model generate freely, then validate and repair in code. The repair layer is deterministic, auditable, cheap and — critically — it logs what it fixed, turning silent failures back into measurable ones. Four rungs: syntactic repair of near-JSON, type coercion, enum nearest-match snapping with a distance threshold, then a bounded reprompt carrying the validator's own error message back to the model.

import json
import re
from difflib import get_close_matches


def call_model(prompt, *, model, tools, temperature=0.0):
    """Thin provider-agnostic shim. Returns {"name": str, "arguments": str|dict}.
    Swap the body for your SDK of choice; nothing below depends on it."""
    raise NotImplementedError


def repair_json(raw):
    """Best-effort syntactic repair of almost-JSON tool arguments."""
    if isinstance(raw, dict):
        return raw
    text = str(raw).strip()
    text = re.sub(r"^```(?:json)?\s*|\s*```$", "", text)   # strip code fences
    start, end = text.find("{"), text.rfind("}")           # drop commentary
    if start == -1 or end <= start:
        return None
    text = text[start:end + 1]
    candidates = (
        text,
        re.sub(r",(\s*[}\]])", r"\1", text),               # trailing commas
        re.sub(r",(\s*[}\]])", r"\1", text.replace("'", '"')),
    )
    for candidate in candidates:
        try:
            return json.loads(candidate)
        except json.JSONDecodeError:
            continue
    return None


PYTHON_TYPES = {"integer": int, "number": (int, float),
                "boolean": bool, "string": str}

COERCERS = {
    "integer": lambda v: int(str(v).replace(",", "").strip()),
    "number":  lambda v: float(str(v).replace(",", "").strip()),
    "boolean": lambda v: str(v).strip().lower() in {"true", "yes", "1"},
    "string":  lambda v: str(v),
}


def _type_ok(value, expected):
    if expected == "integer" and isinstance(value, bool):
        return False                    # bool is an int in Python; not here
    return isinstance(value, PYTHON_TYPES.get(expected, object))


def coerce_and_snap(args, schema, enum_cutoff=0.85):
    """Coerce scalar types and snap near-miss enum values. Returns (args, notes)."""
    out, notes = dict(args), []
    for field, spec in schema.get("properties", {}).items():
        if field not in out:
            continue
        expected = spec.get("type")
        if expected in COERCERS and not _type_ok(out[field], expected):
            try:
                out[field] = COERCERS[expected](out[field])
                notes.append(f"coerced {field} to {expected}")
            except (TypeError, ValueError):
                pass                    # leave it; the validator will reject it
        allowed = spec.get("enum")
        if allowed and out[field] not in allowed:
            lowered = [str(a).lower() for a in allowed]
            near = get_close_matches(str(out[field]).lower(), lowered,
                                     n=1, cutoff=enum_cutoff)
            if near:
                snapped = allowed[lowered.index(near[0])]
                notes.append(f"snapped {field}: {out[field]!r} to {snapped!r}")
                out[field] = snapped
    return out, notes


def parse_tool_call(raw_arguments, schema, validate):
    """validate(args, schema) returns a list of error strings; empty means valid.
    Returns (args_or_None, errors, repair_notes)."""
    args = repair_json(raw_arguments)
    if args is None:
        return None, ["arguments were not parseable as JSON"], []
    args, notes = coerce_and_snap(args, schema)
    return args, validate(args, schema), notes

Three details there are load-bearing. The enum cutoff is a threshold, not a free-for-all: at 0.85 you snap "gbp" to "GBP" and "duplicated" to "duplicate", but you leave "GBP_STERLING" alone for the validator to reject, and you never silently convert "damaged" into "duplicate". Set the cutoff too low and repair becomes its own source of confidently wrong arguments. The boolean carve-out in _type_ok exists because Python treats True as an integer and you do not want true passing as a quantity. And every repair appends a note: a field needing coercion on thirty per cent of calls is a schema defect, not a model defect, and it should send you back to the previous section rather than deeper into this one.

When validation still fails, reprompt with the error text rather than "try again". Small models respond well to concrete, mechanical feedback: currency: "GBP_STERLING" is not one of ["GBP","INR","EUR","USD"] is a fixable instruction in a way that "your output was invalid" is not.

Retry ladders that terminate

A retry loop that resends an identical prompt to a temperature-zero model will reproduce an identical failure, burn your budget and eventually time out. Every rung of a ladder must change something about the input, and the ladder must have a top.

The escalation economics only work if the residue stays small, which is why the escalation rate belongs on a dashboard from day one. A ladder quietly escalating forty per cent of calls is a frontier deployment paying a small-model tax on every request.

import hashlib
import json
import time


class ToolCallUnresolved(Exception):
    """Raised when every rung of the ladder has been exhausted."""


SMALL = "gemma-4-27b"
LARGE = "frontier-model"

RUNGS = (
    {"model": SMALL, "temperature": 0.0, "feedback": False,
     "example": False, "narrow": False},
    {"model": SMALL, "temperature": 0.0, "feedback": True,
     "example": False, "narrow": False},
    {"model": SMALL, "temperature": 0.2, "feedback": True,
     "example": True,  "narrow": True},
    {"model": LARGE, "temperature": 0.0, "feedback": True,
     "example": True,  "narrow": True},
)


def idempotency_key(task_id, tool_name, args):
    """Stable across retries: identical intent yields an identical key."""
    payload = json.dumps({"task": task_id, "tool": tool_name, "args": args},
                         sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:32]


def resolve_tool_call(task_id, prompt, tools, schema, validate,
                      execute, executed):
    """executed: a durable dict-like store mapping idempotency key to result."""
    errors, trace = [], []
    for attempt, rung in enumerate(RUNGS, start=1):
        turn = build_prompt(prompt, schema,
                            errors=errors if rung["feedback"] else [],
                            example=rung["example"])
        catalogue = narrow(tools, schema) if rung["narrow"] else tools
        raw = call_model(turn, model=rung["model"], tools=catalogue,
                         temperature=rung["temperature"])
        args, errors, notes = parse_tool_call(raw["arguments"], schema, validate)
        trace.append({"attempt": attempt, "model": rung["model"],
                      "errors": errors, "repairs": notes})
        if args is not None and not errors:
            key = idempotency_key(task_id, raw["name"], args)
            if key in executed:
                return executed[key]        # a retry, not a second call
            result = execute(raw["name"], args, idempotency_key=key)
            executed[key] = result
            return result
        time.sleep(0.2 * attempt)            # short, bounded backoff
    raise ToolCallUnresolved({"task_id": task_id, "trace": trace})

The idempotency key is the part people skip and regret. It is derived from the task identifier, the tool name and the normalised arguments, so a retry triggered by a network timeout on a call that actually succeeded resolves to the stored result rather than issuing a second refund. Pass the same key to the downstream API wherever one is supported, so deduplication holds even if your own store loses the entry.

Verifier passes before anything irreversible

Repair fixes shape. Verification fixes meaning, and it is the only layer that addresses the highest-blast-radius failure in the taxonomy — the right tool called with wrong-but-plausible arguments. A verifier sits between a validated call and its execution, in two useful forms: a deterministic rule engine asserting business invariants, and a cheap second model pass asking whether these arguments actually serve the user's stated intent.

The deterministic checker comes first because it is effectively free. Does the refund amount exceed the order total? Does the requested date fall in the past? Does this account belong to the tenant in session? Is the currency consistent with the order's market? These catch the arguments a schema will never reject, and they are ordinary code — no model, no measurable latency, fully testable. A surprising share of what teams reach for a verifier model to do is expressible this way.

The model verifier earns its place only for judgements a rule cannot express: whether the reason code matches what the customer described, whether the record being modified is the one the conversation referred to, whether the call is a sensible step given the plan so far. Run it as a short separate call carrying the tool schema, the proposed arguments and the user turn, and ask for accept or reject with a one-line justification. The same 27B model is usually adequate as its own verifier, because verification is a far easier task than generation.

What makes this affordable is splitting the tool catalogue in two. Read-only tools — searches, lookups, retrievals — execute optimistically, because the cheapest verifier available is the tool itself returning an error or an empty result, and the agent recovers from that in the loop. Side-effecting tools — payments, writes, outbound messages, anything with an audit trail — verify first, always. The figures below are an illustrative worked example rather than measured results; substitute your own.

The rule generalises. If verification costs v per call and a bad call costs c to remediate — refund reversal, support contact, regulatory exposure — verification pays whenever the residual error rate times c exceeds v. For a customer-visible financial action, c is tens of pounds or thousands of rupees of handling time and v is a fraction of a penny. For a search query, c is one wasted call. The mistake is applying one policy to both.

Measure the thing that actually matters

Everything above is unfalsifiable without an evaluation harness, and the harness most teams build measures the wrong quantity. Schema validity is a plumbing metric: useful while your parser is broken, saturated and uninformative the moment constraints are switched on. Tool-selection accuracy is usually already high for a capable small model, which is precisely why it flatters the system. The two numbers that carry information are end-to-end task success and argument-level exact match, reported alongside the wrong-but-valid rate that the constraint tax inflates.

from collections import Counter
from dataclasses import dataclass
from typing import Any, Callable


@dataclass
class Case:
    case_id: str
    prompt: str
    expected_tool: str
    expected_args: dict
    normalize: Callable[[dict], dict] = lambda a: a   # e.g. date, case, units


def score(cases, run):
    """run(case) returns:
       {"tool": str, "args": dict, "valid": bool,
        "attempts": int, "cost_usd": float, "task_ok": bool}"""
    tally, total_cost = Counter(), 0.0
    for case in cases:
        r = run(case)
        tally["cases"] += 1
        tally["schema_valid"] += int(bool(r["valid"]))
        tool_ok = r["tool"] == case.expected_tool
        args_ok = case.normalize(r["args"]) == case.normalize(case.expected_args)
        tally["tool_selected"] += int(tool_ok)
        tally["args_exact"] += int(tool_ok and args_ok)
        tally["task_success"] += int(bool(r["task_ok"]))
        # The number hard constraints hide: well-formed and wrong.
        tally["wrong_but_valid"] += int(bool(r["valid"]) and not r["task_ok"])
        tally["calls"] += r["attempts"]
        total_cost += r["cost_usd"]

    n = tally["cases"] or 1
    successes = tally["task_success"] or 1     # avoid divide-by-zero on a bad run
    return {
        "schema_validity":  tally["schema_valid"] / n,
        "tool_selection":   tally["tool_selected"] / n,
        "argument_exact":   tally["args_exact"] / n,
        "task_success":     tally["task_success"] / n,
        "wrong_but_valid":  tally["wrong_but_valid"] / n,
        "calls_per_task":   tally["calls"] / n,
        "cost_per_success": total_cost / successes,
    }

The normalize hook on each case is not decoration. Argument exact match without it punishes the model for writing "GBP" where the reference says "gbp" — a comparison bug rather than a model failure, and one that will send you optimising the wrong thing for a week. Normalise case, whitespace, date format and units before comparing, and keep the rules in the case so they stay visible and reviewable.

Run this suite in continuous integration on every schema change, prompt change and model version bump, and keep a held-out slice you never tune against. The step-level view complements the sequence-level view — whether the agent took a sensible path, not just whether each call was well-formed — which we covered in evaluating AI agents by trajectory, tools and outcomes. The same fields belong on your production traces: attempt count, repair notes, validator errors and an escalation flag per call, which is exactly the data OpenTelemetry-based agent observability is built to carry. The repair-note stream is the highest-yield log in the system, because it names the schema fields that are hostile to your model before the failure rate does.

Is a small model the right choice at all?

The honest answer is sometimes, and the decision has less to do with benchmarks than with the shape of your constraints. An arXiv survey of small language models for agentic systems (arXiv 2510.03847) lays out the architectural and deployment trade-offs in more depth than a guide like this can; the practical ladder is shorter.

Choose a small open-weight model when at least one of these holds: the data cannot leave a jurisdiction or a network; latency matters more than headroom and you can serve locally; volume is high enough that per-token pricing dominates unit economics; or the task family is narrow enough that a fixed tool catalogue covers it. Choose a frontier model when the task needs long-horizon planning over an open-ended tool space, when your team cannot own an inference stack, or when the reliability layer above would cost more engineering time than the token savings are worth. That last condition disqualifies more teams than expect it, and there is no shame in it.

The India and UK cases look different

Both markets push towards local inference, for different reasons. In India, the DPDP Act does not itself mandate data localisation, but the contractual expectations that have grown up around it mean many enterprise buyers now require personal data to be processed in-country, and tool arguments are personal data — an order identifier plus an email address plus an amount is a complete record. Running Gemma 4 27B or Qwen3 32B on a GPU in an Indian region keeps the whole tool-calling loop inside the boundary, and the cost case is often favourable outright: at sustained volume, an owned or reserved GPU in a domestic region can beat per-token frontier pricing well before you have exhausted the machine. In the UK, the constraint is UK GDPR's restrictions on international transfers absent an adequacy decision or appropriate safeguards, which for regulated clients turns into a preference for inference pinned to a UK or EEA region — sometimes to the client's own infrastructure. Verify the specifics with counsel for your sector rather than taking a technical article's word for it; the engineering conclusion, though, is the same in both places.

If that is your situation, serving decisions matter as much as model choice. Our playbook on self-hosting open-weight LLMs in production with vLLM covers throughput, batching and the memory arithmetic, and building a local AI agent with Ollama, a small model and MCP tools is the shortest path from nothing to a working loop. One warning belongs here: quantisation and tool-call reliability interact badly and quietly. An aggressive quantisation costing a point or two on a general benchmark can cost considerably more on strict schema adherence, because emitting an exact structure is precisely the low-entropy, high-precision generation that quantisation error disrupts. Re-run the harness after every quantisation change, not just after model changes.

And if argument accuracy is still short of your bar on a narrow, high-volume task family with plenty of logged examples, that is the specific situation where fine-tuning is the correct next move rather than the fashionable one — the case we set out in fine-tuning for tool-call accuracy, and when prompting stops working. It is the last rung, not the first, because everything before it is cheaper, reversible and survives a base-model upgrade.

The through-line carries away on its own. Small models are close enough on capability that the remaining gap is engineering, and the engineering is unglamorous: shape the schema for the model, repair what you can, escalate what you cannot, verify anything irreversible, and measure what your users would recognise as success. The trap worth naming twice is the comfortable one — the pipeline where validity reads 100 per cent, nothing errors, and the correctness you never measured has quietly halved.