Fine-tuning for tool-call accuracy: the five failure modes, and which ones an adapter can actually fix

Why "the agent called the wrong tool" is a useless bug report

Every team that ships an agent eventually files the same ticket. Something went wrong in production, someone pulled the trace, and the summary line reads: the model called the wrong tool. It is the sort of statement that feels like a diagnosis and functions like a shrug. It tells you nothing about what to change, which is why the next thing that usually happens is a proposal to fine-tune — because fine-tuning is the largest, most expensive-looking lever in the drawer, and reaching for it feels proportionate to the frustration.

The problem is that "called the wrong tool" is a category that quietly contains at least five unrelated engineering failures. A model that picks search_orders when it needed search_shipments has failed at retrieval over its own catalogue. A model that picks the right tool but passes a date parsed as the wrong century has failed at extraction. A model that emits a call your parser rejects has failed at format. A model that calls a tool when the answer was already sitting in the conversation has failed at restraint. And a model that issues three correct calls in an order that makes the second one operate on stale state has failed at planning. Those five failures share a symptom and share nothing else. They have different detection methods, different root causes and — this is the part that costs money — different fixes.

Here is the compressed version of everything that follows. Measure per class rather than in aggregate, because one accuracy number will not tell you which lever to pull. Then work the cheap fixes in order: tool descriptions, catalogue size, retrieval over tools, few-shot selection, constrained decoding. Most teams reach an acceptable place before they open a training script. Fine-tuning becomes the right answer under a narrow, identifiable set of conditions — a large stable catalogue, house-specific argument conventions, a small model you want to run cheaply in your own region, or a latency budget that rules out a bigger one — and even then the dataset, not the hyperparameters, is where the work lives.

One framing to carry through the article: fine-tuning is good at tone, structured output, domain vocabulary and tool-call format, retrieval is right for fresh knowledge, and anything you train in is frozen at training time. Tool calling straddles that line awkwardly, because a tool catalogue is simultaneously a format problem, which an adapter handles well, and a knowledge problem, which it handles badly because your catalogue will change next quarter. Knowing which half of your failure is which is the entire skill.

Decompose the failure: five classes, five different fixes

Start by refusing to accept an undifferentiated error count. Every failed tool call in your logs belongs to exactly one of the classes below, and the assignment is nearly always mechanical once you have the gold call to compare against. The table is the spine of this article; the rest is elaboration.

(a) Selection: right shape, wrong destination

Selection errors scale with catalogue size and with semantic overlap between tools. Two tools whose descriptions could plausibly both apply to the same user sentence will be confused, and no amount of instruction-following will reliably separate them if the underlying descriptions are ambiguous. The tell is that the same pairs of tools get confused repeatedly — a confusion matrix over tool names, not a scalar accuracy figure, is the artefact you want here. If three tools account for most of your selection errors, you have a description problem, not a model problem.

(b) Argument extraction: right destination, wrong cargo

This is where the interesting errors live and where fine-tuning most often earns its keep. Argument errors split further into three flavours worth counting separately: omission, where a required field is missing; fabrication, where the model fills a field with something plausible that the user never said; and transformation, where the model had the right information and converted it wrongly — a date rendered in the wrong format, an amount in the wrong currency unit, an identifier stripped of its prefix. Fabrication is the most dangerous of the three because the resulting call is entirely well-formed and will execute happily against your production systems.

(c) Schema violation: the class you should never be tuning for

Constrained decoding masks invalid tokens during generation so that the output conforms to a predefined structure. That is not a heuristic improvement; it is a structural guarantee for the grammar the engine actually supports. If your failure is malformed calls, this class disappears when you turn constrained decoding on, and no training run is required. That is worth being blunt about, because it is the single most common misallocation of a fine-tuning budget: teams observe broken JSON, conclude the model "does not understand the schema", and spend two weeks and a GPU bill on a problem that a decoding configuration solves outright.

The caveat matters as much as the guarantee. Constrained decoding solves class (c) and none of the others. A schema-valid call can still name the wrong tool, carry fabricated arguments, fire when it should not have fired, and sit in the wrong place in a sequence. Research on structured generation has characterised exactly this gap between correctness and format compliance: a model can produce a schema-valid call that is semantically wrong, and — in engines with incomplete grammar support — can produce a semantically sensible call that fails validation. Coverage is also not uniform across engines. JSONSchemaBench, a benchmark built from roughly ten thousand real-world JSON schemas, evaluates engines on efficiency, coverage of advanced JSON Schema features and output quality, and finds that engines differ markedly in what they support and that real-world schemas remain a genuine challenge. If you are relying on constrained decoding to guarantee class (c), verify that your engine actually covers the features your tool schemas use before you assume the class is closed. Our companion guide on getting reliable JSON out of an LLM in production goes through the engine landscape in detail.

(d) Spurious invocation and the abstention problem

Two symmetric failures hide in this class, and teams almost always measure only one. A model that calls a search tool to answer a question already answered three turns ago is burning latency and money; a model that answers from memory when it should have queried the live system is producing confidently stale output. The first is annoying, the second is a correctness incident. Because most evaluation harnesses are built from traces of successful tool use, they contain no examples where the right answer was to call nothing at all — so abstention is invisible in the metrics and untrained in the model.

(e) Sequencing and dependency errors

The last class is the one that per-call scoring is structurally incapable of catching. If your agent calls get_customer, then update_address, then send_confirmation, every individual call may be perfectly correct while the trajectory is wrong because the confirmation used an address fetched before the update. You only see this by comparing whole trajectories, and you only fix it by changing the planner, the tool boundaries, or how state flows between calls — which is a design problem rather than a weights problem. Our guide to designing tools for AI agents covers the schema and error-handling side of that design work; this article deliberately does not repeat it.

Build the eval first, or you are guessing

You cannot allocate effort across five classes without a harness that reports five numbers. Build it before you change anything else — including before you turn on constrained decoding — so that every subsequent intervention has a measured before and after.

Scoring operates at two levels, and you need both. Call-level scoring compares a single predicted call against a single gold call and assigns it to one of the classes above; it is cheap, deterministic and catches classes (a) through (d). Trajectory-level scoring compares an ordered sequence of calls against a reference sequence and is the only way to see class (e). A harness that reports only call-level accuracy will show a healthy number for an agent that is failing consistently on ordering. The broader mechanics of trajectory scoring are covered in our guide to evaluating AI agents across trajectory, tool calls and outcome; what follows is the tool-call-specific layer that sits inside it.

For the golden set, aim at 200 to 500 calls drawn from real production traces rather than invented scenarios. Sample deliberately rather than uniformly: take the head of your traffic distribution so the common paths are represented, then over-sample the tail, because rare tools are where selection errors concentrate. Crucially, include turns where the correct behaviour was no call — target something in the region of one in five of your golden set, adjusted to match how often your real traffic should abstain.

Argument comparison needs a decision per field, made once and written down. Identifiers, enums, numeric limits and currency codes should be compared byte-for-byte: any difference is an error. Free-text arguments such as a search query or a note field cannot be exact-matched without producing a stream of false failures, so those need a semantic comparison — an LLM judge with a tight rubric, or an embedding-similarity threshold you have calibrated against hand-labelled pairs. Classify each argument once, into an exact set or a semantic set, and keep the classification in version control next to the schema.

from dataclasses import dataclass
from itertools import zip_longest

@dataclass
class Call:
    tool: str
    args: dict

# Decide this ONCE per tool, keep it in version control next to the schema.
EXACT_ARGS    = {"order_id", "currency", "limit", "status", "region"}
SEMANTIC_ARGS = {"query", "note", "description"}

def score_call(pred, gold, judge):
    """Classify one (predicted, gold) pair into exactly one failure class.

    pred / gold are Call or None. None means 'no tool call was made'.
    Schema violations are handled upstream: if the raw output failed to
    parse or validate, record 'schema_violation' and never reach here.
    """
    if gold is None and pred is None:
        return {"class": "correct_abstention"}
    if gold is None and pred is not None:
        return {"class": "spurious_invocation", "got": pred.tool}
    if pred is None:
        return {"class": "missed_invocation", "expected": gold.tool}

    if pred.tool != gold.tool:
        return {"class": "selection_error",
                "expected": gold.tool, "got": pred.tool}

    missing = set(gold.args) - set(pred.args)
    if missing:
        return {"class": "argument_error", "mode": "omission",
                "fields": sorted(missing)}

    extra = set(pred.args) - set(gold.args)
    if extra:
        return {"class": "argument_error", "mode": "fabrication",
                "fields": sorted(extra)}

    for key, want in gold.args.items():
        got = pred.args[key]
        if key in EXACT_ARGS and got != want:
            return {"class": "argument_error", "mode": "transformation",
                    "field": key}
        if key in SEMANTIC_ARGS and not judge(key, got, want):
            return {"class": "argument_error", "mode": "semantic",
                    "field": key}

    return {"class": "correct"}


def score_trajectory(pred_calls, gold_calls, judge):
    """Per-call classes PLUS the ordering check that per-call scoring misses."""
    per_call = [
        score_call(p, g, judge)
        for p, g in zip_longest(pred_calls, gold_calls)
    ]
    pred_order = [c.tool for c in pred_calls if c]
    gold_order = [c.tool for c in gold_calls if c]

    # Same multiset of tools, different order => sequencing error.
    sequencing = (sorted(pred_order) == sorted(gold_order)
                  and pred_order != gold_order)

    return {
        "calls": per_call,
        "sequencing_error": sequencing,
        "trajectory_exact": all(c["class"] == "correct" for c in per_call)
                            and not sequencing,
    }

Exhaust the cheap fixes — most teams stop here successfully

With per-class numbers in hand, work the ladder in cost order. Each rung is measured in hours rather than GPU days, and each one addresses a specific subset of classes.

Two rungs deserve elaboration. Catalogue partitioning is underrated because it looks like architecture work rather than model work, but selection accuracy degrades with the number of plausible candidates in a way no prompt fixes. If you expose several dozen tools in one turn, routing to a narrower sub-agent will usually beat any amount of description tuning. Retrieval over tools is the same idea made dynamic: index your descriptions, retrieve the handful relevant to the turn, present only those. Adding a tool then requires no retraining — precisely the property a fine-tuned catalogue gives up.

When tuning is genuinely the answer

The standard guidance holds here as it does elsewhere: start with retrieval plus prompt engineering, and fine-tune only when you have a stable task with a known output schema, a real evaluation harness showing the base model has plateaued, and 500 or more high-quality examples. Tool calling adds a specific condition on top — the classes remaining after the cheap fixes must be ones an adapter can move. Four situations pass that test.

The self-hosting row carries more weight in 2026 than the raw accuracy argument does. Teams running under India's data-protection regime or under UK and EU data-residency expectations frequently cannot send customer records to a third-party inference endpoint at all, which turns the question from "which model is best" into "which model can we run in the Mumbai or London region, and how do we make it good enough at our tools". A small open-weight base with a tool-call adapter is a perfectly reasonable answer to that question, and it is one of the few cases where fine-tuning is driven by a constraint rather than by a benchmark. The usual bases as of mid-2026 are Llama 3, Qwen 3, Gemma 4 and Mistral. Inkling, released by Thinking Machines Lab on 15 July 2026 under an Apache 2.0 licence with 975B total parameters and 41B active and native text, image and audio reasoning, is positioned by its makers as a strong base to fine-tune — worth evaluating for genuinely multimodal tool calling, though its scale sits in a different serving bracket from the small-model case above.

If you are still unsure whether tuning is warranted at all, the general version of this decision lives in our fine-tuning decision ladder. This article assumes you have already climbed it and landed on "probably yes, for tool calls specifically".

The dataset is the whole job

Once you decide to tune, the hyperparameters take an afternoon and the dataset takes three weeks. Typical production jobs run on 500 to 2,000 hand-curated examples in ChatML format, and for tool calling the composition of those examples matters more than the count.

Mine them from production traces rather than writing them. Your traces contain the real distribution of phrasings, the real proportion of ambiguous turns and the real identifier formats — none of which you will reproduce by hand. Pull turns where the tool call succeeded and the downstream outcome was good, then have a human confirm the call was actually the right one rather than merely non-failing. A call that executed cleanly and returned the wrong thing is a poisoned example.

Then do the step almost everyone skips. A dataset built only from successful calls teaches the model that the correct response to any input is a tool call, because that is the only behaviour it ever observes. You need three kinds of negative example woven through the set:

• Abstention examples — the user asked something answerable from the conversation, or from general knowledge, and the correct assistant turn contains no call at all. Target roughly a fifth of the set, tuned to your real traffic.
• Clarification examples — the request was ambiguous and the correct behaviour was to ask rather than to guess an argument. This is the direct antidote to fabrication.
• Hard negatives — turns where a confusable sibling tool was tempting and the gold call is the other one. Mine these from your class (a) confusion matrix; they are the highest-value examples in the whole dataset.

Deduplicate aggressively before you train. Production traces are heavily skewed towards a handful of high-frequency intents, and an undeduplicated set will contain the same three phrasings hundreds of times over. Cluster by embedding similarity, cap the number of near-duplicates per cluster, and keep the rare-tool examples intact even when that means an unrepresentative distribution — you are training a behaviour, not modelling your traffic. Where a genuine gap remains that logs cannot fill, generated examples are legitimate, but filter them hard rather than asking a model for two thousand variations and shipping the lot.

// tool_calls.jsonl — one JSON object per line, ChatML with tool calls.
// Keep the system prompt byte-identical to what you serve in production.

// 1. POSITIVE — correct tool, house-format identifier, no invented fields
{"messages": [
  {"role": "system", "content": "You may call at most one tool per turn. If the answer is already available in the conversation, answer directly and call nothing."},
  {"role": "user", "content": "Refund order GB-88431, the parcel never turned up."},
  {"role": "assistant", "content": null, "tool_calls": [
    {"id": "c1", "type": "function", "function": {
      "name": "create_refund",
      "arguments": "{\"order_id\": \"GB-88431\", \"reason_code\": \"NOT_RECEIVED\"}"
    }}
  ]}
]}

// 2. ABSTENTION — answerable from context, so no call at all
{"messages": [
  {"role": "system", "content": "You may call at most one tool per turn. If the answer is already available in the conversation, answer directly and call nothing."},
  {"role": "user", "content": "What does reason_code NOT_RECEIVED actually mean?"},
  {"role": "assistant", "content": "It marks a refund raised because the parcel was never delivered, as distinct from RETURNED, which is used once the goods are back with us."}
]}

// 3. CLARIFICATION — ambiguous, so ask rather than fabricate an argument
{"messages": [
  {"role": "system", "content": "You may call at most one tool per turn. If the answer is already available in the conversation, answer directly and call nothing."},
  {"role": "user", "content": "Refund the Chennai order."},
  {"role": "assistant", "content": "I can see two open orders shipping to Chennai — IN-40218 and IN-40233. Which one should I refund?"}
]}

// 4. HARD NEGATIVE — search_orders is tempting; the tracking reference makes
//    search_shipments correct. Mined straight from the class (a) confusion matrix.
{"messages": [
  {"role": "system", "content": "You may call at most one tool per turn. If the answer is already available in the conversation, answer directly and call nothing."},
  {"role": "user", "content": "Where has TRK-9920-AB got to?"},
  {"role": "assistant", "content": null, "tool_calls": [
    {"id": "c1", "type": "function", "function": {
      "name": "search_shipments",
      "arguments": "{\"tracking_reference\": \"TRK-9920-AB\"}"
    }}
  ]}
]}

Run it, then gate the release honestly

The training run is the least interesting part. As of mid-2026, QLoRA — a 4-bit NF4 quantised base with an fp16 LoRA adapter trained on top — is the default approach, and rank 16 is a common starting point. The memory arithmetic is what makes this practical: a 70B model fits in roughly 48 GB of VRAM under QLoRA rather than roughly 140 GB, and quality lands within 1 to 2 per cent of full LoRA on standard benchmarks. On the tooling side, Unsloth is the usual choice on a single GPU, Axolotl for multi-GPU production runs, MLX-LoRA on a Mac, and TRL when you want raw control over the loop. Many jobs of this size complete in under an hour, which is worth internalising: the run is cheap, so the expensive mistake is always the dataset, never the compute.

from peft import LoraConfig

# Rank 16 is the common starting point. Change it only in response to
# eval-set movement, not intuition.
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)

# --- Release gate: per-class, plus a general-capability guard -------------
GENERAL_TOLERANCE = 0.02   # how much ordinary-chat quality you will accept losing

def release_gate(after, before):
    """after / before are dicts of per-class rates from the SAME held-out set."""
    checks = {
        "selection improved":
            after["selection_accuracy"]   >  before["selection_accuracy"],
        "arguments improved":
            after["argument_accuracy"]    >  before["argument_accuracy"],
        "abstention not traded away":
            after["correct_abstention"]   >= before["correct_abstention"],
        "no new spurious calls":
            after["spurious_invocation"]  <= before["spurious_invocation"],
        "general chat not degraded":
            after["general_chat"] >= before["general_chat"] - GENERAL_TOLERANCE,
    }
    failed = [name for name, ok in checks.items() if not ok]
    return (not failed), failed

ship, blockers = release_gate(adapter_scores, baseline_scores)
if not ship:
    raise SystemExit(f"Adapter blocked by: {blockers}")

Hold out three things, not one. A standard random test split tells you least. Hold out a slice of unseen tools — tools present in the catalogue but absent from training — to see whether the adapter has learned tool calling or merely memorised your catalogue. Hold out a slice of unseen phrasings for tools it has seen. And hold out a general-conversation slice that has nothing to do with tools at all, because that is your early-warning system for the regression that matters most.

That regression is the reason the gate above has five conditions rather than two. A model tuned hard on tool calls gets better at emitting tool calls and worse at everything adjacent: free-form explanation flattens, clarification questions disappear, and refusal behaviour softens. This is catastrophic forgetting operating exactly as documented, and it is well enough understood that there is no excuse for being surprised by it — the mitigations, from rank choice to data mixing, are covered in our catastrophic forgetting playbook. For the adapter mechanics themselves — rank and alpha selection, prompt-format parity, merge precision — our eval-driven LoRA and QLoRA recipe is the companion piece to this one.

The table below shows the shape of a healthy result, not measurements. There are no numbers in it deliberately: any figure printed here would be a number from someone else's system, and the entire argument of this article is that you must generate your own.

Pitfalls, and a checklist to run before you commit

The failure patterns are consistent across teams. In rough order of frequency:

1. Tuning to fix malformed JSON. Class (c) has a deterministic fix that costs a configuration change. Spending a training budget on it is the most common misallocation in this entire area.
2. No negative examples. A dataset of pure successes produces a model that calls a tool on every turn, and the harness that was also built from successes will not notice.
3. Aggregate accuracy only. One number cannot tell you which lever to pull, and it lets a class (d) regression hide behind a class (b) improvement.
4. Freezing a catalogue that moves. Anything trained in is frozen at training time. If you add tools monthly, retrieval over the catalogue ages far better than an adapter does.
5. Template drift between training and serving. The one failure mode that silently discards most of your training run.
6. Exact-matching free-text arguments. Produces a flood of false failures that swamp the real signal and erodes trust in the harness.
7. No general-capability holdout. You will find out about the regression from a user rather than from your gate.

The uncomfortable conclusion of the decomposition is that fine-tuning for tool-call accuracy is a narrower intervention than its reputation suggests. It moves argument extraction well, selection reasonably, abstention only when you have done the dataset work properly, format redundantly, and sequencing barely at all. That is a genuinely useful tool for a genuinely specific problem — and it is a very expensive way to fix broken JSON. Measure per class, exhaust the cheap rungs, and let the distribution of your own errors decide, rather than the size of the lever.