Bounding Agent Tool Output Without Wrecking Your Cache

What the measurement actually showed

• Token reduction and cost reduction came apart. Across 2,908 paired executions, an aggressive compression arm removed 38.4 percent of estimated raw tool-output tokens and raised billed cost by 6.8 percent.
• You are optimising a rounding error. Prompt-cache traffic accounted for roughly 80 percent of actual cost. The compressed tool outputs were about 3.3 percent of total cost.
• The mechanism is extra turns. Aggressive compression triggered additional model turns that re-transmitted the entire cached prefix, so the local saving was offset and then some.
• Destroying verbatim anchors breaks the task. On 40 SWE-bench-derived Go rows, patch application fell from 27/40 raw to 15/40 compressed, because compression corrupted the exact strings the agent had to match on.
• Conservative bounding still pays. A deterministic hook-based arm removed only 1.3 percent of tokens and reduced cost by 2.7 percent. Small, safe, and the right sign.
• Measure cost per successful task. Tokens per turn is the metric that produced the wrong answer. Provider-billed cost divided by successful outcomes is the metric that produced the right one.

Every engineer who has watched an agent paste a 40,000-token test log into its own context has had the same thought: bound that. Cap the tool result, summarise the stack trace, drop the columns nobody reads. It is the most obvious cost optimisation in the entire agent stack, it takes an afternoon to implement, and there is a growing shelf of middleware that will do it for you.

As of 2026, we now have a careful measurement of what that actually does to the bill, and the answer is uncomfortable. The paper is Token Reduction Is Not Cost Reduction: An Empirical Study of End-to-End Efficiency in API-Based Coding Agents, arXiv 2607.12161, published in July 2026. Its authoritative campaign ran 2,908 paired executions — identical tasks, fresh working copies, randomised ordering — and recorded provider-billed cost, token counts, task success and trajectory length for each one. The headline finding is that the most effective token reducer in the study was also the most expensive to run.

This guide takes that finding seriously and turns it into a discipline. It is not another explainer on prompt caching, and it is not a rehash of compaction strategy — our guides to programmatic tool calling and cache breakpoints and to compaction for long-running agents already cover those mechanics. What follows is the narrower, sharper thing: why naive tool-output compression raises billed cost, and what a cache-aware bounding discipline looks like once you accept that.

Why the intuition fails: the billing arithmetic

The intuition is that tokens are the unit of cost, so removing tokens removes cost. That is true for a single API call and false for an agent, because an agent is not a single call. It is a loop in which almost everything is charged repeatedly and only the newest thing is charged once.

Consider what actually gets transmitted on each turn of a coding agent. The system prompt goes up. The tool schemas go up. The repository map, the instructions file, the accumulated transcript of every previous message and every previous tool result — all of it goes up, on every turn, for the agent to condition on. At the discounted cache-read rate this is cheap per token, but it is paid over and over. The 40,000-token test log, by contrast, is expensive per token exactly once, on the turn it arrives, and thereafter it is just more cached prefix.

That asymmetry is what the study measured. Prompt-cache traffic dominated billing at roughly 80 percent of actual cost. The tool outputs eligible for compression were about 3.3 percent of total cost. When a compression layer removes 38.4 percent of the tokens in a 3.3 percent slice, the ceiling on what it can possibly save is small. When the same layer removes something the agent needed and provokes one more pass around the loop, the additional pass re-transmits the prefix — the 80 percent — and the arithmetic reverses.

A worked block: one extra turn against the saving

The block below normalises a whole task to a 100-unit bill and divides out the two share figures the paper reports. It contains no new measurements — every input is either from the study or an assumption stated on the line where it appears. The simplification is that the cached prefix is treated as re-transmitted once per turn at a uniform average size, when in reality it grows as the trajectory lengthens. That simplification makes the estimate conservative, not generous.

Read the last line again, because it is the whole article in one number. Under these shares, a single additional model turn wipes out the entire compression saving unless the trajectory was already longer than about sixty-three turns. Most coding tasks are nowhere near that. A compression layer that provokes one extra turn on even a modest fraction of tasks is losing money in aggregate, and it is losing it invisibly, because the token dashboard is showing a beautiful 38 percent reduction the whole time.

The measured result was worse than this sketch — plus 6.8 percent rather than plus 2.7 — which is consistent with aggressive compression provoking more than one extra turn on some tasks, or provoking them at points in the trajectory where the prefix is already large. The direction and the order of magnitude match. Treat the arithmetic as the mechanism and the measurement as the ground truth.

Four arms, two of which made things worse

The campaign evaluated four arms, all against Claude Code specifically — not SWE-agent, not OpenHands, and that specificity matters when you generalise. The baseline ran the agent unmodified. RTK was an open-source hook-based deterministic compressor. RTK-ML added flag-gated machine-learning-enhanced variants on top. Headroom was a closed-box proxy operating at the API boundary.

Three things are worth pulling out of that table. First, the arm that removed almost nothing — 1.3 percent of tokens — was the only one that reduced cost, and it did so by 2.7 percent with an interval that only just excludes zero. Second, the relationship between token reduction and cost reduction is not merely weak in this data; across these arms it is inverted. The authors' phrasing is that token reduction and cost reduction decouple. Third, the closed-box proxy at plus 48.4 percent is a warning about a whole category of product: something that sits at the API boundary and rewrites traffic has no visibility into whether its rewriting caused a retry, so it cannot know it is losing.

Measure the right thing: cost per successful task

The metric that produced the wrong answer is tokens per turn. The metric that produces the right answer is billed cost per successful task, and the two words doing the work are billed and successful.

Billed, because estimated tokens and charged tokens are different quantities. A local tokeniser cannot tell you how much of your input hit the cache, how much was written to the cache at the higher write rate, or how the provider bucketed a request. Every serious provider returns usage fields on the response; those are the numbers to record. Anything you compute yourself from a tokeniser is a model of the bill, and this entire finding is a story about a model of the bill diverging from the bill.

Successful, because a cheaper run that fails is not cheaper. This is where the study's SWE-bench-derived evidence lands hardest. On 40 Go rows, patch application fell from 27/40 raw to 15/40 compressed — from roughly two-thirds of patches applying cleanly to well under half. Task resolution was 2/40 raw against 1/40 compressed. Be honest about what those last two figures can and cannot support: on a 40-row slice, two versus one is a difference you should not lean on. The patch-application collapse is the real signal, and its stated cause is precise — compression corrupted verbatim edit anchors.

Cost per successful execution captures both concerns in a single ratio. Normalised to a baseline of 1.000, RTK-ML came in at 1.051 and Headroom at 1.464. Our guide to LLM unit economics and cost per task works through how to build that denominator properly when success is graded rather than binary; the short version is that you need a definition of success your finance model and your eval harness both accept.

What to instrument

You need four quantities per tool call and three per run. Per call: bytes returned to the agent, bytes before bounding, the provider-billed cost delta attributable to the turn that consumed it, and the outcome. Per run: total billed cost, trajectory length in model turns, and task success. That is enough to answer the only question that matters — did bounding change the number of turns — and it is little enough to add in an afternoon.

# toolmeter.py — record what a tool call actually cost, not what it looked like.
#
# The one rule: billed_cost_delta comes from the provider's usage block on the
# response. Never from a local tokeniser. The gap between those two numbers is
# the entire subject of this guide.

import json
import time
import uuid
from dataclasses import dataclass, asdict, field

PRICE = load_price_table()   # per-MTok rates, from config, per model and region


@dataclass
class TurnCost:
    """Provider-reported usage for one model turn, priced."""
    input_tokens: int
    output_tokens: int
    cache_read_input_tokens: int
    cache_creation_input_tokens: int

    def billed(self, model: str) -> float:
        p = PRICE[model]
        return (
            self.input_tokens                * p["input"]
            + self.output_tokens             * p["output"]
            + self.cache_read_input_tokens   * p["cache_read"]
            + self.cache_creation_input_tokens * p["cache_write"]
        ) / 1_000_000


@dataclass
class ToolRecord:
    run_id: str
    task_id: str
    arm: str                 # "baseline" | "bounded" — paired, same task_id
    tool_name: str
    turn_index: int          # which model turn issued this call
    bytes_raw: int           # size the tool produced
    bytes_returned: int      # size the agent actually received
    billed_cost_delta: float # cost of the turn that consumed this result
    outcome: str             # "ok" | "tool_error" | "retried" | "abandoned"
    anchors_preserved: bool  # did bounding keep every verbatim anchor?
    latency_ms: int


def record_turn(response, model: str) -> TurnCost:
    """Pull usage straight off the API response. Field names are Anthropic's;
    OpenAI and Gemini expose equivalents under different keys."""
    u = response.usage
    return TurnCost(
        input_tokens=u.input_tokens,
        output_tokens=u.output_tokens,
        cache_read_input_tokens=getattr(u, "cache_read_input_tokens", 0),
        cache_creation_input_tokens=getattr(u, "cache_creation_input_tokens", 0),
    )


def instrumented_call(tool, args, ctx, bounder=None):
    started = time.monotonic()
    raw = tool(**args)
    bounded, anchors_ok = (bounder(raw) if bounder else (raw, True))

    ctx.pending.append(ToolRecord(
        run_id=ctx.run_id,
        task_id=ctx.task_id,
        arm=ctx.arm,
        tool_name=tool.__name__,
        turn_index=ctx.turn_index,
        bytes_raw=len(raw.encode()),
        bytes_returned=len(bounded.encode()),
        billed_cost_delta=0.0,        # filled in after the next model turn
        outcome="ok",
        anchors_preserved=anchors_ok,
        latency_ms=int((time.monotonic() - started) * 1000),
    ))
    return bounded


# After each model turn, attribute that turn's billed cost to the tool results
# it consumed. Attribution is approximate; the run-level total is not, and the
# run-level total is what the decision rests on.
def settle(ctx, response, model):
    cost = record_turn(response, model).billed(model)
    if ctx.pending:
        share = cost / len(ctx.pending)
        for rec in ctx.pending:
            rec.billed_cost_delta = share
            ctx.sink.write(json.dumps(asdict(rec)) + "\n")
        ctx.pending.clear()
    ctx.run_cost += cost
    ctx.turn_index += 1

The anchors_preserved flag is the field most teams leave out and most need. It lets you split your failures into two populations — tasks that failed with anchors intact, which is ordinary agent variance, and tasks that failed after bounding touched an anchor, which is your bug. Without it you will spend a fortnight arguing about whether the compressor is responsible. With it, the answer arrives in a single query.

Trajectory length deserves a first-class chart rather than a column in a table. It is the causal variable. If bounding is going to cost you money, the mechanism is turns, so a histogram of turns-per-task for the two arms will show the damage before the cost aggregate does, and it will show it with less noise. Our guide to evaluating agents on trajectory, tool calls and outcome covers the harness side of collecting that cleanly.

Safe bounding versus destructive bounding

The paper's SWE-bench result names the failure mechanism precisely enough to build a rule on: compression corrupted verbatim edit anchors. An agent editing a file has to reproduce exact strings — the line it is replacing, the surrounding context a patch tool matches on, the offsets in a hunk header. Paraphrase any of that and the edit does not apply. The agent then has to re-read the file, which is another tool call, which is another turn, which is the 80 percent again.

So the distinction is not between more and less compression. It is between bulk the agent will never reference and anchors the agent must reproduce byte for byte. Dropping the first is close to free. Touching the second is how you end up at 15/40.

Implemented as a rule rather than a model, this is a short function. Notice that the version below is deliberately deterministic. Determinism is not a stylistic preference here; it is what lets you assert, in a test, that a given input produces a given bounded output with every anchor intact. A learned summariser cannot give you that assertion, which is precisely why the ML-enhanced arm was the one that broke things.

# bound.py — deterministic, anchor-preserving bounding for tool output.
#
# Design rules, in priority order:
#   1. Never alter a line matching an anchor pattern.
#   2. Never alter anything inside a diff region.
#   3. Remove only classes of content on the safe list.
#   4. If still over budget, keep head and tail verbatim, elide the middle,
#      and say so explicitly with a cursor the agent can follow.

import re

ANSI       = re.compile(r"\x1b\[[0-9;?]*[a-zA-Z]")
PROGRESS   = re.compile(r"^.*\r(?=.)", re.MULTILINE)   # overwritten spinner rows
DIFF_START = re.compile(r"^(diff --git |@@ |--- |\+\+\+ )")
TIMESTAMP  = re.compile(r"^\[?\d{4}-\d{2}-\d{2}[T ]\d{2}:\d{2}:\d{2}[^\]]*\]?\s*")

# Lines the agent may need to reproduce byte for byte. Conservative by design:
# a false positive here costs a few tokens, a false negative costs a turn.
ANCHOR = re.compile(
    r"(error|Error|ERROR|panic:|assert|Assertion|FAIL|Traceback|"
    r"undefined|cannot find|expected .* got |[0-9a-f]{7,40}|"
    r"^\s*at .*:\d+|:\d+:\d+:)"
)


def bound(text: str, budget_bytes: int) -> tuple:
    """Return (bounded_text, anchors_preserved). Never raises."""
    lines = text.splitlines()
    kept, in_diff, dropped, seen_frames = [], False, 0, {}

    for line in lines:
        if DIFF_START.match(line):
            in_diff = True
        if in_diff:
            kept.append(line)                     # rule 2: diffs pass through
            continue

        if ANCHOR.search(line):
            frame = line.strip()
            seen_frames[frame] = seen_frames.get(frame, 0) + 1
            if seen_frames[frame] == 1:
                kept.append(line)                 # rule 1: first occurrence verbatim
            else:
                dropped += 1                      # rule 3: fold the repeats
            continue

        cleaned = TIMESTAMP.sub("", ANSI.sub("", line)).rstrip()
        if not cleaned:
            dropped += 1
            continue
        kept.append(cleaned)

    for frame, n in seen_frames.items():
        if n > 1:
            kept.append(f"[{n - 1} further identical occurrences folded]")

    out = PROGRESS.sub("", "\n".join(kept))
    if len(out.encode()) <= budget_bytes:
        return out, True

    # Rule 4: still too large. Head and tail verbatim, explicit elision.
    head = out[: budget_bytes // 2]
    tail = out[-(budget_bytes // 2) :]
    marker = (
        f"\n\n[... {len(out.encode()) - budget_bytes} bytes elided from the "
        f"middle. Full output available via read_tool_output(cursor=..., "
        f"offset={budget_bytes // 2}) ...]\n\n"
    )
    return head + marker + tail, True

Two details in that function matter more than they look. The elision marker names its own remedy, so the agent is never left guessing whether more exists — it is told, in the same breath, how to fetch it. And the anchor regex is deliberately over-inclusive. It will match lines that were not really anchors and keep them. That trade is correct: a false positive costs you a handful of tokens out of the 3.3 percent slice, and a false negative costs you a turn out of the 80 percent slice. The asymmetry is roughly sixty to one and it should shape every threshold you set.

Paginate, do not truncate

Truncation is the default because it is one line of code. It is also the single worst bounding strategy available, for a reason that has nothing to do with how many tokens it removes: it is silent. The agent receives forty search results, has no way of knowing there were three hundred and forty, and reasons confidently from a premise that is wrong. If the answer was in result 87, the agent does not retry — it concludes, incorrectly, and you pay for a wrong answer at full price.

Pagination costs the same tokens and removes the silence. A page-one payload that states the total, states what it is showing, and carries a cursor gives the model a decision it is well equipped to make: is the remainder worth a turn? In practice the answer is usually no, which is why teams who switch from truncation to pagination frequently see trajectory length fall rather than rise. The turns you were spending on discovery-by-failure disappear.

# A tool result shape that bounds without lying.
#
# Every field here exists to answer a question the agent would otherwise
# have to spend a turn discovering.

{
  "tool": "search_code",
  "query": "cache_read_input_tokens",
  "total_matches": 340,          # the truth, always stated
  "returned": 40,               # what is in this payload
  "truncated_fields": [],       # names of any per-row fields projected away
  "next_cursor": "eyJvIjo0MCwicSI6ImNhY2hlX3JlYWQifQ==",
  "cursor_expires_in_s": 900,
  "hint": "call search_code(cursor=next_cursor) for matches 41-80",
  "matches": [
    {
      "path": "agent/loop.py",
      "line": 412,                                  # never rounded
      "text": "u.cache_read_input_tokens,",        # verbatim, never reflowed
      "context_before": ["    return TurnCost("],
      "context_after":  ["    u.cache_creation_input_tokens,"]
    }
  ]
}

# The equivalent database-query shape. Same principles: state the total,
# name what was projected away, hand back a stable cursor.
{
  "tool": "run_query",
  "row_count": 18420,
  "returned": 50,
  "columns_returned": ["order_id", "status", "created_at", "total_gbp"],
  "columns_omitted": ["raw_payload", "embedding", "internal_notes"],
  "next_cursor": "b3JkZXJfaWQ6ODgxMjM=",
  "rows": [ ... ]
}

Three properties make a cursor worth having. It must be stable, so that the same cursor returns the same page even if the underlying data has moved on — otherwise a retry becomes a source of corruption rather than a recovery. It must be opaque, so the agent does not try to construct one arithmetically and land on a page that does not exist. And it must expire, so your tool layer is not obliged to hold result sets indefinitely for agents that abandoned the thread ten minutes ago. Our guide to designing tools for AI agents goes further into the schema and error-contract side of this; the bounding-specific rule is simply that the shape must make withholding visible.

Cache-aware placement: where bounded content lives

This is where bounding stops being a text problem and becomes a message-sequence problem, and it is where most implementations quietly lose the money they thought they were saving.

Prompt caching works on prefixes. Everything up to a cache breakpoint is matched byte for byte; the moment one byte changes, that match fails and everything from the change onward is re-processed at full input rates. The mechanics are covered in our guide to programmatic tool calling and cache breakpoints, so the point here is narrower: bounding is an edit, and where you place an edit determines what it costs.

Bounding content that sits before a breakpoint invalidates the prefix. If you rewrite a tool result from turn three while the agent is on turn twelve, you have not saved the tokens in that result — you have invalidated nine turns of cached prefix and will pay full input price to rebuild it. Against a saving drawn from the 3.3 percent slice, that is a catastrophic trade, and it is a plausible contributor to what the study measured, though the paper attributes the effect to additional model turns rather than to invalidation specifically. Either way the remedy is the same.

The rule, stated as a rule you can enforce in code review:

• Before the last breakpoint, everything is append-only and byte-stable. System prompt, tool schemas, repository map, durable instructions. These are never bounded, never reordered, never rewritten. If they are too large, fix them at design time, not at run time.
• Bounding happens once, at the moment a tool result enters the transcript. That is the only point at which the content is at the tail and no prefix depends on it. Bound it there or do not bound it.
• Never retroactively rewrite an earlier tool result in place. If an old result genuinely must go, do not edit history — append a note at the tail saying it is superseded, or start a fresh context with a handover summary. Both cost less than invalidating the prefix.
• Keep volatile content out of the prefix entirely. Timestamps, request IDs and rotating session values in a system prompt will bust your cache every single turn regardless of any bounding you do. This is the cheapest bug to fix and the most common one to have.

A decision table by tool type

The taxonomy above resolves, for the six tool types most agents actually have, into concrete recommendations. As of 2026 these are the defaults we would start a project with and then tune against measurement, not the other way round.

The pattern across the rows is worth naming. Bounding at the tool boundary — projecting fields, capping rows, structuring output before the model sees it — is safe, because it happens once and never touches cached history. Bounding in the transcript, after content has already entered the conversation, is where the money goes wrong. Push every bound you can as far upstream as it will go.

How to run this experiment on your own agent

None of the above should be adopted on faith, including the parts drawn from the paper. This is one study, on one agent harness, with one provider's billing model. The mechanism — cache traffic dominates, extra turns re-transmit the prefix, corrupted anchors cause retries — generalises considerably further than the exact percentages do. The percentages themselves belong to Claude Code, to those four arms and to that billing structure, and you should not quote them as though they describe your stack.

Fortunately the methodology is reproducible in a week. Five requirements:

• Paired runs. Every task runs under both arms. Never compare a bounded population against a baseline population drawn from different tasks; task difficulty varies far more than the effect you are measuring.
• Fresh working copies. A repository left dirty by a previous run leaks state into the next one, and the leak is systematically correlated with the arm that ran first.
• Randomised ordering. Provider-side conditions drift over hours — load, routing, model version rollouts. Randomising which arm runs first per pair converts that drift from a bias into noise.
• Billed cost, from the provider. Record usage fields per turn, price them from a table you control, and reconcile the monthly total against the actual invoice at least once. If your reconciliation is out by more than a percent or two, fix that before trusting any experiment.
• Success recorded alongside cost, always. A cost number without a success number is not a result. So is trajectory length: it is the mechanism, so it belongs in every report.

On sample size, the honest answer is that agents are noisy enough to punish small experiments severely. The published campaign ran 2,908 paired executions, and even at that scale the interval on the headline effect ran from plus 2.8 to plus 11.3 percent — 8.5 percentage points wide. The conservative arm's interval, minus 5.6 to minus 0.1, is 5.5 points wide and only just clears zero. If a well-resourced study needs thousands of pairs to bound an effect that loosely, a twenty-run comparison on your laptop is measuring its own variance. Run the largest paired sample your budget allows, report the interval rather than the point estimate, and be willing to conclude that you cannot tell — which is a legitimate and common result.

India and the UK: the same mechanism, different arithmetic

The mechanism is provider-side and travels intact, but two regional factors shift where the break-even sits, and both are worth a paragraph.

The first is wall-clock. An extra model turn is not only an extra billing event; it is an extra round trip. A team serving from ap-south-1 in Mumbai or Hyderabad and a team serving from eu-west-2 in London are hitting different provider endpoints with different path lengths, and for an interactive agent that a developer is watching, the turn count sets the perceived latency far more than the payload size does. That reframes the trade: bounding that adds turns is a product regression as well as a cost regression, and it is felt hardest by whichever team sits furthest from the endpoint.

The second is commercial. Committed-use agreements, provisioned throughput and enterprise pricing all change the marginal price of a cache read, and since cache reads are the large term in this arithmetic, they change the break-even trajectory length directly. A team on heavily discounted committed capacity has less to gain from bounding than a team on list pricing, because the 80 percent slice is cheaper for them per token. As of 2026, the practical instruction is to recompute the worked block above using your own contracted rates before deciding anything — the shape of the argument holds, the crossover point moves. Scheduling changes it too, in the same direction: our guide to off-peak scheduling and time-of-day pricing covers moving batch agent work into windows where the same tokens cost less, which is a larger lever than bounding for any workload that can tolerate delay.

What to do on Monday

The instruction that follows from all of this is short, and it is not "do not bound tool output". Unbounded output has real costs — context exhaustion, attention dilution, latency — and the conservative arm in the study did save money. The instruction is to bound with the arithmetic in view.

Start by instrumenting billed cost, trajectory length and task success before you change anything, because without a baseline you cannot detect a regression and you will be arguing from token charts within a fortnight. Then push every bound you can to the tool boundary, where it happens once and touches nothing cached: project your API responses, cap your query rows, structure your test output. Replace every silent truncation with pagination and a stable cursor. Write the anchor rules as a regression suite before you write the bounder, and let the suite fail loudly. Keep bounded content strictly after your last cache breakpoint, and never rewrite a tool result that is already in the transcript. Then run the paired experiment on your own harness and let the number decide.

What survives whichever tools you are running in eighteen months is the reframing. In an agent loop the expensive thing is not the payload; it is the pass. Anything that adds a pass over the cached prefix is expensive, and anything that removes one is valuable, and tool-output size is only interesting insofar as it changes the number of passes. Once you are optimising passes rather than tokens, the decisions in this guide follow on their own — and the token dashboard stops being the thing that misleads you.

Sources

• Token Reduction Is Not Cost Reduction: An Empirical Study of End-to-End Efficiency in API-Based Coding Agents — arXiv 2607.12161 (July 2026)