Programmatic Tool Calling and Cache Breakpoints

What you need to know

There is a particular shock that arrives a few weeks after a team ships its first genuinely useful agent. The demo was cheap; production is not. Nobody added a bigger model, yet the per-task cost has quietly multiplied. The instinct is to blame the reasoning, and the instinct is usually wrong. In a tool-heavy loop, the expensive part is everything the model has to re-read before it is allowed to think — and two things get re-read on every single turn. Your tool definitions: the full schema for every tool the agent might call, whether it calls them or not. And your tool results: every row and payload a previous tool handed back, sitting in the conversation history. One grows with your integration surface, the other with the length of the task.

During 2026 both major providers shipped API surface changes aimed at these two halves specifically. Programmatic tool calling lets the model write a short program that calls the tools, processes the intermediate results, and returns only the distilled answer — so the bulk of the data never enters the context window. Explicit cache breakpoints let you decide precisely where your reusable prefix ends, so the stable blocks at the front of your request are read from cache at a fraction of the input rate.

They are complementary rather than competing, and both are decisions about the shape of your API request rather than how you word your prompt. One boundary to set up front: this is not a prompt-caching primer — if you need the fundamentals of stable-prefix-first construction, our guide to prompt caching across Claude, GPT and Gemini covers them and this piece assumes them. Nor is it a guide to reducing your tool count, which is handled in dynamic tool retrieval for agents with 200 tools. This is about the two features and what they do to the loop.

Where the tokens actually go in a tool-heavy loop

Walk through a single turn of an agent request as the API sees it. The request carries, in order: your system prompt; the tools array containing a full schema for every tool the agent is permitted to use; and the message history, holding the user's task, every assistant turn so far, and every tool result returned so far. The model reads all of that, decides on one more action, and emits it. Your loop then appends that action's result to the history and sends the whole thing again. That is the whole cost story: the loop is not incremental, and each turn re-sends everything before it plus one more result.

The arithmetic of a six-turn task

Take an illustrative shape — round hypothetical figures for the arithmetic, not measurements. Suppose your system prompt is 1,000 tokens, your tools array holds forty definitions averaging 300 tokens each for 12,000 tokens, and the user's task is 200 tokens. Your fixed floor, before the agent has done anything, is 13,200 tokens. Now suppose each tool result averages 4,000 tokens, because real tools return real data: a list of open deals, a page of search results, a table from a warehouse.

Turn one costs 13,200 tokens of input. Turn two costs 17,200. By turn six the request carries 33,200 tokens, and the billed input across the task is the sum of all six turns, not the largest one. The fixed floor alone has been paid six times over — 79,200 tokens of definitions and instructions for a task that needed them once. The results are paid for on a triangular schedule: the first is re-read five times, the second four times, and so on.

Two properties fall out, and they determine which lever helps you. Turn count multiplies the fixed floor linearly. Result size multiplies roughly with the square of turn count, because each result is re-read on every subsequent turn. A long agentic loop over fat tool results is the worst case in the space — and, unfortunately, the exact shape of most genuinely useful business agents.

Two halves of the same bill

The two halves respond to different medicine. The definitions half is stable: your forty schemas are byte-identical on turn six and turn one, and identical across every session until you deploy. Stable content is cacheable content, so this half is a caching problem. The results half is never stable — every result is new — so you cannot cache your way out of it. The only structural fix is to stop putting the data in the window at all, which makes it an orchestration problem.

That split is why the two features here are complementary rather than alternatives. Caching does nothing for a fat tool result. Programmatic tool calling does nothing for a 12,000-token tools array re-read on every turn. Apply only one and you have fixed half a problem — which half matters more depends on your agent's shape, which is what the decision table below is for.

Programmatic tool calling: keep the results out of the window

The conventional loop treats the model as the integration layer: a tool returns data, the data goes into the window, the model reads it and decides on the next call. Every join, filter and aggregation happens inside the context window, in natural language, at input-token prices. Programmatic tool calling inverts that. The model writes a short program that calls the tools directly, does the joining and filtering in code, and returns only the distilled result. The intermediate data — the 40,000-token CSV, the paginated API response — is handled by the program and never enters the conversation.

How it appears on Claude

On Anthropic's platform this rides on the code execution sandbox. The tool version code_execution_20260120 added programmatic tool calling from within the sandbox; a later version, code_execution_20260521, discloses the per-cell time limit in the tool description, which matters if your generated programs might run long enough to be cut off.

One distinction is worth precision. Claude exposes server tools, which execute on Anthropic's infrastructure, and client tools, where Anthropic defines the schema but your application executes them. Both appear in the request's tools array alongside your own user-defined tools — the single array that gets re-sent every turn, and the one an explicit cache breakpoint can pin down.

How it appears on GPT-5.6

The GPT-5.6 family — Sol, Terra and Luna — reached general availability on 9 July 2026, and programmatic tool calling arrived with it in the Responses API. OpenAI's description is that the model writes and runs programs in-memory that coordinate tools and process intermediate results; it can write JavaScript to orchestrate tools, filter data and process outputs outside the model's context window. It is Zero Data Retention compatible, often the detail that decides whether a regulated customer can use it at all.

OpenAI frames its new Responses API primitives as targeting three areas: reusing previous work through persistent reasoning and conversation compaction, parallel decomposition through native multi-agent orchestration, and moving deterministic work into code through programmatic tool calling. That third framing is the one to keep. The question is not "how do I make the model cleverer" but "which parts of this loop were never model work in the first place".

What it actually saves, and where it does not

Anthropic has published two figures that give the shape of the win. On a 75-tool project-management agent benchmark, enabling programmatic tool calling reduced billed input tokens by roughly 38% with no change in task accuracy. Across production API traffic, requests whose tools array contains 10 to 49 tool definitions see typical token savings of 20% to 40% with the feature enabled.

Read the two together and the scaling law is obvious: savings track the number of tools and the size of intermediate results. Below that band the published evidence does not speak, and the honest extrapolation is downward, not flat. So say it plainly: a three-tool agent will see little benefit. An agent calling a weather API, a calendar and a database lookup, each returning a small JSON object, has barely any intermediate data to keep out of the window and barely any definition weight to amortise. What you buy instead is a sandbox in the critical path and a program that has to be correct before anything useful happens. Anyone selling this as a universal cost win is not being straight with you.

The related manoeuvre of having agents call tools through generated code is explored further in our guide to code execution and MCP code mode; the difference here is that in 2026 this is a first-party API primitive rather than a pattern you assemble yourself.

Cache breakpoints: fixing the prefix on purpose

The definitions half needs the other lever, and here the mechanism is not orchestration but a decision about where your reusable prefix ends. The ordering rule that governs all prefix caching is not negotiable: static content must come before dynamic content for a prefix cache to be useful. If a session identifier or a timestamp sits ahead of your 12,000-token tools array, there is no stable prefix to reuse and no breakpoint will save you. Assume that discipline is in place; if it is not, fix it first.

Automatic placement versus explicit control

On Claude, prefixes are cached with cache_control, using either automatic caching or explicit breakpoints, with 5-minute or 1-hour time-to-live. Cache reads cost 0.1x the input rate. The choice between the two modes is a choice about who draws the line.

Automatic caching lets the provider place the boundary, which is the right default for a simple two-block prompt. Explicit placement earns its keep when your request contains several large blocks changing at different rates and you want the boundary at the last genuinely stable one. You can place cache_control directly on specific blocks for manual control over which large blocks form the reusable prefix — a long legal agreement, a schema, a codebase summary. Each is the kind of object that is enormous, stable for days, and sitting in front of a tiny volatile tail.

GPT-5.6 introduced more predictable prompt caching, including support for explicit cache breakpoints and a 30-minute minimum cache life. That minimum is the underrated part: caching has historically punished bursty traffic, and a five-minute window that is generous for a chat session is useless for an agent firing every eleven minutes, so a 30-minute floor changes which workloads are economically cacheable at all. OpenAI caching activates automatically from 1,024 tokens and bills cached input at one tenth of the standard rate.

The breakpoint the API places for you

One behaviour inside the agentic loop is easy to miss and worth designing around. When prompt caching is enabled and Claude uses a server tool — web search, web fetch, code execution — the API automatically places a cache breakpoint on the server tool result before running the next iteration of the agentic loop. Later iterations within the same request therefore read the growing prefix from cache instead of reprocessing it.

Go back to the six-turn arithmetic and this changes the picture materially: the triangular re-reading of earlier results is exactly what that automatic breakpoint flattens, turning a loop that was quadratically punishing into one where that re-reading is billed at a tenth of the rate. The caveat is in the wording — it applies to server tool results, so a loop built entirely on client tools your own application executes does not get the same treatment for free.

The second benefit is latency, which often changes a product decision rather than a budget line. Latency typically drops 30% to 80% on cache hits because prefill is often the slowest part of a request, and across six turns that compounds. A team in Bengaluru shipping a support agent whose customers expect a reply inside a few seconds may find the latency argument decides the design before the cost argument gets a hearing.

Which lever for which agent shape

This is the table to actually use. Find the row that matches your agent, and note that the last column matters as much as the middle two — a lever you cannot measure is a lever you cannot defend at the next budget review.

Agent shape	Programmatic tool calling?	Explicit cache breakpoint?	What to measure
Few tools (3–5), small results Weather, calendar, one lookup	No. Little intermediate data to keep out of the window; you buy complexity, not savings.	Only if the system prompt itself is large and stable. Automatic caching is usually enough.	Billed input tokens per task. If the definitions are under a few thousand tokens, stop optimising here.
Many tools (10–49), large results Ops agent over a warehouse and a CRM	Yes — this is the published sweet spot: typical savings of 20%–40%.	Yes. The tools array is the single largest stable block you re-send every turn.	Ratio of definition tokens to result tokens; savings on each half separately.
Very many tools (50+), heavy payloads Project-management or platform agent	Yes. Roughly 38% billed-input reduction on the 75-tool benchmark, accuracy unchanged.	Yes, and place it explicitly — several large blocks with different change rates.	Accuracy alongside cost. A savings number without an accuracy check is not a result.
Long agentic loop, many turns Research, migration, multi-step workflows	Yes, especially if intermediate results accumulate turn on turn.	Yes. Also exploit the automatic breakpoint on server tool results inside the loop.	Turns per completed task, and billed input per task — not per request.
Single-shot RAG or extraction One turn, big context, no loop	No. There is no loop and no accumulating intermediate data.	Yes, and this is the classic case: a large stable schema or corpus in front of a short query.	Cache hit rate on the prefix; cached-read share of total input tokens.
Multi-agent fan-out Concurrent subagents on one task	Yes, per subagent — each one carries its own tools array and results.	Yes. A shared stable prefix across subagents is worth pinning deliberately.	Total billed tokens per task across all agents, plus wall-clock time to result.

Published savings figures are Anthropic's, as of August 2026, for programmatic tool calling. Your result depends on tool count, payload size and turn count. Measure before and after on your own traffic.

Anthropic and OpenAI, side by side

The two platforms converged on similar ideas in 2026 but with different surfaces. The table below covers only what is publicly stated as of August 2026; an em dash means the item is not something we can confirm for that platform, not that it is absent. API surfaces move quickly — check the primary documentation linked at the end before you design around any single row.

Feature	Anthropic / Claude	OpenAI / GPT-5.6
Programmatic tool calling	Yes — from within the code execution sandbox (code_execution_20260120)	Yes — in the Responses API; programs run in-memory
Orchestration language	Runs inside the code execution sandbox	Can write JavaScript to orchestrate tools and filter data
Published token savings	~38% billed input on a 75-tool benchmark; 20–40% typical at 10–49 tools	—
Explicit cache breakpoints	Yes — cache_control on specific blocks, or automatic caching	Yes — explicit cache breakpoints, as part of more predictable caching
Cache lifetime	5-minute or 1-hour TTL	30-minute minimum cache life
Automatic caching threshold	Automatic caching available alongside explicit breakpoints	Activates automatically from 1,024 tokens
Cached-read price multiplier	0.1x the input rate	One tenth of the standard rate
Auto breakpoint on server tool results	Yes — placed before the next loop iteration when caching is enabled	—
Multi-agent orchestration	—	Native, initially beta; "Ultra" coordinates 4 agents by default, 16-agent configurations for select benchmarks
Zero Data Retention compatibility	—	Programmatic tool calling is stated ZDR compatible

As of August 2026. Em dash = not confirmed for that platform from primary sources, not necessarily unavailable. Verify against the linked documentation before committing.

One row deserves a warning label. OpenAI's multi-agent orchestration lets GPT-5.6 run concurrent subagents and synthesise their work in a single request, with "Ultra" mode coordinating four agents in parallel by default and 16-agent configurations used for select benchmarks. OpenAI is explicit that this trades a higher token bill for faster time-to-result. It is a latency feature, and it sits in the opposite direction to everything else in this article. Reach for it when a human is waiting; do not reach for it when the constraint is the invoice.

The wrong shape and the right shape

The blocks below are deliberately schematic pseudocode. The only real identifiers are cache_control and the two code_execution version strings; everything else stands in for whatever your SDK actually calls it. That is on purpose — an invented parameter name copied into production is worse than no example at all. Check the primary docs for exact signatures.

The wrong shape

This is the loop most teams ship first, and it reads perfectly naturally. It makes both mistakes at once: volatile content in front of the stable prefix, and every raw tool result appended to a history that gets re-sent forever.

# WRONG SHAPE — illustrative pseudocode.
# Two defects: (1) volatile content leads the prefix, (2) raw results accumulate in context.

history = []

def run_task(task, session_id, user):
    request = {
        # Defect 1: a session id and timestamp sit AHEAD of everything stable,
        # so the cacheable prefix is unique on every single request.
        "system": f"Session {session_id} started {now()}\n" + AGENT_INSTRUCTIONS + DOMAIN_SCHEMA,

        # Tools assembled in whatever order the loop happens to produce.
        # Different order -> different bytes -> no prefix match.
        "tools": [tool_schema(t) for t in user.enabled_tools],

        "messages": history,
    }

    while not done(request):
        reply = call_model(request)
        for call in reply.tool_calls:
            result = run_tool(call)          # e.g. 40,000 tokens of deal rows
            # Defect 2: the ENTIRE payload goes into the window, and is
            # re-tokenised on every subsequent turn of this task.
            history.append({"role": "tool", "content": result})
        request["messages"] = history        # whole history re-sent, in full

The right shape

The fix is two structural moves, neither of which touches your prompt wording. Pull everything stable to the front in a deterministic order and end it with an explicit breakpoint. Then push the intermediate processing into code so only the distilled result comes back into the window.

# RIGHT SHAPE — illustrative pseudocode.
# Real identifiers here: cache_control, code_execution_20260521. The rest is schematic.

# 1. Stable prefix first, in a fixed order, ending in an explicit breakpoint.
system = [
    {"type": "text", "text": AGENT_INSTRUCTIONS},   # changes on deploy, never per turn
    {"type": "text", "text": DOMAIN_SCHEMA,         # large, stable, worth pinning
     "cache_control": {"type": "ephemeral"}},       # breakpoint ends the reusable prefix
]

# 2. The tools array is part of that prefix too. Build it deterministically:
#    same set, same order, same bytes, every single request.
tools = (
    [{"type": "code_execution_20260521", "name": "code_execution"}]
    + sorted(BUSINESS_TOOLS, key=lambda t: t["name"])
)

# 3. Everything volatile goes AFTER the breakpoint, where it invalidates nothing.
messages = [{"role": "user",
             "content": f"Session {session_id}\nTask: {task}"}]

# 4. The model now writes a program that calls the tools and returns a summary,
#    rather than pulling every raw payload back through the context window.

The third block is the part that does the actual saving — the program the model writes and runs. Note what crosses the boundary back into the conversation: a few hundred tokens, not forty thousand.

# Illustrative: the shape of the program the model writes inside the sandbox.
# The 40,000-token payload lives here and dies here. It never enters the window.

rows = crm.list_deals(stage="negotiation")            # ~40,000 tokens of JSON
warehouse_rows = analytics.usage_by_account([r.account_id for r in rows])

at_risk = [
    r for r in rows
    if r.days_since_contact > 14
    and usage_dropped(warehouse_rows[r.account_id])
]

# Only this crosses back into the model's context — a few hundred tokens.
print(summarise(at_risk, fields=["account", "value", "days_since_contact"]))

Three things carry the change. The session identifier moved behind the breakpoint, so the prefix is byte-stable. The tools array is built in a deterministic order, which sounds trivial and is the single most common reason a cache silently never hits. And the join between two large data sources happens in code, where it always belonged, rather than in the context window at input-token prices.

How to actually measure this

Both features are easy to adopt and easy to fool yourself about. The failure is almost never that they do nothing — it is that you measured the wrong denominator and declared victory.

Measure two things. First, cache hit rate per layer: break it into the system prompt, the tools array and the conversation prefix, because those three fail independently and for different reasons. A blended 60% could be a perfectly cached system prompt sitting next to a tools array that never hits because you rebuild it in a different order per user, and the blended figure will never tell you that. Second, and this is the one that pays your bill, billed input tokens per completed task — not per request. A task is what your user asked for; a request is one turn of your loop.

Here is the concrete trap. You ship a change, per-request billed input drops 30%, cache hit rate climbs to 80%, the dashboard is green. Meanwhile the average task now takes eight turns instead of five, because the model is making more, smaller calls. Per-task cost went up. Every headline number improved and the invoice got worse. A cache hit rate reported without a per-task cost denominator hides exactly this regression.

Add two supporting lines: turns per completed task, so you can see the mechanism behind any movement, and task success rate alongside cost, because a cost reduction that quietly drops accuracy is not a saving. Anthropic's 38% figure was reported with no change in task accuracy, and that pairing is what makes it a result rather than a number. For the full accounting frame, our guide to LLM unit economics, cost per task and margin covers how to connect these to a price you can defend.

This is where the dual-market picture gets concrete. An Indian startup earning in rupees and paying an API bill in dollars carries the currency gap on every task, so a 30% cut in billed input is a direct margin gain on a spread it cannot otherwise control. A UK seed-stage team is usually solving a different equation: sterling runway measured in months, where the question is whether the same round buys four more months before the next raise. Both often serve customers priced in dollars, which is exactly why cost per task rather than cost per month is the number that travels between those two conversations — it is the only figure that survives translation into a price list.

Pitfalls that quietly eat the saving

Four ways teams lose the benefit after doing the work correctly.

Placing a breakpoint on content that is not actually stable. An explicit breakpoint is a claim that everything before it is byte-identical next time. If a schema block is regenerated per request with keys in a fresh order, or a tools array is filtered by user permissions so every permission combination is a different prefix, the claim is false and the cache write is wasted. Hash the exact bytes of your prefix across a sample of real requests before you pin a breakpoint behind them. If the hash varies, the breakpoint is decoration.

Assuming a retention or compliance property without checking it. OpenAI states its programmatic tool calling is Zero Data Retention compatible. That is a specific claim about a specific feature on a specific platform, and it does not generalise to every execution surface, region, or tool you might wire into a sandbox. For a fintech in Mumbai under DPDP obligations or a health-tech team in Manchester under UK GDPR, the compliance answer must come from the provider's own documentation and your contract — not from a blog post, including this one.

Chasing multi-agent parallelism when the bill, not the latency, is the constraint. Running four agents in parallel, or sixteen, gets you an answer faster and costs more, and OpenAI says so directly. If your problem is a cost line, parallel decomposition is the wrong end of the shop; if your problem is a user staring at a spinner, it is exactly right. Know which you have before you turn it on, and if the answer is "both", accept that they are separate projects.

Optimising the shape instead of the workload. Neither feature rescues an agent calling six tools where two would do, or one carrying forty definitions because nobody has pruned the list since March. If your tool count is the actual problem, dynamic tool retrieval attacks it directly; if your conversation history is the problem, context engineering and compaction for long-running agents is the lever; and if the model tier is oversized for the work, model routing and cascades will beat both features here combined.

The bottom line for builders

The durable insight outlives any particular API version. In a tool-heavy agent, the bill is dominated by re-reading — the same tool definitions on every turn, and the same tool results on every turn after they arrive. The definitions half is stable, so it is a caching problem, and an explicit breakpoint is you telling the provider where your stable region ends instead of hoping the heuristic guesses right. The results half is never stable, so it is an orchestration problem, and programmatic tool calling solves it by keeping the data in code where it never becomes tokens.

The practical sequence is short. Instrument billed input tokens per completed task, split it into definitions and results, and see which half dominates. Restructure the request so static content leads and volatile content trails — that costs nothing and is a prerequisite for everything else. Add an explicit breakpoint if your prompt has several large blocks changing at different rates. Then, only if your tool count and payload sizes put you in the published band, adopt programmatic tool calling and measure the before-and-after on cost and accuracy together. Be honest about the floor: with three tools and small results, neither feature is your answer, and the broader ladder of levers in our cache, route and compress cost playbook is the better place to spend the afternoon.

One last note on dates. Everything specific here — TTLs, price multipliers, token thresholds, tool version identifiers — is accurate as of August 2026 and will move. API surfaces change faster than the ideas underneath them. Treat the mechanism as durable, treat every number as a reading taken on a particular day, and confirm the current figures in the primary documentation before you build a cost model on them.

Sources

• Anthropic — Programmatic tool calling documentation
• Anthropic — Prompt caching documentation
• OpenAI — GPT-5.6 and Responses API announcements