The Batch API Playbook: Cut LLM Costs 50% on Async Jobs (2026)

What you need to know

There is a 50% discount sitting unused in most production LLM bills, and the only thing standing between a team and claiming it is a willingness to wait a few hours for results that nobody is watching in real time. Both the Anthropic Message Batches API and the OpenAI Batch API process requests asynchronously and return results within a 24-hour service-level agreement — frequently much sooner — at exactly half the standard token price, on both input and output. There is no catch hidden in the model: the same model produces the same output in the same format. You are simply telling the provider that this work is not urgent, and being paid for that flexibility.

The reason so many teams leave the money on the table is that they built their first pipeline on the synchronous API during the prototype, when everything felt urgent, and never went back to ask which parts actually were. A document-classification job that runs overnight does not need a sub-second response. A nightly analytics summary does not need to come back while an engineer waits. An offline evaluation over ten thousand examples is, by definition, offline. Every one of those is a candidate for batch, and every one of them is, on most teams, still being billed at full sync price out of habit rather than need.

This guide is the playbook for fixing that. It explains exactly how batch processing works under both major providers, gives you the Python to submit a JSONL file of requests and join the results back to your inputs, builds a worked savings model you can adapt to your own numbers, and shows how to stack batch with prompt caching and the right model tier for savings of up to roughly 75% on suitable workloads. It also draws the line clearly: batch is wrong for anything interactive, and we will be specific about where not to use it. Whether you run an Indian e-commerce catalogue-enrichment pipeline or a UK media archive transcription job, the mechanics and the maths are the same.

How batch processing works

The mental model is simple: instead of sending requests one at a time and waiting for each response, you hand the provider a file containing many requests at once, walk away, and come back later for the whole set of results. The provider processes them on its own schedule — wherever there is spare capacity — and in exchange for that flexibility it bills you at half the standard rate. The 50% applies to both input and output tokens, so it is a clean halving of the bill for the work you move across, not a discount on one side only.

Concretely, the flow is the same under both Anthropic and OpenAI, and it has five steps. First, you build a JSONL file where each line is one request in the same schema you already use for the synchronous API, with one addition: a custom_id that you choose, so you can match each result back to its input later. Second, you submit that file and receive a batch or job id. Third, you poll that id for status — the job moves from in-progress to completed. Fourth, once complete, you download the results file. Fifth, you join each result back to its original input by custom_id, because the results may not arrive in the order you submitted them.

One detail matters from the start: failures are reported per request, not for the batch as a whole. A single malformed line or a content filter triggered on one input does not sink the other 9,999 — it comes back marked as an error for that custom_id while everything else succeeds. This is a feature, because it means a large job degrades gracefully rather than failing wholesale, but it means your code must handle partial failures: read every result, separate the successes from the errors, and decide what to do with the errors (retry on the sync API, log for review, or skip). Treat the results file as a mixed bag, never as a guaranteed clean set.

Sync API vs Batch API: the trade-off

The decision to batch a workload comes down to one question — can it tolerate a few hours of latency? — and everything else follows from the answer. The table below lays the two side by side so the trade-off is explicit. Note the limits in particular, because they shape how you chunk a large job: OpenAI accepts far more requests per file, while Anthropic's per-batch ceiling is lower, so a very large Anthropic job needs to be split across multiple batches.

Pricing as of June 2026. The 50% discount and the 24-hour SLA apply to both the Anthropic Message Batches API and the OpenAI Batch API; request and file limits are provider-specific and change over time, so confirm current figures with each provider before you design around them.

The clarifying way to read this table is that the synchronous API is not "better" — it is faster and dearer, and speed is a feature you should only buy where it is used. A reader is waiting on a chat reply, so that path earns its sync price. A back-office job summarising last night's transactions is waited on by nobody, so paying sync rates for it is pure waste. The art of batch optimisation is mostly the discipline of auditing your traffic honestly and moving everything that can move. If you have already worked through caching and routing, batch is the orthogonal third lever; our companion guide on cutting LLM costs 70–90% with cache, route and compress sets out how these stack.

The code: submit, poll, join

Here is the end-to-end pattern in Python: build a JSONL of requests with a meaningful custom_id, submit the batch, poll until it completes, then download and join the results — handling partial failures explicitly. This example uses the Anthropic Message Batches API; the OpenAI Batch API follows the same shape (upload a JSONL file, create a batch referencing it, poll, retrieve the output file). Code stays in US English, as is conventional.

import time
from anthropic import Anthropic
from anthropic.types.messages.batch_create_params import Request
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming

client = Anthropic()

# 1. Build requests. Each carries a custom_id so we can rejoin results.
#    The body is a normal Messages request — same schema as the sync API.
documents = load_documents()   # e.g. [{"sku": "48213", "text": "..."}, ...]

requests = []
for doc in documents:
    requests.append(
        Request(
            custom_id=f"sku-{doc['sku']}-desc",   # meaningful id, not a counter
            params=MessageCreateParamsNonStreaming(
                model="claude-haiku",             # cheapest sufficient tier
                max_tokens=512,
                system="Extract product attributes as JSON.",
                messages=[{"role": "user", "content": doc["text"]}],
            ),
        )
    )

# 2. Submit the batch. Anthropic allows up to 10,000 requests per batch,
#    so chunk larger jobs. (OpenAI allows up to 50,000 per file.)
batch = client.messages.batches.create(requests=requests)
batch_id = batch.id

# 3. Poll for completion. Results arrive within the 24h SLA, often sooner.
while True:
    status = client.messages.batches.retrieve(batch_id)
    if status.processing_status == "ended":
        break
    time.sleep(60)   # back off; do not hammer the endpoint

# 4. Retrieve results and 5. join them back to inputs by custom_id.
results, failures = {}, {}
for entry in client.messages.batches.results(batch_id):
    cid = entry.custom_id
    if entry.result.type == "succeeded":
        results[cid] = entry.result.message.content
    else:
        # Partial failures are reported PER REQUEST — handle them.
        failures[cid] = entry.result.type   # "errored" | "canceled" | "expired"

print(f"ok={len(results)} failed={len(failures)}")
# Retry failures on the sync API, log them, or skip — your call.

Three things in that snippet are load-bearing. The custom_id is what makes the results joinable, because results need not arrive in submission order — never rely on position. The processing_status == "ended" check with a 60-second back-off is a polite poll; do not spin tightly on the status endpoint. And the split into results and failures is the partial-failure handling the spec quietly demands of you: a real job will have a few errored or expired lines, and ignoring them silently drops records from your pipeline. For the OpenAI equivalent, you first upload the JSONL with the files endpoint, create a batch with the /v1/batches endpoint pointing at that file id and the 24h completion window, poll the batch object, then download the output file id and parse it line by line — same five steps, different method names.

A worked savings model

Numbers make the case better than adjectives, so here is a worked example you can re-run with your own figures. Picture a service processing 500,000 requests per month — an Indian e-commerce catalogue-enrichment job tagging product listings, say, or a UK media archive transcription job working through a back catalogue. Assume, illustratively, that each request costs $0.01 on the synchronous API at the chosen model tier. The table indexes the three states: full sync, the same work on batch at half price, and batch stacked with prompt caching on the shared static prefix.

Pricing as of June 2026. The $0.01 per-request baseline is illustrative and chosen for clean arithmetic, not a vendor list price; your real figure depends on model tier, token counts and region. The batch line is an exact 50% cut; the batch-plus-cache line assumes a workload with a large shared static prefix where caching approaches its ceiling, landing near the ~75% total-savings band.

Read the right-hand column and the compounding is plain. Moving the work to batch halves the bill from $5,000 to $2,500 with no change to the model or the output — a flat 50% saved for accepting asynchronous processing. Stacking prompt caching on top, where every request in the batch shares the same large system prompt or extraction schema, removes most of the input cost on that shared prefix and pulls the bill down toward $1,250 — roughly 75% off the synchronous baseline. That upper figure is workload-dependent: caching only approaches 90% on the cached portion when the static prefix is large relative to the variable input, which is exactly the shape of a bulk-extraction job.

The same arithmetic scales to partial adoption, which is how most teams actually start. Suppose your monthly API spend is $2,000 and you identify that 60% of your requests are offline work that can move to batch. Halving the price on that 60% saves you about $600 per month — roughly $7,200 per year — with zero change to model quality or output format. That is a meaningful number for a change that is, in engineering terms, a weekend: rewrite the offline jobs to assemble a JSONL, submit, poll and join, and leave the interactive paths exactly as they are. For a deeper look at the caching half of the stack, see our guide to cutting LLM API costs up to 90% with prompt caching.

Which workloads to batch — and which never to

The single most important skill in batch optimisation is sorting your traffic correctly, because the discount is only ever as good as your honesty about what can wait. The good fits share one property: no human is blocked on any individual result. That covers document processing pipelines, data enrichment at scale, nightly analytics jobs, offline evaluations, content and embedding generation queues, and bulk classification, extraction or summarisation. If the work runs on a schedule, fills a queue, or backfills a dataset, it almost certainly belongs on batch.

Two concrete examples make the pattern tangible across both our markets. An Indian e-commerce catalogue-enrichment job takes tens of thousands of raw product listings each night and rewrites titles, extracts structured attributes and generates short descriptions — a perfect batch workload, because the catalogue is read by shoppers in the morning, not the instant the model finishes. A UK media archive transcription job works through a decades-deep back catalogue of recordings, transcribing and summarising each one for search; nobody is waiting on any single file, so paying double for speed would be indefensible. Both jobs assemble a JSONL, submit overnight, and reconcile results before the working day starts.

Stack it further: model tier and caching

Batch is one lever of three, and the largest savings come from pulling all of them at once because they are orthogonal — each operates on a different part of the cost, so they multiply rather than overlap. The first lever is model choice: pick the cheapest model tier that is sufficient for the task. A great deal of batch work is extraction, classification and short summarisation, which a small, fast model such as a Haiku-class tier handles perfectly well; reserve the frontier tier for the genuinely hard requests. Because batch is independent of which model you choose, you get the 50% on top of whatever tier you select, so getting the tier right first multiplies the benefit.

The second lever is prompt caching, which cuts the cost of cached input tokens by up to 90% by storing the processed form of a static prefix and reusing it across requests. In a batch job this is almost free money, because a bulk job is precisely the case where thousands of requests share the same large system prompt, the same extraction schema or the same policy text. Order that static content at the front of every request, mark the cache breakpoint, and the shared prefix is processed once rather than 10,000 times. Stacked with the 50% batch discount on a workload that suits both, the combination reaches up to roughly 75% total savings.

The discipline, then, is to layer deliberately rather than reach for one trick. Choose the right model per task — batch is orthogonal to model choice, so this is a separate decision you make first. Then move every offline-tolerant request to batch for the flat 50%. Then cache the shared static content for up to 90% off the cached portion on top. Each layer compounds on the smaller bill the previous one produced, and none of them touches the quality or format of the output. For teams weighing whether to run their own inference instead, our analysis of self-hosting versus API LLM inference in 2026 sets the build-versus-buy maths against exactly these API economics, and our broader guide to cutting LLM costs with prompt caching and model routing rounds out the picture.

Conclusion and next steps

The headline is unusually simple for a cost-optimisation piece: there is a flat 50% discount available on every request you can afford to run asynchronously, under both major providers, with no change to model quality or output format. The work to claim it is modest — rewrite your offline jobs to build a JSONL with meaningful custom_ids, submit, poll, and join the results while handling partial failures — and it compounds beautifully with the cheapest sufficient model tier and prompt caching for up to roughly 75% off on the right workloads. The only real discipline required is honesty about which paths can wait and which cannot.

Start this week with one job. Find the offline pipeline that nobody waits on in real time — the nightly enrichment, the back-catalogue transcription, the weekly eval — move it to batch, measure the bill before and after, and let the number make the case for the next one. If you are the engineer who quietly halved a production LLM bill without anyone noticing a quality change, that is exactly the shipped, measurable work the people hiring in AI want to see, and a Verified Builder profile on AI Tech Connect is where you put it in front of them.

Sources

• Anthropic — Message Batches API documentation
• OpenAI — Batch API guide
• Respan — A practical look at the Anthropic Message Batches API
• VentureBeat — Anthropic challenges OpenAI with affordable batch processing
• Finout — Anthropic API pricing explained