Forecast Your LLM Bill Before Launch, Not After

What a usable forecast looks like

Caching, routing, batching, prompt compression, per-tenant showback: every one of those is a response to an invoice that has already arrived. They are good techniques and they belong in the toolkit, but none of them can be practised before you have production traffic. Forecasting can. It is the only cost discipline available to a team that has not launched yet, and it is the one that decides whether the price on your pricing page survives contact with real users.

Most teams do it badly, and the reported numbers are unflattering. Industry survey figures suggest only about 15 per cent of companies can forecast AI costs accurately, with most unable to land within plus or minus 10 per cent of the eventual bill. Seventy-three per cent overshoot their AI budget. Eighty-four per cent report margin erosion from AI costs. Those are self-reported survey figures rather than audited accounts, and they should be read as a description of a widespread pattern rather than a precise measurement — but the pattern is consistent enough to plan around.

The argument of this guide is that the failure is not one of arithmetic. It is a failure of artefact. Teams produce a point estimate — one number, in one cell, usually the output of requests multiplied by tokens multiplied by a published rate — and a point estimate is structurally the wrong shape for a workload whose cost per request has a long tail. Guidance from cost-forecasting practice says as much directly: point forecasts fail for AI workloads, and teams should forecast ranges of roughly plus or minus 40 per cent, then configure threshold alerts rather than hard stops so there is time to react before the budget actually breaks.

So the spine of the method is three moves. Build a three-scenario range rather than a number. Name every multiplier that sits between the naive formula and the real bill, and put each one in its own row so it can be argued with. Then instrument the product before launch so that two weeks of real traffic collapses the range into something you can price against. Everything below is those three moves in order, with a worked illustration you can copy.

Why point forecasts fail

The structural reason is distributional. Token consumption per request does not cluster around a mean the way, say, database query latency on a well-indexed table does. It has a long right tail, and the tail is not an anomaly — it is the part of the workload your best customers generate. Two requests that look identical to a product manager can differ by an order of magnitude in tokens consumed. A user asking your assistant to "summarise this thread" might mean a four-message thread or a four-hundred-message one. A retrieval step might pull three short passages or twelve long ones. An agent might resolve a task in one tool call or grind through nine.

Average that distribution into a single number and you get an estimate that is wrong in a specific and predictable direction. The mean of a long-tailed distribution sits above its median, so a model built on measured averages tends to overstate the typical request and still understate the total, because the tail is where the volume of tokens actually lives. A model built on the request you tried in the demo — which is almost always a short, clean, happy-path query — understates both.

There is a second failure that has nothing to do with statistics. A point estimate invites a false conversation. Presented with one number, a leadership team debates whether the number is right. Presented with a range and a list of named drivers, the same team debates which driver is most uncertain and what would resolve it — which is the conversation that produces a better product decision. The forecast is a communication artefact before it is a financial one.

This is why the plus or minus 40 per cent guidance is worth taking literally rather than treating as a hedge. A band that wide looks unserious to anyone used to forecasting seat-based SaaS revenue. It is honest for a workload where consumption per user is a behavioural variable rather than a contractual one, and it is narrow enough to be actionable: it tells finance what to reserve, and it tells engineering how much headroom the architecture needs.

The base formula, and why it is not enough

Almost every first LLM budget is built from the same three lines. They are not wrong. They are just radically incomplete, and it is worth writing them down explicitly so the omissions become visible.

# The formula every first LLM budget is built from.
# It is arithmetically correct. On its own it is also wrong,
# by a reported 2x to 10x, once real traffic arrives.

monthly_requests   = mau * sessions_per_user * requests_per_session
tokens_per_request = input_tokens + output_tokens
monthly_cost       = monthly_requests * tokens_per_request * rate_per_token

Read it as a set of assumptions rather than a calculation and the gaps announce themselves. It assumes exactly one model call per user-visible request, so every agent loop and tool-call round trip is invisible. It assumes no call ever fails and is retried. It assumes input_tokens is a constant, when in a conversational feature the input grows with every turn as history accumulates. It assumes the prompt contains only the user's words, when in practice a system prompt, formatting instructions and a handful of few-shot examples are prepended to every single call. It assumes no retrieved context, when a RAG feature may inject several thousand tokens per request before the user's question is even considered. And it assumes a single flat rate, when input, output, cached input and reasoning tokens are commonly billed differently.

None of those assumptions is unreasonable at the whiteboard stage. The problem is that they are multiplicative. A forecast that is 1.6 times low on prompt size, 1.6 times low on calls per request and 1.06 times low on retries is not 3.2 times low in total; it is 2.7 times low, and each of the three errors individually looked small enough to wave through. This is the mechanism behind the reported two-to-ten-times gap between initial estimates and real bills.

The fix is not a better single number. It is to promote each hidden assumption into an explicit row with its own value per scenario, which is what the next two sections build.

The five hidden multipliers

Five factors sit between the base formula and the invoice. Reported figures attribute the two-to-ten-times overshoot mainly to retries, agent loops and context growth, and in practice the other two — fixed prompt overhead and retrieved context — are the ones teams discover last because they are invisible in a code review. Take each in turn, and note that every one of them can be bounded before launch if you are willing to measure rather than assume.

Retries and the failure tax

Client libraries retry. Gateways retry. Your own orchestration layer probably retries. Rate-limit responses, timeouts, transient provider errors and schema-validation failures on structured output all produce a second attempt, and in many billing models a call that produced tokens before failing is a call you pay for. Worse, retries correlate with load: they cluster at exactly the moment your traffic is highest, so their contribution to the bill is not evenly spread.

To bound it, count attempts rather than successes in your load testing, and separate them by cause. A retry rate driven by rate limits is a capacity problem you can fix; a retry rate driven by structured-output validation failures is a prompt problem that will not improve on its own. Carry the rate as an explicit percentage in the model — a few per cent in the central case, a low double-digit percentage in the stress case.

Agent loops and tool-call round trips

This is the largest multiplier in most agentic products and the one least visible in a product specification. A single user-visible action — "book this, then confirm it" — becomes a plan step, a tool call, a result observation, possibly a correction, and a final response. Each of those is a billed model call, and each carries the full accumulated context forward.

The important property is that the number of round trips is not fixed. It is a function of task difficulty, tool reliability and how well the model is grounded, all of which vary by customer. Bound it by measuring the distribution of round trips per completed task in your evaluation harness, then carry the median as the central case and the ninetieth percentile as the stress case. If your product allows an agent to loop without a hard iteration ceiling, the stress case is whatever your ceiling would have been, and you should add one.

Context growth over a session

In any conversational feature, the input to turn ten contains most of turns one through nine. Cost per turn therefore rises across a session even though the user's typing does not get longer. A ten-turn conversation does not cost ten times a single turn; it costs considerably more, and the shape of that growth depends entirely on your history-management strategy.

Bound it by deciding the strategy before you forecast, not after. A fixed sliding window, a summarisation step at a token threshold, or a hard turn limit each give you a defensible ceiling on input tokens per call. Without one of those, the honest stress-case answer is your context window, which is a number nobody wants in a budget.

System-prompt and few-shot overhead

The quiet one. Every call carries your system prompt, your output-format instructions, your safety preamble and any few-shot examples — and it carries them whether the user typed three words or three hundred. On a short-query feature this fixed overhead can dominate the variable part entirely, which means shaving it is one of the highest-leverage optimisations available and inflating it is one of the easiest ways to break a forecast after launch.

Bound it exactly, because unlike the others this one is knowable before a single user arrives: tokenise your actual system prompt and few-shot block and put the number in the model as a constant. Then add a governance note, because the failure mode here is organisational. System prompts grow by accretion as teams patch behaviour, and nobody re-runs the forecast when three sentences are added. Prompt caching changes the arithmetic considerably where it applies, which is covered in the guide to caching, routing and compressing LLM calls — but treat cache hit rate as a forecast input with its own uncertainty, not as a discount you can assume.

Retrieved context from RAG

A retrieval step injects passages into the prompt, and the token cost of those passages is usually larger than the user's question by an order of magnitude. The driver is your top-k setting multiplied by your chunk size, both of which are tuning parameters someone will change to improve answer quality without anyone re-opening the cost model. Reranking, query expansion and multi-hop retrieval each add further calls on top.

Bound it by treating top-k times chunk size as a hard budget rather than a tuning knob, and by forecasting the retrieval calls separately from the generation calls. If your retrieval quality work is likely to increase k over the next two quarters, put that in the stress scenario rather than discovering it in an invoice.

Build the three-scenario model

Now assemble it. The artefact is a small table with one row per driver and one column per scenario: a central case, a stress case and a worst case. Keep it in a spreadsheet if that is where your finance partner lives; keep it in code if you want it re-runnable. The structure matters more than the tool.

The worked illustration below uses a deliberately simple product: a chat feature. As a clearly illustrative figure and not a market rate, assume a blended cost of 0.005 US dollars per baseline request — one model call, baseline prompt, baseline output, no retry, no cache. With 100,000 monthly active users averaging 20 queries each, that is 2,000,000 requests, and 2,000,000 multiplied by 0.005 gives 10,000 US dollars per month of inference cost before any infrastructure overhead. That figure is the naive point estimate, and it is the number this guide exists to improve on. Substitute your own rate card for the 0.005; it is an input, not a benchmark.

Driver	Central case (P50)	Stress case (P90)	Worst case	How to set it
Monthly active users	100,000	120,000	150,000	Growth plan; worst case is the plan going right
Sessions per user per month	8	9	10	Analogous feature in your own product
Requests per session	2.5	3.0	3.6	Beta traffic, or the evaluation harness
Prompt and output size vs baseline	1.6×	2.2×	3.0×	System prompt plus few-shot plus retrieved context
Model calls per user-visible request	1.0	1.6	2.2	Median and p90 round trips from the eval harness
Retry rate	3%	6%	12%	Attempts, not successes, under load test
Cache hit rate (billable input avoided)	0%	20%	25%	Assume zero until measured
Multiplier stack	1.65×	2.98×	5.54×	Product of the four rows above
Monthly inference cost (USD)	$16,480	$48,356	$149,688	Requests × unit cost × stack
Cost per active user per month (USD)	$0.16	$0.40	$1.00	The number your pricing must survive

Three things in that table are worth dwelling on. First, the central case is 16,480 US dollars, not 10,000 — the naive point estimate turns out to sit at the optimistic edge of the realistic band rather than at its centre. Apply the plus or minus 40 per cent guidance to the central case and the planning band runs from about 9,900 to about 23,100 US dollars a month, which is a range a finance partner can actually reserve against.

Second, the worst case is roughly fifteen times the naive estimate, and that is not a scare number — it decomposes cleanly. About 2.7 times of it is simply more demand than planned, and about 5.5 times is the multiplier stack, which sits comfortably inside the two-to-ten-times range reported for hidden multipliers. Separating those two sources is the whole point of the table: one is a commercial upside you would welcome, the other is an engineering liability you should fix.

Third, the same product costs somewhere between roughly 16 US cents and one US dollar per active user per month depending on which scenario lands. That six-fold spread is the number your pricing page has to survive, and it is invisible in any point estimate.

"""Three-scenario LLM cost model.

Every number below is an INPUT from your own product, not a market rate.
`unit_cost` is a blended cost per BASELINE request from your own rate
card: one model call, baseline prompt, baseline output, no retry, no
cache. Rates move; this structure does not.
"""

BASELINE_COST_PER_REQUEST = 0.005   # USD. ILLUSTRATIVE -- use your own.


def monthly_cost(mau,
                 sessions_per_user,
                 requests_per_session,
                 context_factor,          # prompt + output size vs baseline
                 calls_per_request,       # model calls per USER-VISIBLE request
                 retry_rate,              # 0.06 == 6% of calls retried once
                 cache_hit_rate,          # fraction of billable input avoided
                 unit_cost=BASELINE_COST_PER_REQUEST):
    user_visible = mau * sessions_per_user * requests_per_session
    stack = (context_factor
             * calls_per_request
             * (1.0 + retry_rate)
             * (1.0 - cache_hit_rate))
    return user_visible * unit_cost * stack, stack


SCENARIOS = {
    "P50":   dict(mau=100_000, sessions_per_user=8,  requests_per_session=2.5,
                  context_factor=1.6, calls_per_request=1.0,
                  retry_rate=0.03, cache_hit_rate=0.00),
    "P90":   dict(mau=120_000, sessions_per_user=9,  requests_per_session=3.0,
                  context_factor=2.2, calls_per_request=1.6,
                  retry_rate=0.06, cache_hit_rate=0.20),
    "Worst": dict(mau=150_000, sessions_per_user=10, requests_per_session=3.6,
                  context_factor=3.0, calls_per_request=2.2,
                  retry_rate=0.12, cache_hit_rate=0.25),
}

if __name__ == "__main__":
    for name, kw in SCENARIOS.items():
        cost, stack = monthly_cost(**kw)
        per_user = cost / kw["mau"]
        print(f"{name:<6} ${cost:>10,.0f}/mo  stack x{stack:4.2f}  "
              f"${per_user:.2f}/MAU")
    # P50   $    16,480/mo  stack x1.65  $0.16/MAU
    # P90   $    48,356/mo  stack x2.98  $0.40/MAU
    # Worst $   149,688/mo  stack x5.54  $1.00/MAU

    # Planning band: apply +/-40% to the CENTRAL case, not to the worst.
    p50, _ = monthly_cost(**SCENARIOS["P50"])
    print(f"planning band: ${p50 * 0.6:,.0f} to ${p50 * 1.4:,.0f} per month")
    # planning band: $9,888 to $23,072 per month

Keep every figure in one currency inside the model and convert once at the finance layer. Provider rate cards are quoted in US dollars almost everywhere, including by Indian and UK resellers, while your revenue may arrive in rupees from a Bengaluru customer and in pounds from a Manchester one. That mismatch is a real exposure sitting inside your gross margin, and it belongs in the forecast as its own row with its own uncertainty — not buried inside a blended average at whatever rate happened to apply on the day you built the spreadsheet.

Instrument before launch, not after

A forecast built from assumptions is only useful if reality is allowed to correct it quickly. The goal of pre-launch instrumentation is not to compute your bill — the provider invoice does that, eventually and unhelpfully late. The goal is to emit the drivers of the bill, tagged, so that two weeks of real traffic collapses a plus or minus 40 per cent band into something narrow enough to price against.

The minimum viable record is one structured log line per model call. It should carry the token counts, the model identifier, the attempt number, the tool-call count and the cache read tokens — and, critically, three business dimensions: which feature, which route, which tenant. Token counts without those dimensions tell you the total and nothing else, which is the position most teams find themselves in when they try to work out which customer is unprofitable.

For attribute naming, the OpenTelemetry GenAI semantic conventions give you conventional field names for the token counts, notably gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Adopting them costs nothing at the point of writing the wrapper and means your fields line up with tracing you add later; support varies by SDK and version, so verify rather than assume. The broader tracing picture is covered in the guide to agent observability with OpenTelemetry.

"""Minimal per-request cost logging. Provider-agnostic, illustrative.

Wrap every model call from day one. This does not compute your bill --
it emits the DRIVERS of the bill, tagged, so the forecast can be checked
against reality in weeks rather than quarters.

Token attribute names follow the OpenTelemetry GenAI semantic
conventions. Support varies by SDK and version -- check yours.
"""

import time
import uuid
from contextlib import contextmanager


@contextmanager
def cost_span(logger, *, feature, route, tenant_id, model, trace_id=None):
    record = {
        "trace_id": trace_id or str(uuid.uuid4()),

        # Business dimensions. Without these you can attribute NOTHING:
        # not per-feature, not per-tenant, not per-route.
        "feature": feature,
        "route": route,
        "tenant_id": tenant_id,

        # OpenTelemetry GenAI semantic-convention attribute names.
        "gen_ai.request.model": model,
        "gen_ai.usage.input_tokens": 0,
        "gen_ai.usage.output_tokens": 0,

        # The multipliers. Log them even when they are zero -- a field
        # that is always 0 is evidence; a missing field is a blind spot.
        "attempt": 1,
        "tool_calls": 0,
        "cache_read_tokens": 0,
        "error": None,
    }
    started = time.perf_counter()
    try:
        yield record
    except Exception as exc:
        record["error"] = type(exc).__name__
        raise
    finally:
        elapsed = (time.perf_counter() - started) * 1000
        record["latency_ms"] = round(elapsed, 2)
        logger.info("llm_call", extra={"llm": record})


# --- usage -------------------------------------------------------------
with cost_span(log,
               feature="summarise_thread",
               route="POST /v1/threads/summary",
               tenant_id=tenant.id,
               model=MODEL) as rec:

    for attempt in range(1, MAX_ATTEMPTS + 1):
        rec["attempt"] = attempt          # counts ATTEMPTS, not successes
        try:
            resp = client.complete(prompt=prompt, model=MODEL)
            break
        except TransientError:
            if attempt == MAX_ATTEMPTS:
                raise

    usage = resp.usage
    rec["gen_ai.usage.input_tokens"] = usage.input_tokens
    rec["gen_ai.usage.output_tokens"] = usage.output_tokens
    rec["cache_read_tokens"] = getattr(usage, "cache_read_tokens", 0)
    rec["tool_calls"] = len(resp.tool_calls or ())

Two weeks of this data answers the questions the forecast could only guess at: what is the real distribution of tokens per request, what is the real retry rate under real load, how many model calls does a real user action take, and which feature is generating the tail. Roll those measured values back into the scenario table and the band narrows dramatically — usually with one or two drivers turning out to be badly wrong in a direction nobody predicted.

Once the dimensions are in place, cost attribution follows almost for free. Aggregating the same records by feature gives you the showback view described in the guide to per-feature LLM cost attribution and showback; aggregating by tenant gives you the per-customer view that per-tenant cost attribution for agent products builds on. Both are considerably harder to retrofit than to add on day one, which is the practical argument for doing this before launch rather than after the first alarming invoice.

Alerts that work: thresholds, not hard stops

With drivers instrumented, the question becomes what to do when reality departs from the forecast. The instinct is a hard budget cap: stop spending at a fixed figure. Resist it on user-facing paths.

The workable pattern is a ladder of alerts with different time constants, each catching a different failure and each with an explicit statement of what it must not do automatically. Automation belongs on the detection side; the response should be a human decision unless the fallback is genuinely graceful.

Alert	What it catches	What it must NOT do
Forecast-to-month-end breach — extrapolate month-to-date spend daily; fire when the projection exceeds the stress-case line	Slow drift that no single day would flag; the gradual creep that produces a 73 per cent overshoot rate	Throttle or block requests. Its output is a decision meeting, not an action
Day-over-day step change — spend for a feature moves beyond a set band against the trailing median for the same weekday	A deploy that changed the prompt, the model, the retrieval top-k or the retry policy	Page anyone during a known launch window. Suppress it deliberately, with an expiry
Per-tenant anomaly — a tenant's cost per active user leaves its own trailing band	Abuse, a runaway agent loop, or one customer with a pathological workload	Suspend the tenant automatically. Enterprise customers do not forgive that
Token tail alert — p99 tokens per request for a feature crosses a ceiling	Context growth and retrieval bloat, days before either reaches the invoice	Truncate context silently. Silent truncation degrades answers invisibly
Multiplier drift — measured retry rate or calls per request falls outside the forecast band	The model's assumptions going stale, which is the root cause of most of the others	Auto-adjust the forecast. A human should approve every re-baseline

Two implementation notes. Alert on rate of change as well as absolute level, because absolute thresholds either fire constantly during growth or never fire at all. And route cost alerts to the team that owns the feature, not only to finance — the person who can shorten a system prompt is the person who should see the token tail alert first.

Where automation is genuinely safe is graceful degradation rather than refusal: falling back to a smaller model, serving a cached answer, or shortening the context budget on a non-critical path. That is a routing decision rather than a stop, and the mechanics are covered in the guide to model routing and cascades. Keep a documented manual kill switch as well, for the abuse case where stopping really is the right answer.

Turning the forecast into pricing

A forecast that does not change how you price is an accounting exercise. This is where the range earns its keep, and where an honest conversation about margin has to happen.

The reported margin picture for AI products is sobering. Industry projections put AI gross margins at around 45 per cent in 2025, rising to 53 per cent in 2026 and 59 per cent in 2027, against 80 to 90 per cent for traditional SaaS. The fastest-scaling cohort of AI-first companies reportedly shows roughly 25 per cent gross margin, and often negative. Eighty-four per cent of companies report margin erosion from AI costs. These are reported and projected figures rather than audited results, but the direction is consistent: an AI feature is a fundamentally more expensive thing to sell than a seat in a database-backed application, and pricing that assumes otherwise fails quietly.

Put the worked illustration against that backdrop. To hold the 53 per cent blended gross margin reported for 2026, your entire cost of revenue — inference, retrieval infrastructure, vector storage, egress and the support burden — has to fit inside 47 per cent of revenue. At the stress-case line of 48,356 US dollars a month, inference alone would need revenue above roughly 102,900 US dollars a month just to sit at that ceiling, before a single other cost of revenue is counted. That is the arithmetic that should drive the pricing conversation, not the naive 10,000 US dollar figure.

The per-seat versus usage-based choice then becomes concrete rather than philosophical. Per-seat pricing decouples what you charge from what a customer consumes, which customers like and which means your price has to cover your heaviest seats, not your average ones — with a six-fold spread between the central and worst-case cost per active user, that is a wide margin to carry. Usage-based pricing aligns price with cost and transfers the volatility to the customer, which protects your margin and makes your product harder to budget for. Hybrid structures, where a per-seat base includes a usage allowance and overage is metered, are the common compromise, and the allowance should be set from your measured distribution rather than your average.

The regional dimension is real and cuts in opposite directions. A UK enterprise buyer running a formal procurement process typically wants a fixed annual figure in pounds sterling and will push back hard on a meter they cannot forecast, which argues for a per-seat structure with a generous but capped allowance. An Indian SaaS buyer is often sharper on price and more comfortable with consumption tiers, which argues for usage-based with a free allowance that lets the product prove itself first. Both are consuming the same dollar-denominated inference, so the pound-denominated contract and the rupee-denominated contract carry different currency exposure on identical costs. Price each market on its own landed margin; do not assume a single global list price converts sensibly, and do not build the model on a fixed exchange-rate assumption.

Finally, the trap. One analysis covering 2.4 billion enterprise API calls reported blended token cost falling from 18.40 to 6.07 US dollars per million tokens between the first quarter of 2025 and the first quarter of 2026 — a 67 per cent decline. It is tempting to price off whatever rates apply on the day you build the model and let the decline expand your margin. It will not, because the decline is available to your competitors on the same terms, and the saving tends to flow into price competition rather than into anyone's gross margin. Price on your consumption model and your differentiation; treat the rate as the input most likely to change, and re-run the model when it does. The unit-economics framing in the guide to LLM unit economics, cost per task and margin picks up where this one stops.

What to re-forecast, and when

A forecast is a perishable artefact, and the value of the method is that it is cheap to re-run. Two cadences are enough.

Monthly, on a calendar entry. Compare realised spend against the scenario band and record which scenario the month actually landed in. Update the demand drivers — monthly active users, sessions per user, requests per session — from real product analytics rather than the growth plan. Re-measure the four multiplier rows from the logs, because retry rate and calls per request drift with every release. Then treat any gap above a few per cent between predicted and realised spend as a defect to be explained rather than noise to be absorbed; that gap is usually where an untagged feature, a forgotten background job or a quietly re-priced tier is hiding.

On change, not on schedule. Some events invalidate the model immediately and should trigger a re-run regardless of where you are in the month: switching or upgrading a model, a provider changing its rate card, adding or removing a retrieval step, changing top-k or chunk size, altering the system prompt materially, changing retry or timeout policy, enabling or disabling prompt caching, and onboarding a customer whose usage profile differs from your existing base. Adding that list to your release checklist costs nothing and catches most of the surprises.

Two structural habits make both cadences cheap. Keep the model in code so it is diffable and re-runnable, and keep the tags stable so month-over-month comparisons remain valid — a renamed feature tag silently breaks a year of history. Where a workload turns out to be deferrable, moving it to asynchronous processing changes the unit cost enough to justify a fresh scenario rather than an adjustment to the old one. The wider direction of travel in rates and margins is tracked in our reporting on AI inference cost economics.

Every rate quoted in this guide will be wrong eventually, and the 0.005 US dollars per request illustration was never right for anyone but the example. The method is what lasts: a range instead of a number, one row per named multiplier, instrumentation that collapses the range within a fortnight, alerts that inform rather than interrupt, and pricing set against a margin you have actually computed. As of September 2026 that discipline remains rare enough — reportedly practised well by around 15 per cent of companies — that being able to demonstrate it is a genuine professional differentiator. If you have built and defended a cost model for a real AI product, put it on a Builder profile; the teams hiring for platform and cost-engineering work across Bengaluru, Chennai, London and Manchester are looking for exactly that evidence.