Cost-Aware Evals: Score Quality Per Pound, Not Just Quality

What a single quality score hides

Your eval suite finishes and prints one number. Say it is 0.87. That tells you the configuration you tested beat the one scoring 0.84 and lost to the one scoring 0.91, and nothing else. It cannot answer what the next planning meeting will turn on: whether the small model — the one that scored 0.86 — is good enough at a fifth of the price.

That gap is a design problem in the harness, not a reporting problem. A harness recording only quality threw away the information needed to answer the question before anyone thought to ask it, and no dashboard work recovers it: you cannot compute the cost of a run you did not measure.

The fix is small and structural. Make cost and latency first-class scored dimensions of every eval case, exactly as quality already is. Once a run emits three numbers per case, model selection stops being a leaderboard ranking and becomes a Pareto frontier — the set of configurations nothing else beats on every axis at once — plus a mechanical rule for choosing among them. Whether the flagship is worth it stops being settled by rhetoric and starts being settled by a table.

• Three numbers per case — quality, cost and latency — in the same row.
• Store tokens, never currency. Prices change; token counts do not.
• A price table held separately, applied at report time, so Bengaluru and London read identical measurements through different rates.
• A quality floor set before you see prices, turning the frontier into one defensible choice.
• Latency as a third axis, which prunes the frontier first on interactive workloads.

Two adjacent questions are out of scope. This is not a guide to your product's economics — see LLM unit economics and cost per successful task for that. Nor is it about sending different traffic to different models at runtime, covered in model routing and cascades. Both are downstream: you need a harness that can measure a cost difference before you can justify a routing policy or defend a margin.

The three numbers every eval case should emit

Start with the record, not the report. Capture everything cheap at the moment of the call and impossible to reconstruct later, and summarise nothing at write time.

Quality is whatever your grader already produces, normalised to a consistent scale, plus metadata: which grader, which rubric version, which judge model if any. Without those fields you cannot tell six weeks later whether two runs were scored by the same instrument.

Cost is not a number you record. It is one you derive from token counts you record — the most consequential decision in the design, and one that gets its own section below.

Latency is two numbers: wall clock for the whole case, retries and tool round trips included, because that is what a user waits; and time to first token, the only latency figure that means anything for a streaming interface.

Record tokens, never currency

It is tempting to have the runner compute a cost in pounds or rupees and store that. Do not. Store the token counts and convert to money only when you generate a report.

Prices are volatile and measurements are not. Rates change, tiers appear, a contract lands, a regional price diverges from list, a batch discount covers half your workload. Every one of those invalidates a stored currency figure; none of them touch a stored token count. A harness that persisted pounds has an eval history that became fiction the day pricing moved, and it cannot be repaired, because the tokens that would let you recompute were never written down.

A harness that persisted tokens is permanently re-scorable: edit one price file, re-run the report over results gathered months ago, and every historical comparison is correct again under the new rates — which is what lets you answer, in an afternoon and for nothing, whether a price change has altered what you should be running.

The fields teams forget

Three omissions recur. Retries: most wrappers report usage for the successful attempt and discard the failed ones, so a case that timed out twice is billed three times and recorded once. Tool round trips: an agent re-sends the transcript every step, so input tokens grow super-linearly in the step count and a harness logging only the final call understates an expensive case by an order of magnitude. Cached versus uncached input, which routinely swings a comparison.

Fix the wrapper before widening the record — a wide record of untruthful numbers is worse than a narrow honest one. Force a case to retry and check the recorded input tokens roughly double; if not, the accounting is wrong at source.

Instrumenting the harness

The shape below is deliberately plain: a dataclass for the per-case record, a runner that wraps any model callable, and nothing that knows about a specific provider. The call_model function is the only thing you adapt — any provider will do, as long as it returns text plus a usage dictionary.

from dataclasses import dataclass
from typing import Any
import time


@dataclass
class CaseResult:
    case_id: str
    model: str
    prompt_version: str
    expected: Any
    actual: Any
    quality: float                    # 0.0-1.0, from your grader
    input_tokens: int = 0             # total, INCLUDING cached
    cached_input_tokens: int = 0
    output_tokens: int = 0
    retries: int = 0
    tool_calls: int = 0
    latency_ms: int = 0               # wall clock, retries included
    ttft_ms: int | None = None
    error: str | None = None
    # Deliberately absent: any field denominated in money.


def run_case(case, model, prompt_version, call_model, grade, max_retries=2):
    """call_model(prompt, model) -> (text, usage)

    usage must carry cumulative counts for the WHOLE case:
    input_tokens / cached_input_tokens / output_tokens / tool_calls / ttft_ms.
    Wrap any provider to satisfy that contract and the rest is portable.
    """
    retries, text, usage, err = 0, None, {}, None
    started = time.perf_counter()

    while True:
        try:
            text, usage = call_model(case["prompt"], model)
            break
        except Exception as exc:
            retries += 1
            if retries > max_retries:
                err = f"{type(exc).__name__}: {exc}"
                break
            time.sleep(2 ** retries)          # billed, and counted below

    return CaseResult(
        case_id=case["id"],
        model=model,
        prompt_version=prompt_version,
        expected=case.get("expected"),
        actual=text,
        quality=grade(case, text) if text is not None else 0.0,
        input_tokens=usage.get("input_tokens", 0),
        cached_input_tokens=usage.get("cached_input_tokens", 0),
        output_tokens=usage.get("output_tokens", 0),
        retries=retries,
        tool_calls=usage.get("tool_calls", 0),
        latency_ms=int((time.perf_counter() - started) * 1000),
        ttft_ms=usage.get("ttft_ms"),
        error=err,
    )


def run_suite(cases, model, prompt_version, call_model, grade):
    return [run_case(c, model, prompt_version, call_model, grade)
            for c in cases]

Two details do real work. The retry counter survives into the record, so a run polluted by rate limiting is visible rather than merely expensive. And latency_ms wraps the entire loop, so a case that succeeded on the third attempt reports the time a user would have waited. One gap to close before you rely on this: when every attempt raises, no usage object comes back, so the case lands with zero tokens and zero cost despite having been billed several times over. Capture usage from the failed attempts in your provider wrapper, or the traps below will bite you in your own harness. Write results to JSON Lines, one object per case, and keep every run.

Turning tokens into money at report time

Money enters in exactly one file, which the runner never imports. That separation is what makes the design durable: the code gathering evidence has no opinion about prices, and the code with opinions about prices touches no evidence.

The table below uses illustrative placeholder figures, not real vendor rates. This guide was written in August 2026; check current provider pricing rather than relying on any number here. Tier names are generic on purpose: the structure is the point and real rates move. Substitute your own contracted rates.

Note the shape rather than the digits. Output is priced well above input at every tier, which is why a verbose configuration is expensive even when it is not obviously worse. And cached input sits well below fresh input — an order of magnitude in this illustrative table, though real providers' cache discounts vary widely, so check your own. It is the detail that most often flips a comparison. A retrieval-augmented workload with a large stable system prompt might have eighty per cent of its input served from cache; price that at the uncached rate and you reject the cheapest configuration on a number several times too large. Our guide to prompt caching across Claude, GPT and Gemini covers how to make the cache actually hit; here, the eval need only account for the two separately.

One token count, two price tables

Here storing tokens pays a second dividend. Teams in London and Bengaluru measuring the same system produce identical CaseResult rows — same tokens, same latencies, same quality scores — and read them through different price tables. Neither re-runs anything, and neither converts the other's currency figure at a rate accurate on some unrecorded date.

# prices.py — the only module in the codebase that knows about money.
# Rates are per 1,000,000 tokens, in the currency of the table.
# ILLUSTRATIVE placeholder figures, not real vendor rates. Replace with
# your own contracted rates and re-date this comment whenever you do.

PRICES_GBP = {
    "frontier-large": {"input": 12.00, "cached": 1.20, "output": 60.00},
    "mid-standard":   {"input":  2.40, "cached": 0.24, "output": 12.00},
    "small-fast":     {"input":  0.60, "cached": 0.06, "output":  2.40},
}

# NOTE: for brevity these INR figures are a flat conversion of the GBP ones.
# In a real codebase, do not do this — hold each entity's own contracted rates,
# because they do not move together with the exchange rate. See the text below.
PRICES_INR = {
    "frontier-large": {"input": 1290.0, "cached": 129.0, "output": 6450.0},
    "mid-standard":   {"input":  258.0, "cached":  25.8, "output": 1290.0},
    "small-fast":     {"input":   64.5, "cached":   6.5, "output":  258.0},
}


def case_cost(r, prices):
    p = prices[r.model]
    uncached = max(r.input_tokens - r.cached_input_tokens, 0)
    return (
        uncached              * p["input"]  / 1_000_000
        + r.cached_input_tokens * p["cached"] / 1_000_000
        + r.output_tokens       * p["output"] / 1_000_000
    )


def suite_report(results, prices, per_n=1000):
    n = len(results)
    lat = sorted(r.latency_ms for r in results)
    total = sum(case_cost(r, prices) for r in results)
    return {
        "cases":            n,
        "mean_quality":     sum(r.quality for r in results) / n,
        "p10_quality":      sorted(r.quality for r in results)[int(0.10 * (n - 1))],
        "cost_per_n_cases": total / n * per_n,
        "p50_latency_ms":   lat[int(0.50 * (n - 1))],
        "p95_latency_ms":   lat[int(0.95 * (n - 1))],
        "retry_rate":       sum(1 for r in results if r.retries) / n,
    }

One caution before trusting the arithmetic: some APIs report cached tokens as a subset of the input count, others alongside it as a separate figure. The function above assumes the former and subtracts. Verify your provider's convention on a deliberately cache-heavy call and normalise inside the wrapper, so input_tokens always means the total. Getting this backwards charges your cheapest tokens at your dearest rate.

Keep one table per billing entity, not one table and a conversion. Contracted rates, regional list prices and tax treatment do not move in lockstep with the exchange rate, so a UK subsidiary and an Indian one may face genuinely different economics for identical work. The token counts underneath stay identical — one measurement, two commercial readings.

Reading the frontier

With three numbers per case, the report becomes a scatter plot: quality on one axis, cost per thousand cases on the other, one point per configuration. A configuration dominates another when it is at least as good on every axis and strictly better on one. Dominated points are deleted — nobody should run something both worse and dearer, yet teams do, because without this plot nobody noticed. What remains is the Pareto frontier: the genuine trade-offs.

A frontier is a set and you need one answer. The rule that produces it is mechanical, and belongs in writing before anyone sees a price:

1. Set the quality floor first — the minimum acceptable score for this workload, taken from the product requirement, not from what the models achieved.
2. Discard everything below it, however cheap or fast. Not good enough is not a candidate at any price.
3. Among the survivors, choose the cheapest. Quality above the floor is not a tie-breaker: you decided the floor was sufficient, so paying to exceed it buys something you said you did not need.
4. Revisit the floor only when nothing clears it. That is a genuine finding. Lowering it because the winner is unglamorous is not.

Setting the floor before seeing results stops the exercise degenerating into justification. It is much harder to argue 0.82 is the requirement once you know only the flagship reaches 0.83.

The table below is an illustrative worked example with invented figures, not measured benchmark data. It shows the shape of a result, and makes no claim about any real model. Quality floor for this hypothetical workload: 0.82.

The fourth row is what justifies building any of this. Prompt work on the small model bought thirteen points of quality for eighty pence per thousand cases, clearing the floor at a thirteenth of the flagship's cost and a quarter of its latency. A leaderboard would have ranked the frontier model first and the conversation would have ended there. Note too that the same model appears in rows three and four with opposite verdicts — the clearest demonstration that "which model" is the less interesting half of the question.

The INR column is a straight conversion of the GBP column for illustration; in practice each entity reads the same token counts through its own contracted table, and the two can legitimately tell different stories.

Latency as the third axis

Cost and quality get the attention, but for interactive workloads latency prunes the frontier first, and it does so absolutely rather than gradually. If the requirement is that a suggestion appears within two seconds, a configuration with a p95 of seven seconds is not expensive, it is disqualified — exactly as one below the quality floor is. Apply latency as a second floor in step two of the decision rule, not as a factor traded against the others.

This is where the interactive-versus-batch distinction earns its keep. For a chat assistant, a code completion or an in-product search, latency is a hard constraint and the frontier is often cut to two or three viable points before price is considered at all. For an overnight batch — enriching a catalogue, summarising a week of support tickets, embedding a corpus — latency is nearly free, and configurations that are unusably slow online become perfectly reasonable. A single global model choice for an organisation running both shapes is almost always wrong in one direction or the other.

Use percentiles, never averages. An average latency number is useless for a decision because the distribution of response times is not symmetrical: it has a long right tail driven by retries, long generations and provider-side variance, and the mean sits comfortably below what a meaningful fraction of users experience. A configuration averaging 1.8 seconds with a p95 of nine seconds feels broken to one user in twenty, and nothing in the mean warns you. Report p50 and p95 side by side; the gap is itself diagnostic, usually pointing at retries or at a few unusually long generations, both visible in the fields you are now recording.

One caution on conditions: latency measured from a laptop in Chennai or Manchester against a provider region on another continent includes network round trips production may not incur. Run the harness from somewhere whose network position resembles production, and record the region.

Four traps that make a cost-aware eval lie

All four produce plausible-looking output, which is what makes them dangerous. A harness that fails loudly costs an afternoon; one that quietly names the wrong winner costs a quarter.

Quality scores that are not comparable across models. The method rests on a 0.87 from one configuration meaning the same as a 0.87 from another, and model judges break that in two ways. They reward verbosity, so a chattier model scores higher on the same substance while also costing more, corrupting both axes at once and in the same direction. And they exhibit self-preference, so a judge from a candidate's own family flatters it. Prefer deterministic assertions wherever the task admits them, never judge a model with its own family, and calibrate any model judge against human labels first — our guide to calibrating an LLM judge against human agreement covers how, and how often to re-check.

Cost measured on a case mix that does not resemble production. Suites drift towards interesting cases, because interesting cases are what people add when something goes wrong; production traffic is mostly boring. If your suite is sixty per cent hard cases and production is ten per cent, your cost figures are inflated unevenly, because small models degrade faster on hard cases — biasing the comparison towards the expensive option. Stratify the suite to match production or weight the report by it, and record the assumed mix.

Ignoring retries and tool round trips. This is where agent cost actually lives. A median agent case might make three model calls; the ninety-fifth-percentile case might make fifteen, re-sending a growing transcript each time. Averaged into a suite total, that tail disappears. Report the distribution of tool_calls and retries, not their means, and cost your most expensive decile separately — in agent workloads it frequently accounts for the majority of spend, and it is the only part worth optimising.

Benchmarking on a warm cache and deploying cold. If your eval runs the same system prompt across two hundred consecutive cases, every case after the first gets a cache hit production may never see, and the measured cost is a fraction of the real one. The reverse error is as common: benchmarking cold and rejecting a configuration production would run warm. Only one regime matches your deployment — decide which, and record it as a field on the run.

Wiring it into CI

The final step is to make this a property of every change. If you already run evals on pull requests — and if not, start with our guide to putting evals in CI for prompt and agent regression testing — the extension is small: report the cost delta alongside the quality delta, and set a budget regression threshold exactly as you set a quality one.

The asymmetry to avoid is a pipeline that blocks a two-point quality drop while waving through a forty per cent cost increase. Both are regressions, and both belong in the same comment on the same pull request, before the change merges.

# .github/workflows/evals.yml (excerpt)
- name: Run cost-aware eval suite
  run: |
    python -m evals.run \
      --suite golden \
      --model ${{ matrix.model }} \
      --prompt-version ${{ github.sha }} \
      --out results.jsonl

- name: Compare against baseline
  run: |
    python -m evals.compare \
      --baseline artifacts/main/results.jsonl \
      --candidate results.jsonl \
      --prices gbp \
      --quality-floor 0.82 \
      --max-quality-drop 0.02 \
      --max-cost-increase-pct 10 \
      --max-p95-latency-ms 4000 \
      --comment-on-pr

# Emitted as a PR comment (illustrative figures, continuing the example above):
#   quality   0.871 -> 0.869   (-0.002)   ok
#   cost/1k   £3.10 -> £4.55   (+46.8%)   FAIL  budget regression
#   p95       1,600 -> 1,720ms (+7.5%)    ok
#   retries   0.4% -> 0.4%                ok

Three notes. Express the cost threshold as a percentage rather than an absolute, so it survives suite growth. Make the failure informative — the comment above tells a reviewer exactly which axis moved and by how much, which is the difference between a gate people respect and one they override. And compare against a stored baseline rather than re-running it: a suite that is expensive to run gets run less often, and one that runs less often stops catching anything.

Key takeaways

The change is small and the payoff is structural: widen the per-case record from one number to three, keep money out of the runner, and set a floor before you look at prices.

• Emit quality, cost and latency for every case, in the same row. Summarising at run time destroys information.
• Store tokens, not currency — input, cached input and output separately, plus retries and tool round trips.
• Hold prices in one dated file the runner never imports, one table per billing entity, so London and Bengaluru read the same measurements at their own rates.
• Delete dominated configurations, apply the floor, take the cheapest survivor. Quality above the floor is not a tie-breaker.
• Treat p95 latency as a second floor for interactive workloads, and use percentiles rather than averages throughout.
• Gate cost regressions in CI beside quality ones, at a threshold loose enough to be respected.

If you do one thing this week, add the token fields to your result record and start writing them. You do not need the frontier, the price table or the CI gate yet — you need the measurements, because they are the only part you cannot reconstruct later. Whatever rates you eventually put in that price file will need re-verifying long before your token counts do.