When a Second Model Checking the First Pays for Itself

What you need to work out first

There is a familiar sequence in production LLM work. Something goes wrong, someone reads a paper about self-consistency or debate or LLM-as-a-judge, and by the end of the fortnight there is a critic pass wired into the hot path. The accuracy numbers on the internal eval move a little. The bill moves a lot. Nobody ever revisits it, because unwiring a safety measure feels like an admission.

The mistake is not adding verification. It is adding it without an answer to the only question that matters: does this pattern, on this route, remove more expected loss than it adds in cost and delay? That question has a shape, and the shape does not change when the models change. Four things go into it, and you can measure all four on your own traffic in an afternoon:

• How often the unverified system is wrong in a way that reaches the user or the downstream system. Not benchmark accuracy — the observed rate on the route you are considering.
• How much a single escaped error actually costs you. This is the term teams never quantify, and it is the term that decides the answer. Everything else is small by comparison.
• What the verification pattern adds in tokens, in wall-clock latency, and in engineering surface.
• What it breaks. Verifiers overturn correct answers. If you do not measure that, your net accuracy change can be negative while your caught-error dashboard looks superb.

Work those out and the decision usually makes itself, often in a direction that surprises people. A wrong answer in a support summary costs pennies. A wrong answer in a payments reconciliation at a Bengaluru fintech, or in a claims triage note at a Leeds insurer, costs a very great deal more. Same model, same prompt structure, opposite conclusion about whether a verifier belongs in the loop.

The equation that actually decides it

Write the whole thing down once and it stops being a matter of opinion. Per output, verification is worth adding when:

expected_benefit  >  expected_harm  +  added_cost

where
  expected_benefit = catch_rate      x  p_bad        x  cost_escaped_error
  expected_harm    = overturn_rate   x  (1 - p_bad)  x  cost_wrong_overturn
  added_cost       = verifier_token_cost  +  (latency_added x latency_price)

Each symbol is something you can measure rather than argue about. p_bad is the share of outputs that are wrong in a way you care about. catch_rate is the share of those wrong outputs the verifier flags. overturn_rate is the share of correct outputs the verifier wrongly flags or changes. cost_escaped_error and cost_wrong_overturn are money. latency_price is money per second of added delay, which is uncomfortable to estimate but not impossible — for an interactive product it falls out of your own funnel data, and the mechanics of pricing delay are covered in our guide to latency budgets for chat UX.

Notice what the equation does not contain: any claim about which model is best at critique, any benchmark delta from a paper, any notion of sophistication. It contains one term that is usually enormous and four that are usually small. That asymmetry is the whole game.

Pricing an escaped error, honestly

Most teams behave as though cost_escaped_error is unknowable, and by treating it as unknowable they implicitly set it to zero — which is why so many verification decisions get made on vibes. It is estimable, and the estimate does not need to be good. It needs to be within an order of magnitude, because that is the resolution at which the decision flips.

The practical method: take the last twenty incidents traceable to a wrong model output and cost each one out. Staff minutes at a loaded rate. Any credit, refund or goodwill gesture. Rework downstream, including the human who had to redo the work properly. Where it applies, the regulatory or contractual exposure — a mis-stated figure in a regulated communication is not the same animal as a clumsy paragraph. Average the twenty. Then, crucially, split the average by route, because it is not one number. The same system serving a customer-support summariser and a reconciliation agent has two escaped-error costs separated by three or four orders of magnitude, and using a blended figure will over-verify the cheap route and under-verify the expensive one simultaneously. If you already track spend per task through the approach in LLM unit economics: cost per task and margin, this is the same discipline pointed at the loss side of the ledger.

The pattern ladder, cheapest first

Verification is not one thing. It is a ladder of seven rungs with wildly different cost profiles, and the useful discipline is to know what each rung actually buys before you climb it. The table below orders them by cost per verified output, cheapest first.

Rungs (d) and (g) have real research behind them as mechanisms. Self-consistency was introduced by Wang and colleagues as a decoding strategy that samples several reasoning paths and takes the most consistent answer instead of decoding greedily (arXiv 2203.11171). Multi-agent debate, from Du and colleagues, has several model instances propose and debate their answers and reasoning over multiple rounds before converging on a shared final answer (arXiv 2305.14325). Both are sound mechanisms. Neither comes with a percentage improvement you can transfer to your workload, and any figure you have seen quoted was measured on a benchmark that is not your product. Use the mechanism; measure the delta yourself.

Order matters more than sophistication

Here is the finding that repeats across almost every production system I have looked at, and it is worth more than any model choice: the deterministic rungs catch a surprising share of the failures that actually reach users, and most teams reach for (e) or (g) before they have finished (a) and (b).

The reason is psychological rather than technical. Rungs (a) and (b) are unglamorous. Writing a validator for your output schema, or an assertion that the line items sum to the stated total, or a resolver that confirms every cited invoice number exists in the ledger — none of that feels like AI engineering. Wiring up a critic model does. So the boring layer gets skipped, and the expensive layer inherits work it is bad at. A model-based critic asked to check arithmetic is strictly worse than the arithmetic: slower, dearer, and occasionally wrong in a way that a subtraction cannot be.

Do the boring work first. Constrain the output shape at generation time using the techniques in structured output prompting patterns in production so that rung (a) has very little left to catch. Then write assertions for every invariant you can name. Then — and only then — measure what still gets through, and let that residue tell you which model-based rung you need, if any. Teams that do this in order frequently discover they never reach rung (c). Teams that start at rung (e) never find out what the assertions would have caught for free, and they pay for that ignorance every single call, forever.

Error correlation: why self-critique buys less than it looks

The single most useful concept in this whole area is decorrelation, and it explains why two verification patterns that cost roughly the same can differ enormously in what they are worth.

A model critiquing its own output shares the machinery that produced the output. If it misread an ambiguous clause in your prompt, the critic pass reads the same clause the same way and endorses the result. If its training left it with a particular misconception about how Indian GST invoices are structured, or how a UK motor claim excess interacts with a protected no-claims discount, that misconception shows up in both the generation and the review. The critic is not an independent opinion; it is the same opinion, restated in the register of review.

What you actually want from a verifier is a check whose failure mode is unrelated to the generator's. There is a rough hierarchy of decorrelation, and it maps neatly onto value per token:

• Different modality of check — an assertion, a database lookup, a unit test, a recomputation. Maximally decorrelated: it fails for reasons that have nothing to do with language modelling at all. This is rung (b), and it is why rung (b) punches so far above its cost.
• Different model family — a verifier from a different lab, trained on a different mixture, with different failure tendencies. Meaningfully decorrelated, though not perfectly: models trained on overlapping web corpora share some blind spots, and no amount of vendor diversity fixes an ambiguity in your own specification.
• Different role, same model — the same model prompted as a critic. Weakly decorrelated. It catches slips and rubric violations, which are real, but it systematically misses the errors that came from a shared misreading.
• Same model, same role, sampled again — this is rung (d). It measures instability rather than correctness, which is genuinely useful information, but a confidently wrong model votes for its wrong answer every time.

Measuring correlation on your own traces

You do not have to take this on faith. Take a few hundred outputs from your own system, run two candidate verifiers over all of them, and build the two-by-two contingency table of who flagged what. If verifier A and verifier B flag almost exactly the same items, the second one is buying you nothing — you are paying twice for one opinion. If their flag sets overlap only partially, the union is genuinely larger than either alone, and that difference is exactly what decorrelation is worth in money.

This is the same measurement discipline that makes an LLM judge trustworthy in the first place. If you are going to lean on a model as a verifier, the practices in LLM-as-a-judge: rubrics, bias and calibration and calibrating your LLM judge against humans are not optional extras. An uncalibrated verifier is an unmeasured cost with a confident interface.

When verification makes things worse

Verification is usually discussed as though its worst case is wasted money. It is not. Its worst case is a system that is less accurate than it was before, more expensive, and slower — and that is a genuinely common outcome, because three failure modes are all easy to miss.

The critic overturns correct answers. A verifier asked "is this right?" several thousand times will find fault sometimes, because finding fault is what it has been asked to do. Every correct answer it overturns is a defect you manufactured. In a pipeline where flagging triggers regeneration, you have replaced a good answer with a fresh sample; in a pipeline where flagging routes to a human, you have burned a review that was never needed. This is why overturn_rate sits in the equation with the same weight as catch_rate, and why a dashboard that only counts caught errors is worse than no dashboard.

The loop does not terminate. Generate, critique, regenerate, critique again. Without a hard iteration cap the system can oscillate between two answers the critic dislikes in different ways, and every cycle is a full-price call. Cap the loop at two attempts, escalate on the third, and log every escalation — the escalation rate is one of the better early-warning signals you will have.

Latency destroys the product. In an interactive assistant, a serial critic pass can double time-to-answer. A verifier that improves accuracy by a modest amount while making the experience noticeably slower can be a straightforward net loss, and it will not show up in any accuracy metric you are watching. Offline and batch workloads have almost no latency price, which is a large part of why verification is so much easier to justify there.

Measuring it on your own traces

Everything above is theory until you have a labelled set. Building one is a day of work and it is the highest-return day in this entire exercise.

You need two halves. The known-bad half comes from incidents, from customer complaints, from human corrections in your review queue, from anything a person has already flagged as wrong. The known-good half comes from outputs a human has confirmed as correct — and it is the half everyone skips, because collecting confirmations of things that went right feels pointless. It is not pointless. Without it you cannot compute an overturn rate, and without an overturn rate you cannot tell an improvement from a regression. Aim for a couple of hundred of each to start, stratified so that the mix of routes and difficulty roughly matches production. If you need to industrialise the labelling, the workflow in building a human annotation pipeline for LLM evals is the version that scales.

Then run the measurement. It is a small amount of code and it produces four numbers that settle the argument.

from dataclasses import dataclass

@dataclass
class VerifierReport:
    catch_rate: float          # of known-bad outputs, share flagged
    overturn_rate: float       # of known-good outputs, share flagged
    net_accuracy_change: float # points of accuracy gained (or lost)
    cost_per_caught_error: float
    spend_per_overturn: float

def measure(verifier, known_bad, known_good, p_bad,
            cost_per_verification):
    """Score a verifier on a labelled set.

    known_bad / known_good are lists of production outputs a human
    has already labelled. p_bad is the observed error rate on live
    traffic for THIS route -- not the ratio inside the labelled set,
    which is deliberately unrepresentative.

    net_accuracy_change is an upper bound in both directions: it
    assumes every caught error is then actually fixed, and that
    every overturn destroys an answer that was correct. Neither
    holds in full, so treat it as a ceiling, not an outcome.
    """
    caught    = sum(1 for o in known_bad  if verifier.flags(o))
    overturned = sum(1 for o in known_good if verifier.flags(o))

    catch_rate    = caught / len(known_bad)
    overturn_rate = overturned / len(known_good)

    # The number that decides it: errors removed minus errors created.
    gained = catch_rate * p_bad
    lost   = overturn_rate * (1.0 - p_bad)
    net    = gained - lost

    # Cost is per verification run, so amortise over the whole route.
    # spend_per_overturn is that spend divided across overturn events --
    # not a cost incurred each time one happens.
    per_output = cost_per_verification
    cost_per_catch     = per_output / gained if gained else float("inf")
    spend_per_overturn = per_output / lost   if lost   else float("inf")

    return VerifierReport(catch_rate, overturn_rate, net,
                          cost_per_catch, spend_per_overturn)

def worth_it(report, cost_escaped_error, cost_wrong_overturn,
             p_bad, cost_per_verification,
             latency_added_s, latency_price_per_s):
    """Return expected value per output. Positive means ship it."""
    benefit = report.catch_rate    * p_bad         * cost_escaped_error
    harm    = report.overturn_rate * (1 - p_bad)   * cost_wrong_overturn
    spend   = cost_per_verification + latency_added_s * latency_price_per_s
    return benefit - harm - spend

Run measure() against every rung you are considering, from the schema validator upwards, and put the results in one table. The ordering is almost always instructive: the free rungs post modest catch rates at zero marginal cost, which makes their cost per caught error zero, which makes them unarguable. The expensive rungs have to justify themselves against a residue that is much smaller than anyone expected.

A worked break-even, with the assumptions on the table

The table below is entirely illustrative. Every figure in it is an assumption chosen to demonstrate the arithmetic, not a measurement of any real system, and no figure should be copied into your own model. Substitute your own numbers; the structure is what transfers.

Three routes, all using the same verification pattern — a cheap different-family verifier at rung (c) reading the same source material as the generator, adding roughly 3,500 tokens per output on routes B and C. The rung is the same on all three; the per-output cost is not. Route A runs a smaller verifier model over much shorter source material — a single support thread rather than a reconciliation batch or a claim file — which is why its added verification cost sits roughly an order of magnitude below the other two. Each column is quoted in the currency that team budgets in. We make no claim about the rate between rupees and pounds, and the columns are deliberately not converted into one another; if you want to compare them directly, redo them in one currency at whatever rate your own treasury uses.

Three lessons fall straight out of that table, and none of them are about the models.

First, column A fails on volume, not on quality. The verifier there is good — a 0.70 catch rate against a 0.04 overturn rate is a respectable verifier. It fails because two million cheap outputs multiplied by a small per-output cost is a large number, and the errors it prevents are worth almost nothing individually. Rearranging the equation gives the threshold directly: this route breaks even once an escaped error is worth about ₹5.66. So the real question for column A is not "is my error rate too high?" — it is "is one of these errors worth about six rupees?" If the answer is genuinely no, do not verify. Spend the money on making the generator better instead.

Second, column B is not a close call and should not be treated as one. When the net value per output is roughly seventy times the verification spend that produced it — ₹77.08 against ₹1.10, counted net rather than gross — the correct response is not satisfaction; it is to climb the ladder. At that ratio you can afford a different-family critic at rung (f), or self-consistency on top, and still be far ahead. Teams routinely under-verify their highest-stakes route because the same verification budget conversation is applied to every route equally.

Third, column C is where the interesting engineering is. It pays, but the overturn term eats roughly half the gross benefit. The highest-value work on that route is not a better model — it is reducing the overturn rate, by tightening the rubric, by having the verifier abstain when the evidence is thin rather than guess, or by routing flagged-but-uncertain cases to a human instead of automatically regenerating. Halving the overturn rate on column C is worth more than any plausible improvement in catch rate.

Where caching and batching move the break-even

The arithmetic above treats the verifier's token cost as fixed. It is not. Two provider mechanics can move it by a large multiple, and both are worth understanding precisely because they can flip a marginal decision.

The first is prompt caching. A verifier that reads the same long prefix as the generator — the same policy document, the same retrieved context, the same schema and rubric — is re-sending input the provider has already seen. Where the caching rules apply, that prefix is billed at a fraction of the normal input rate. The second is batching. Verification that does not need to be synchronous — nightly sampling, post-hoc audit, offline scoring of yesterday's traffic — can go through an asynchronous batch endpoint at a documented discount.

Because these are commercial terms rather than laws of nature, the table below states what each provider documents, with the date, and links to the page each row was read from. Re-check them at source before you rely on them.

The design consequence is specific. If you want the caching discount, the verifier's prompt must begin with the same bytes as something recently sent — so put the shared context first and the verification instruction last, not the other way round. That is a five-minute change with a large effect, and the placement rules are covered properly in prompt caching across Claude, GPT and Gemini. For the asynchronous half, the Batch API playbook covers the queue mechanics. And where the same inputs recur across users rather than within a single request, semantic caching is a third lever that can remove the verification call entirely for repeated work.

Rerun the break-even after these are in place, because they change the answer. In column A of the illustrative table, if caching and batching between them cut the added verifier cost from ₹0.12 to something near ₹0.02, the break-even escaped-error cost drops from about ₹5.66 to about ₹2.09 — and a route that clearly did not justify verification becomes genuinely marginal. That is the reason to do the pricing work before concluding that a pattern is unaffordable.

Route the verification, do not spray it

Almost every team that concludes "verification is too expensive" has reached that conclusion by imagining verification applied to one hundred per cent of traffic. That is almost never the right design. The right design is to spend the verification budget where it earns, and there are three well-worn ways to decide where that is.

Verify by confidence. Run the cheap signals first — schema validity, assertion failures, self-consistency across a couple of cheap samples, retrieval scores, token-level uncertainty where you have it — and send only the low-confidence tail to a model verifier. If a fifth of outputs are uncertain, you are paying for verification on a fifth of traffic while catching a disproportionate share of the errors, because errors concentrate in exactly that tail. This is the same shape as a model cascade, and the mechanics carry over directly from model routing and cascades.

Verify by stakes. Classify routes, not requests. The reconciliation agent gets a different-family critic; the support summariser gets schema validation and nothing else. This is the single highest-leverage move available, and it requires no new technology — only that someone writes down the escaped-error cost per route and acts on it.

Verify by sample, for monitoring. Even on routes where per-request verification does not pay, verify a small percentage continuously as an instrument. This is not a safety measure; it is a smoke alarm. A drifting error rate on a route you had written off as cheap is exactly the thing you want to learn about from a dashboard rather than from a customer. Run it through the batch queue at the discounted rate and the cost is negligible. Scoring this properly — quality against spend rather than quality alone — is the subject of cost-aware evals: quality per pound.

The decision ladder you can apply on Monday

Work down this list in order. Stop when you run out of justification; do not skip forward because a rung looks more interesting.

• 1. Is the output shape checkable? If yes, validate it. Always. There is no threshold to clear — rung (a) costs nothing and there is no scenario in which you should not do it.
• 2. Does the output contain anything with a deterministic invariant? Sums, dates, identifiers, ranges, references to records that either exist or do not. Write the assertions. Also free, also unconditional. If you are calling tools, the retry-and-repair patterns in making small models reliable at tool-calling belong here too.
• 3. Do you know what an escaped error costs on this route? If not, stop and find out. Everything below this line is unanswerable without that number, and guessing it as zero — which is what doing nothing amounts to — is itself a decision.
• 4. Do you have a labelled set with both known-bad and known-good outputs? If not, build it. Two hundred of each is enough to start. Without the known-good half you cannot measure overturn, and without overturn you cannot tell whether you are helping.
• 5. Run the equation. Negative? Do not verify; improve the generator, or spend the money elsewhere. Marginal? Apply the caching and batching mechanics and run it again — that alone flips a fair number of marginal cases.
• 6. Positive? Start at rung (c), the cheap verifier, and route it. Confidence-gated, or stakes-gated, or both. Measure catch rate and overturn rate in production, not just on the labelled set.
• 7. Still leaving significant expected loss on the table? Climb to rung (f) — a different-family critic — before rung (e), because decorrelation is worth more per token than familiarity. Self-critique is the rung to reach for when you have no second family available, not the default first step.
• 8. Consider rung (g) only when the expected benefit exceeds the cost by a wide margin and the failure is genuinely a reasoning dispute. Debate is expensive, serial, and can converge confidently on a wrong consensus. It is the last rung for a reason.

What makes this ladder durable is that none of it depends on which models are current. Models will get cheaper, which pushes the break-even down and makes more verification worthwhile. They will also get more accurate, which lowers p_bad and pushes the break-even back up. Those two forces have been pulling against each other for years and will keep doing so. The cost of an escaped error, meanwhile, barely moves — it is a property of your business, not of the model market. Which is precisely why it deserves to be the number you actually measure, and why a team that knows it will make better verification decisions in 2028 than a team that has read every paper and never priced a single incident.