Calibrate Your LLM Judge Against Humans Before You Trust It

Why an uncalibrated judge lies with confidence

An LLM judge produces a number for every output your system generates, at a cost of a fraction of a penny per verdict. That is the whole appeal: a handful of careful human judgements, scaled to volumes no review team could touch. But the number is only worth something if it predicts what a competent human would have said — and most judges in production have never been tested on that question even once.

The failure mode is quiet. An uncalibrated judge does not error out or return garbage; it returns plausible scores with a stable distribution, dashboards trend gently upward, and everyone relaxes. Meanwhile the judge may be systematically rewarding length over correctness, favouring whichever answer it read first, or scoring its own model family's prose more generously than a neutral rater would. Each of these biases is documented in the literature with numbers attached, and each produces a score that looks like measurement while being something else.

The scale of the gap is now quantified. A June 2026 study — the largest systematic evaluation of judge models to date, covering 21 judges from nine providers across roughly 541,000 individual judgements (arXiv 2606.19544) — found that raw exact-match agreement overstated chance-corrected agreement by 33 to 41 percentage points on MT-Bench data. Not for the weak judges. For all 21 of them. Teams reading raw agreement were overestimating their judges by a third or more, universally.

The fix is not a better prompt. It is a measurement discipline with three parts: a human baseline you can defend, an agreement statistic that does not flatter you, and a loop that re-runs both until the judge earns its place in the pipeline. We covered how to design a judge — pointwise versus pairwise, rubric templates, output schemas — in building a reliable LLM-as-a-judge. This guide is the deep end of the part that makes the design trustworthy.

Know the biases before you measure

Calibration will surface disagreement between your judge and your humans. To act on that disagreement you need to recognise the shapes it takes, because judge failures are not random — they are specific, repeatable, and well catalogued since the original LLM-as-a-judge work by Zheng et al. on MT-Bench and Chatbot Arena (arXiv 2306.05685).

Position bias

In pairwise comparison, judges systematically favour an answer based on where it appears, largely independent of content. Wang et al. demonstrated how exploitable this is in Large Language Models are not Fair Evaluators (arXiv 2305.17926): simply by controlling the order in which candidate answers appeared, they could make Vicuna-13B "beat" ChatGPT on 66 of 80 test queries — with ChatGPT itself as the evaluator. The symptom in your data: a pairwise verdict that flips when you swap the order of the two candidates. If the verdict depends on the seating plan, it is not a verdict.

Length and verbosity bias

Longer, more elaborate answers score higher even when they are no more correct. Zheng et al. quantified the severity with a "repetitive list" attack — padding an answer with rephrased duplicate content: the attack fooled Claude-v1 and GPT-3.5 judges in over 90% of cases, while GPT-4 was fooled 8.7% of the time. Better judges resist it better, but none are immune, and the insidious part is that longer answers often genuinely are slightly better, so the correlation is easy to rationalise. The symptom: your judge's scores correlate more tightly with response token count than with your human labels.

Self-preference and family bias

Judges score text produced by their own model or model family more highly than neutral raters do. Zheng et al. observed the effect in their early data — a roughly 10% inflated win rate when GPT-4 judged its own outputs, around 25% for Claude-v1, though they flagged the sample as limited. Later work sharpened the mechanism: Panickssery, Bowman and Feng showed that LLM evaluators can recognise their own generations at non-trivial accuracy, and that self-recognition ability correlates linearly with the strength of self-preference (arXiv 2404.13076). The symptom: a suspiciously favourable gap for one provider's outputs that your human panel does not reproduce. The implication for calibration is direct — if your judge shares a family with the system under test, part of your measured "quality" is family loyalty.

Build the human baseline

The baseline is a set of production outputs that humans have labelled carefully enough that you are willing to treat the labels as ground truth. Everything downstream inherits its quality, so this is where the discipline pays.

Sample 100–300 outputs from production, stratified

Pull real outputs from your production logs, not synthetic test cases — the judge will run on production traffic, so it must be calibrated on production traffic. One to three hundred items is the working range. Below 100, the confidence interval on any agreement statistic is too wide to act on, and your rare classes may land two or three examples. Above 300, each additional label buys less than a fresh sample next quarter would.

Stratify the sample rather than drawing it uniformly. If your system fails 5% of the time, a uniform sample of 200 gives you ten failures — nowhere near enough to measure how the judge handles the class you care most about. Over-sample suspected failures (low user ratings, escalations, retries, long silences after a response) so that each class you intend to measure has at least 25 to 30 items. Record the true production rates so you can reweight later if you need a population-level estimate. If you already run structured error analysis, that pipeline is the natural source — the method in error analysis and evals from production logs produces exactly the stratified pool this step consumes.

Adjudication beats majority vote

Have two reviewers label every item independently against the same rubric the judge uses. Where they agree, the label stands. Where they disagree, do not average, and do not simply add a third vote — adjudicate: the two reviewers (or a senior third) discuss the item, identify why they diverged, and record both the final label and the reason for the divergence.

Majority voting is cheaper and it quietly destroys the most valuable information in the exercise. A 2–1 split resolved by vote gets filed as if it were a clean label, when what actually happened is that your rubric failed to determine an answer. Adjudication converts every disagreement into a rubric finding. In practice, a UK fintech team labelling support-bot transcripts and a Bengaluru team labelling contract-summary outputs will hit the same wall: about 10 to 20% of items generate genuine expert disagreement on the first pass, and nearly all of it traces to an underspecified criterion, not to reviewer carelessness.

Track the criteria that generate disagreement

Keep a per-criterion tally of adjudicated items. If "completeness" produced two disagreements out of forty and "appropriate tone" produced eleven, you have learnt something important before the judge enters the picture: humans cannot apply your tone criterion consistently, so no judge can be calibrated against it. The options are to sharpen the criterion with anchored examples, split it into narrower sub-criteria, or drop it from automated judging and leave it to humans. What you must not do is ship a judge for a criterion your own experts cannot agree on — the judge's kappa against an unstable baseline is noise dressed as measurement.

Measure agreement: kappa, not accuracy — and per class, not in aggregate

With the baseline in hand, run the judge over the same items and compare. The obvious metric — the percentage of items where judge and human chose the same label — is the one that lies. Some of that agreement happens by chance, and the more skewed your label distribution, the more you get for free. A judge that answers "pass" every time scores 85% raw agreement on a set that is 85% passes, while carrying zero information.

Cohen's kappa corrects for exactly this: kappa = (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e is the agreement expected by chance given each rater's label frequencies. The 33-to-41-point deflation reported by the Reliability without Validity study is the difference between these two numbers measured in the wild, across every judge tested. If you report one agreement number, report kappa.

Aggregate numbers hide dead classes

Kappa alone is still not enough, because a single aggregate can look respectable while one class is broken. Judges collapse on minority classes constantly — the class with 15 examples is precisely the class the judge has the least signal on, and usually the class your business cares about most. So stratify: alongside overall kappa, compute per-class agreement from the confusion matrix and set a floor for every class you intend to trust the judge on.

Here is a complete working example — the agreement report we run after every calibration pass. It needs only scikit-learn:

from sklearn.metrics import cohen_kappa_score, confusion_matrix

LABELS = ["pass", "borderline", "fail"]

def agreement_report(human, judge, labels=LABELS):
    """Aggregate and per-class agreement between human and judge labels."""
    raw = sum(h == j for h, j in zip(human, judge)) / len(human)
    kappa = cohen_kappa_score(human, judge, labels=labels)
    print(f"n={len(human)}  raw agreement={raw:.2f}  Cohen's kappa={kappa:.2f}")

    cm = confusion_matrix(human, judge, labels=labels)
    for i, label in enumerate(labels):
        support = cm[i].sum()
        agree = cm[i, i] / support if support else float("nan")
        print(f"  {label:<10} n={support:<4} judge agrees {agree:.0%}")

# Worked example: 200 adjudicated items. In real use, load these two
# lists from your baseline file and the judge's outputs.
human, judge = [], []
def add(h, j, n):
    human.extend([h] * n)
    judge.extend([j] * n)

add("pass", "pass", 156)          # judge agrees on most passes
add("pass", "borderline", 4)
add("borderline", "borderline", 15)
add("borderline", "pass", 10)     # upgrades 10 borderlines to pass
add("fail", "fail", 5)
add("fail", "pass", 10)           # waves through 10 of 15 real failures

agreement_report(human, judge)
# n=200  raw agreement=0.88  Cohen's kappa=0.57
#   pass       n=160  judge agrees 98%
#   borderline n=25   judge agrees 60%
#   fail       n=15   judge agrees 33%

Read the three layers of that output in order. Raw agreement is 0.88 — a number most teams would put on a slide. Kappa is 0.57 — moderate, and a materially different story once chance agreement on the dominant class is stripped out. And the per-class rows show where the damage lives: the judge waves through two-thirds of real failures. A production gate built on this judge would pass 88% of the time and still be blind to the one class it exists to catch. This is not a contrived pattern; it is the default shape of a judge calibrated by vibes, and only the stratified view exposes it.

The calibration loop

Calibration is iterative. The first measurement almost never clears the bar, and the point of the loop is to converge in a small number of disciplined passes rather than to fiddle indefinitely.

Revise, re-run, re-measure

1. Diagnose from the disagreements. Pull every item where judge and human diverged, grouped by class and criterion. Read them. Patterns emerge fast: the judge reads "polite refusal" as failure; it rewards a confident wrong answer over a hedged right one; it never uses the borderline label at all.
2. Revise the rubric, not the labels. Fix the judge prompt with anchored examples taken from the disagreement pile — one exemplar per failure pattern is worth paragraphs of abstract instruction. Resist the urge to "correct" human labels to match the judge; the baseline moves only through re-adjudication, with the change logged.
3. Re-run the judge on the full baseline, not just the items it got wrong. A rubric change that fixes the fail class can silently break the pass class.
4. Re-measure with the same report. Kappa overall, agreement per class, and — for length bias — the correlation between judge score and response token count.

Stopping criteria

Decide the bar before the first measurement, in writing, so the number cannot negotiate with you. A defensible working standard: overall kappa at or above 0.6, no trusted class below 0.4, and no single class carrying the aggregate. Then stop when one of three things happens:

• You clear the bar on two consecutive passes. One pass can be luck on 200 items.
• You plateau. If two successive rubric revisions move kappa by less than 0.05, the remaining disagreement is probably irreducible ambiguity, not prompt quality. Either accept the level and constrain what the judge gates, or split the troublesome criterion out for human review.
• You hit the iteration budget. Three to four full passes is the sensible maximum. Beyond that you are overfitting the rubric to this specific baseline — the judge equivalent of tuning on your test set — and the honest next step is a fresh sample.

And calibration expires. Re-measure whenever the judge model changes, the judge prompt changes, the system under test changes meaningfully, or the input distribution drifts. Absent any of those, refresh quarterly with 100 to 150 fresh adjudicated labels. Every judge verdict a human overturns in production goes into the next baseline with its corrected label — over a few quarters this grows into a hard, representative regression set for free.

Mitigations, bias by bias

Calibration tells you how wrong the judge is; mitigation is how you close the gap. Match the fix to the bias the disagreement analysis surfaced:

Judge biases and the mitigations that measurably reduce them.

Bias	Symptom in your calibration data	Mitigation
Position bias	Pairwise verdicts flip when candidate order is swapped	Run every pairwise judgement in both orderings; accept only verdicts that survive the swap, record the rest as ties. For gates, prefer pointwise scoring, which has no order to exploit.
Length / verbosity bias	Judge scores correlate with token count more than with human labels	Normalise for verbosity: score concision as its own criterion so padding is penalised on a separate axis; instruct the judge explicitly not to reward length; monitor the score–length correlation as a standing metric on every calibration pass.
Self-preference / family bias	One provider's outputs outperform with the judge but not with humans	Use a judge from a different model family to the system under test; blind the judge to provenance (strip model names, formatting tells); for high-stakes gates, use a panel of two or three judges from different families and take majority or mean.
Leniency / central tendency	Judge never uses the extreme labels; everything clusters mid-scale	Anchor every scale point with a concrete example from your adjudicated set; where it fits, use an even-numbered scale so there is no comfortable middle.

The both-orderings rule and the different-family rule are the two with the strongest published evidence behind them — the order-swap manipulation and the self-recognition correlation are not edge cases, they are headline results. Treat those two as non-negotiable defaults rather than options.

What calibration costs

The honest objection to all of this is cost, so here are the numbers. Assume a typical judged item consumes about 1,500 input tokens (rubric plus the output under evaluation) and 300 output tokens (structured verdict plus rationale). Human labelling assumes two reviewers at roughly three minutes per item plus adjudication overhead — about 25 person-hours for a 200-item double-labelled, adjudicated baseline.

Judge-call prices below are first-party list rates as of August 2026 (Anthropic's published pricing; OpenAI's via its pricing page — check both before budgeting, as these move):

Calibration and running costs at August 2026 list prices (USD per million tokens: Claude Haiku 4.5 $1/$5, Claude Sonnet 4.6 $3/$15, Claude Opus 5 $5/$25, GPT-5 mini $0.25/$2).

Line item	Human effort	GPT-5 mini	Claude Haiku 4.5	Claude Sonnet 4.6	Claude Opus 5
Initial baseline, 200 items (judge run once)	~25 hours	$0.20	$0.60	$1.80	$3.00
Calibration loop, 3 further passes over the baseline	~4 hours (disagreement review)	$0.59	$1.80	$5.40	$9.00
Quarterly refresh, 150 fresh items	~19 hours	$0.15	$0.45	$1.35	$2.25
Production judging, 50,000 outputs per month	—	$48.75	$150	$450	$750

Two things jump out of that table. First, the judge-call cost of calibration is a rounding error — a few dollars against tens of hours of expert time. The constraint is always the humans, which is why the sampling and adjudication discipline matters: it is the expensive part, so waste none of it. Second, the human cost is front-loaded while the judge cost recurs monthly, and the ratio between judge tiers is 15× — which is exactly why the calibration loop should also tell you whether the cheap judge is good enough. If GPT-5 mini or Haiku 4.5 clears your kappa bar, the stratified report is the evidence that lets you bank the saving; if only a frontier judge clears it, the same report is the justification for the spend. Either way, for a team in London or Chennai, roughly 25 person-hours a quarter buys the ability to trust — or correctly distrust — every one of the 150,000 automated verdicts that quarter.

One caution on the production line: a calibrated judge embedded in an agent pipeline is also an attack surface. Systems under evaluation can learn to game a grader they can observe — the UK AI Security Institute found every frontier model it tested attempted to cheat on its evaluations — so keep the judge's rubric and the calibration set out of the evaluated system's reach.

Start with one criterion this week

The methodology compresses to a checklist. Sample 100 to 300 production outputs, stratified so failures are represented. Double-label and adjudicate; log which criteria generate expert disagreement and fix those first. Run the judge, and score it with kappa plus a per-class breakdown — never raw accuracy alone. Loop on the rubric with written stopping criteria, mitigate the specific biases your disagreements reveal, and put the refresh on the calendar.

None of this depends on which models are current. Position bias, verbosity bias and self-preference have persisted across every judge generation studied since 2023; chance-corrected agreement was the right statistic before language models existed. Models will keep changing underneath you — the discipline of checking the machine against the humans it claims to replace is the part that lasts. If you build one thing this week, build the 200-item baseline for your single most important criterion. The golden-set plumbing in assembling an LLM evaluation suite is a good scaffold to hang it on, and the first agreement report will tell you more about your judge than any benchmark ever will.