Building a Reliable LLM-as-a-Judge: Bias and Calibration

What you need to know

• The judge is a system you build, not a prompt you paste. An LLM-as-a-judge that you have not designed and calibrated is just a second, unvalidated model whose opinions you happen to trust.
• Decide pointwise or pairwise first. Absolute scores gate releases; head-to-head comparisons rank changes. The choice shapes everything downstream, including which biases you have to fight.
• Write it as a rubric with explicit steps. The G-Eval pattern — chain-of-thought plus a form-filling schema — turns "rate the quality" into a repeatable procedure that two runs will agree on.
• Three biases are well documented. Position bias, verbosity or length bias and self-preference bias all have named mitigations: randomise order, length-normalise, and judge with a different model family.
• Calibrate against humans and report Cohen's kappa. Raw agreement flatters you; kappa corrects for chance. Do not let a judge gate anything until you know its kappa.
• Split a vague judge into narrow sub-judges. When one score is carrying five concerns, decompose it into a small graph of single-purpose judges you can each calibrate.

Almost every team building with language models arrives at the same wall. You have a handful of unit tests, maybe a golden set, and then a large space of outputs where "correct" is a matter of judgement — is this summary faithful, is this answer helpful, is this tone right for a customer in Manchester or Mumbai? Human review does not scale to every pull request, so you reach for an LLM-as-a-judge: a model that reads an output and scores it. Within a week it is quietly gating deploys. The uncomfortable truth is that most of these judges are never actually built. They are one improvised prompt that nobody has calibrated, running on a model that may be biased towards its own outputs, producing numbers that look authoritative and mean very little.

This guide is about constructing the judge properly. It is deliberately not about which retrieval metrics to track — if you want that, our walk-through of evaluating RAG with RAGAS covers faithfulness, context precision and recall. It is also not about wiring evals into your pipeline — for that, see putting your evals in CI. Both of those assume you already have a trustworthy judge. This piece is how you get one: the design choices, the biases, and the calibration loop that separates a judge you can stake a release on from a plausible-looking oracle you cannot. Note that the specific figures cited here are dated to sources published up to mid-2026; treat them as illustrative anchors, verify against the primary papers, and re-check as the field moves.

Pointwise or pairwise: choose the question first

The first design decision is what question the judge answers, and it is more consequential than the model you pick. A pointwise judge reads one response and scores it against an absolute rubric — "rate the faithfulness of this answer from 1 to 5". A pairwise judge reads two responses and picks a winner — "which of these two answers is more helpful, A or B?". These are not interchangeable styles of the same tool; they produce different data, suit different jobs, and expose you to different biases.

When pointwise wins

Reach for pointwise when you need an absolute bar. A release gate that says "block the deploy if median faithfulness drops below 4.0" needs a number on a fixed scale, and only a pointwise judge produces one. The same is true for a dashboard you watch week to week, or a regression threshold in CI. The catch is that absolute scales drift: the same judge, same prompt, run three months apart against a newer model version, can quietly recalibrate what "4" means. You defend against that by anchoring the rubric with concrete descriptions of each score point and, ideally, a couple of worked examples inside the prompt.

When pairwise wins

Reach for pairwise when the decision is inherently relative. Is prompt v2 better than v1? Is the cheaper model good enough to replace the expensive one? Humans are markedly more consistent when comparing two things than when scoring one in isolation, and LLM judges inherit that — pairwise verdicts tend to agree with humans more often. Pairwise is also how you generate preference pairs for reward modelling and preference tuning. The price you pay is position bias, which is severe enough that you must never trust a single-order pairwise verdict, and the combinatorial cost of comparing many options.

Write the judge like a rubric, not a vibe

A judge prompt that says "rate the quality of this answer from 1 to 10" is a coin flip with extra steps. The single most effective upgrade is to move from an implicit ask to an explicit procedure. This is the core idea behind G-Eval (Liu et al., arXiv 2303.16634), which pairs chain-of-thought reasoning with a form-filling paradigm: you give the model the criterion, have it produce or follow a set of explicit evaluation_steps, and then require it to fill in a strict output schema. On summarisation, G-Eval with a strong backbone reported a Spearman correlation of roughly 0.514 with human ratings — well ahead of the older n-gram metrics it replaced. The mechanism matters more than the exact number: naming the steps is what makes the judge repeatable.

The template

Here is a reusable pointwise judge in the G-Eval shape. Note the four moving parts: a tightly scoped criterion, numbered evaluation_steps the model must follow in order, an anchored score scale, and a strict JSON output schema that leaves no room for freeform prose.

You are a strict evaluator. Judge ONE response against the rubric below.
Do NOT reward length, confidence, or writing style — only the criterion.

## Task the response was answering
{task}

## Response to evaluate
{response}

## Reference / ground truth (may be empty)
{reference}

## Criterion: Faithfulness
Every factual claim in the response must be supported by the reference.
A claim that is not entailed by the reference is a hallucination.

## Evaluation steps (follow in order; think step by step)
1. Extract every distinct factual claim the response makes.
2. Label each claim: SUPPORTED, CONTRADICTED, or NOT_IN_REFERENCE.
3. Count unsupported claims = CONTRADICTED + NOT_IN_REFERENCE.
4. Map that count to the score scale below.

## Score scale (anchored)
5 = every claim supported
4 = exactly 1 minor unsupported claim, no contradictions
3 = 2-3 unsupported claims OR 1 contradiction
2 = several unsupported claims or contradictions
1 = mostly fabricated or contradicted

## Output — return ONLY this JSON, no prose before or after:
{
  "claims": [{"claim": "...", "label": "SUPPORTED|CONTRADICTED|NOT_IN_REFERENCE"}],
  "unsupported_count": <int>,
  "score": <1-5>,
  "reasoning": "<= 40 words"
}

Why the schema matters

The strict JSON schema is not decoration. It does three jobs. First, it forces the judge to show its working — the claims array is auditable, so when a human disagrees with a verdict you can see exactly which claim the judge misjudged rather than arguing with a bare number. Second, it makes the output parseable, so you can aggregate thousands of verdicts without brittle regex. Third, by demanding the intermediate unsupported_count before the final score, it enforces the chain-of-thought — the model has to do the counting the evaluation steps described, which is what pushes agreement up. If you want reliable structured output from the judge itself, our guide to running evals in CI pairs well with constrained decoding so the schema never breaks a pipeline. Keep the reasoning field short and last — a long free-text field before the score invites the model to talk itself into a different answer.

The biases every judge inherits — and how to kill them

LLM judges do not fail randomly. They fail in specific, repeatable, well-catalogued ways. Zheng et al.'s work on MT-Bench and Chatbot Arena (arXiv 2306.05685) — the paper that showed strong judges like GPT-4 can reach over 80% agreement with humans, roughly the level humans agree with each other — was equally clear-eyed about the failure modes, naming position, verbosity and self-enhancement bias explicitly. Later surveys, such as Gu et al.'s A Survey on LLM-as-a-Judge (arXiv 2411.15594), extend the catalogue further. The three below are the ones you must address before shipping.

Position bias

Position bias is the one that catches teams out first because it hides inside pairwise setups that otherwise look sound. The fix is cheap and non-negotiable: present the pair in a random order, and for anything you actually care about, run the judgement twice — once as (A, B) and once as (B, A). If the verdict flips when you swap the order, the judge is guessing, and you should record a tie rather than a spurious winner. A judge that only wins in one order has told you nothing about the answers and everything about its own bias. Pointwise judging sidesteps this entirely, which is one more reason to use it for gates.

Verbosity and length bias

Verbosity bias is insidious because longer answers often genuinely are a little better, so the correlation is not pure noise — which makes it easy to rationalise. The discipline is to measure it. Log response length alongside every score, and periodically check the correlation. If your judge's scores track token count more tightly than they track your human labels, the judge is grading length, not quality. Beyond the "do not reward length" instruction, the strongest structural fix is to make concision an explicit, separately-scored criterion, so verbosity is penalised on its own axis instead of leaking into the quality score.

Self-preference bias

Self-preference bias — sometimes called self-enhancement — is the reason you should almost never let a model grade its own homework. If the system under test is built on one provider's model, judge it with a different provider's. There is a growing literature quantifying this effect, including dedicated studies of self-preference in LLM judges (for example arXiv 2410.21819), and the practical takeaway is consistent: cross-family judging, provenance-blinding, and judge panels all reduce it. A panel — say two or three judges from different families, aggregated by majority for pairwise or mean for pointwise — is more expensive but markedly more robust, and it gives you a disagreement signal that flags exactly the items worth sending to a human.

Calibrate against humans: the agreement loop

A judge you have not calibrated is a hypothesis, not a tool. Calibration means measuring how well the judge agrees with humans on a labelled set, and the loop is simple to state: assemble a representative set of outputs, have humans label them, run the judge over the same set, and quantify the agreement. If the agreement is high enough, you can trust the judge to extend those human judgements to volumes no human could review. If it is not, you fix the rubric and go round again. The set you built at the start — see the pro tip above, and our guide to building golden sets and judges — is exactly the ground truth this loop consumes.

Why raw agreement lies

The intuitive metric is raw agreement: the percentage of items where the judge and the human chose the same label. It is also misleading, because some of that agreement happens by chance — and the more skewed your labels are (say 80% of answers are "pass"), the more agreement you get for free. A judge that blindly says "pass" every time will look 80% accurate on that set while being completely useless. Cohen's kappa corrects for exactly this. It measures agreement above what chance alone would produce: kappa = (p_o − p_e) / (1 − p_e), where p_o is observed agreement and p_e is the agreement expected by chance from each rater's label frequencies. This is not a fussy statistical nicety. A large-scale 2026 evaluation of judge models (arXiv 2606.19544) found the gap between exact-match agreement and Cohen's kappa ran to roughly 33 to 41 percentage points on MT-Bench data — meaning teams reading raw agreement were overestimating their judges by a third or more.

A worked kappa

Take 100 outputs, each labelled PASS or FAIL by both your judge and a human. Suppose they agree on 85 of them. The confusion matrix breaks down as: both PASS on 45, both FAIL on 40, judge PASS while human FAIL on 10, and judge FAIL while human PASS on 5. Observed agreement p_o is (45 + 40) / 100 = 0.85. Now the chance term. The judge said PASS 55 times and FAIL 45 times; the human said PASS 50 and FAIL 50. So the agreement expected by chance is (55/100 × 50/100) + (45/100 × 50/100) = 0.275 + 0.225 = 0.50. Therefore kappa = (0.85 − 0.50) / (1 − 0.50) = 0.35 / 0.50 = 0.70. The headline 85% agreement collapses to a kappa of 0.70 once you strip out the coin-flips — still good, but a materially different story from what raw agreement told you.

Here is the same calculation in code, using the same confusion matrix, so you can drop it straight into your calibration script:

from sklearn.metrics import cohen_kappa_score

# 100 items, each labelled PASS (1) or FAIL (0) by the judge and by a human.
# Confusion matrix:
#                 human PASS   human FAIL
#   judge PASS        45           10
#   judge FAIL         5           40
judge = [1]*45 + [1]*10 + [0]*5  + [0]*40
human = [1]*45 + [0]*10 + [1]*5  + [0]*40

kappa = cohen_kappa_score(judge, human)
print(round(kappa, 3))          # -> 0.7
# Raw agreement for comparison:
print(sum(j == h for j, h in zip(judge, human)) / len(judge))   # -> 0.85

# Interpreting kappa (Landis & Koch, 1977 — a convention, not a law):
#   < 0.00   none         0.41-0.60   moderate
#   0.00-0.20 slight       0.61-0.80   substantial
#   0.21-0.40 fair         0.81-1.00   almost perfect

A kappa of 0.70 lands in the "substantial" band on the widely-used Landis and Koch rule of thumb — good enough to let the judge do bulk grading, provided you keep sampling its verdicts for human review. Treat those bands as conventions, not physical constants; a judge gating a safety-critical decision should clear a higher bar than one triaging which transcripts a human should read first. The number to remember is your own: report kappa, per criterion, and re-measure it whenever you change the judge model, the prompt, or the population of inputs. A judge calibrated on last quarter's traffic is not calibrated on this quarter's.

When one judge should become many

The final move, once your single judge is calibrated, is knowing when to stop cramming everything into it. A prompt that asks one model to weigh faithfulness, relevance, safety, tone and format compliance and return a single "quality" score is asking for trouble. It is nearly impossible to calibrate, because a human and the judge can agree on the number while disagreeing wildly about why. It hides failures, because a brilliant, on-brand answer that happens to be unsafe can still score a 4. And it makes debugging a guessing game. The literature on breaking evaluation criteria apart rather than merging them — a point made in the surveys and traceable back to work like Wu and Aji on decomposing criteria — consistently finds that narrow judges beat monolithic ones.

The pattern is to decompose the judge into a small directed acyclic graph of sub-judges, each scoring exactly one concern, wired together with dependencies and gates. A worked example for a customer-support answer:

• Format check (cheap, deterministic where possible): does the output parse, is it in the right shape? If it fails, stop — nothing else matters.
• Safety judge (hard gate): is the answer safe and policy-compliant? A failure here vetoes the whole verdict regardless of every other score.
• Faithfulness judge: is every claim supported by the retrieved context? This is the pointwise template from earlier.
• Relevance judge: does it actually address the user's question? Only worth running if format and safety passed.
• Tone judge: is it appropriately concise and on-brand for the market — a UK bank versus an Indian fintech will differ here?

Aggregation is then explicit and defensible: hard-gate on safety and format, and combine the rest with weights you choose deliberately rather than a number the model invented. Each sub-judge is narrow enough to calibrate against humans on its own — you can compute a separate kappa per judge — and when the aggregate disagrees with a reviewer, you can see which sub-judge was wrong. The cost is more model calls, so decompose only the concerns that genuinely need it; a two-node graph that gates on safety and scores faithfulness already captures most of the benefit. This is also where judge design connects to agent evaluation more broadly, since the same decomposition logic applies to scoring an agent's trajectory step by step.

Build the judge before you trust the scores

An LLM-as-a-judge is one of the highest-leverage tools you can build, because it lets a handful of careful human judgements scale to every output your system produces. But leverage cuts both ways: a miscalibrated, biased judge scales bad decisions just as efficiently as a good one scales good ones. The work is not glamorous — choosing pointwise or pairwise, writing anchored evaluation steps, randomising order, length-normalising, judging across model families, and grinding out a kappa against a human-labelled set — but it is the difference between a number you can defend in a design review and a plausible-looking figure that quietly misleads your whole team. Build the judge like the piece of infrastructure it is, calibrate it, keep re-calibrating it, and only then let it hold the gate.