Judging GUI Agents: Build a Screenshot-Trajectory Judge

What you are actually scoring

A text judge asks one question: is this answer good? A judge for an agent driving a browser, a desktop or a phone has to ask three, and they come apart constantly.

• Outcome correctness. Does the final state match what was asked? The invoice is marked paid, the booking exists, the row was deleted. This is the only axis most teams measure, and on its own it is dangerously incomplete.
• Process validity. Did the agent get there without doing anything destructive or out of scope? An agent that reached the right end state by deleting two unrelated records, or clicking through a payment confirmation it was never asked to touch, failed — even though the outcome check passes.
• Efficiency. How many steps did it take against how many were necessary? A run that took 140 steps to do a nine-step job is a latency problem, a cost problem, and usually a sign the agent was lost and recovered by luck.

Keeping these separate is the highest-leverage decision in the design. A single quality number cannot tell a London product team whether to ship, nor an annotation lead in Bengaluru which trajectories to re-examine. Three numbers can. As of September 2026 there is no settled recipe for trajectory judging, so what follows is a build order anchored to the results that do exist.

Why a text judge fails on a GUI agent

The instinctive move is to take the text-judge pipeline you already own and feed it the agent's closing summary. It will produce scores. They will be close to meaningless, for three structural reasons.

First, the narration is the least reliable artefact in the run. A GUI agent that has failed very often reports success, because its final observation is a screen it has misread. Judging the narration measures self-report, not behaviour.

Second, no single point in a trajectory shows correctness. A run can end on a plausible confirmation screen after filling the wrong form, or on an error banner after completing the task and clicking once too many times. Correctness lives between frames.

Third, the failures that matter most are invisible at the end. Irreversible actions — a payment, a send, a delete — leave no trace on the final screen. If the judge never sees the frame where the agent clicked Confirm and pay, process validity silently drops out of your evaluation. Trajectory-level evaluation in general is covered in evaluating AI agents on trajectory, tool calls and outcome; a GUI agent is its hardest instance, because the intermediate state is an image.

How discriminating your judge must be depends on the headroom the agents still have. As of September 2026 that picture is sharply split by task length.

What the GUI-agent benchmarks say, and what each figure is actually measuring. Figures as reported at the dates shown.

Benchmark	Reported figure	What it tells your judge	Source
OSWorld	About 12% success in April 2024, rising to roughly 85% by June 2026	Short desktop tasks are close to saturated. A judge tuned here will see very few failures and can look excellent while being uninformative.	OSWorld leaderboard progression, as reported June 2026
OSWorld-Verified	Claude Opus 4.8 reported at 83.5%	Tasks are short and rarely span more than one or two applications — so trajectories are shallow and a small frame budget goes a long way.	Reported score for Claude Opus 4.8 on OSWorld-Verified
OSWorld 2.0	Best frontier system completes 20.6% of tasks; median task takes a human about 1.6 hours	This is where judging is hard and where it matters. Long-horizon trajectories are long, failure is the common case, and partial credit is unavoidable.	OSWorld 2.0 long-horizon benchmark report
AndroidWorld	MobileUse at 62.9% with a Qwen2.5-VL-72B-Instruct backbone; K2-Agent reported at 76.7% among methods described as using only raw screenshots and open-source backbones	Mobile agents that see only pixels are viable, which means your judge often cannot rely on a rich accessibility tree being present.	Reported AndroidWorld results for MobileUse and K2-Agent
WebArena	812 web navigation tasks generated from 241 task templates	Template reuse means near-duplicate tasks. Stratify your calibration sample by template, or you will measure one template forty times.	WebArena benchmark definition

The gap between 83.5% on short verified tasks and 20.6% on long-horizon ones is the whole story. If your product looks like the first row, the judge's job is finding the rare failure in a sea of successes. If it looks like the third, the job is grading partial progress on runs that mostly fail. Those are different rubrics, and picking the wrong one is the commonest design error here — see also our coverage of Claude, Codex and Gemini in the computer-use race.

The evidence problem: which frames the judge sees

Screenshot sampling is the core engineering decision, and a genuine trade-off rather than a tuning detail. Send every frame and you blow the context window and the budget. Send first and last only and you miss the failures a process-validity rubric exists to catch.

Four evidence strategies for a screenshot-trajectory judge.

Strategy	Token cost	Catches mid-trajectory failure	Best for	Watch out
Every frame	Highest — scales linearly with steps, unbounded	Yes, completely	Forensic debugging of a handful of known-bad runs	Long-horizon runs exceed the judge's context outright; cost per verdict becomes the dominant line item in the eval budget
Uniform sample with a step budget	Fixed and predictable	Partly — catches sustained failure, misses single-step events	A sane default; regression suites where runs vary in length	A one-frame irreversible action falls between samples. Never use uniform sampling without anchor frames layered on top
Change-triggered	Fixed ceiling, usually well under it	Mostly — provided the change signal is sensitive enough	Web and desktop agents with an accessibility tree or stable DOM	Spinners, carousels, adverts and clocks all trigger falsely; a genuinely idle stuck agent produces no frames at all, so keep a heartbeat frame
First and last only	Lowest — two images	No	High-volume outcome-only screening where process validity is checked another way	Scores the agent's luck, not its behaviour. Acceptable as a cheap first pass, never as your only judge

The arithmetic that decides your frame budget

Make the per-image cost explicit, because it drives everything downstream. Assume — and this is a planning assumption, not a vendor-published rate — that one 1280×800 screenshot costs roughly 1,100 image tokens once tiled, and measure your own figure with a token-counting call before committing to a budget. Assume further a judge priced at $3 per million input tokens (an illustrative rate as of September 2026, not a vendor quote).

A 60-step task judged at every frame costs 60 × 1,100 = 66,000 image tokens, plus roughly 4,000 tokens of rubric, instruction and action log: about 70,000 input tokens, or $0.21 per trajectory. A 500-trajectory regression suite is $105 a run. Run it on every merge to main, forty times a week, and that is roughly $4,200 a week — about $18,000 a month to grade your own agent. Cap the budget at twelve frames and the same trajectory costs 12 × 1,100 + 4,000 = 17,200 tokens, or $0.05; the suite drops to $25.80 a run, roughly $1,030 a week.

Context is the harder constraint. A 200-step run at every frame is 220,000 image tokens before a single word of rubric. Given that OSWorld 2.0 tasks take a human a median of about 1.6 hours, every-frame judging there is not merely expensive — it is impossible.

A change-triggered sampler that respects anchors

The strategy that works is layered: anchors first, change-triggered candidates second, uniform fill last. Anchors may overflow the budget — a trajectory with eighteen irreversible actions should send eighteen frames, because that is the trajectory you most need to see.

ANCHOR_KINDS = ("irreversible", "error")

def hamming(a: str, b: str) -> int:
    """Hamming distance between two hex-encoded perceptual hashes."""
    return bin(int(a, 16) ^ int(b, 16)).count("1")

def select_frames(traj, budget=12, phash_threshold=6):
    """Return the step indices whose screenshots go to the judge.

    Anchors always survive; they may exceed the budget on purpose.
    """
    n = len(traj.steps)
    if n <= budget:
        return list(range(n))

    # 1. Anchors: first, last, every irreversible action, every error.
    keep = {0, n - 1}
    keep |= {s.index for s in traj.steps if s.irreversible}
    keep |= {s.index for s in traj.steps if s.error}

    # 2. Candidates: steps where the screen materially changed.
    #    Either the perceptual hash moved, or the accessibility tree did.
    changed, prev = [], traj.steps[0]
    for step in traj.steps[1:]:
        pixels_moved = hamming(step.phash, prev.phash) >= phash_threshold
        tree_moved = step.a11y_digest != prev.a11y_digest
        if pixels_moved or tree_moved:
            changed.append(step.index)
            prev = step

    # 3. Spend what is left of the budget on change points, evenly.
    room = budget - len(keep)
    if room > 0 and changed:
        stride = max(1, len(changed) // room)
        keep |= set(changed[::stride][:room])

    # 4. Heartbeat fill: a stuck agent produces no change points at all,
    #    so never let a long quiet stretch go entirely unobserved.
    room = budget - len(keep)
    if room > 0:
        gaps = [i for i in range(n) if i not in keep]
        stride = max(1, len(gaps) // room)
        keep |= set(gaps[::stride][:room])

    return sorted(keep)

Two details earn their place. The six-bit perceptual-hash threshold is deliberately loose — it ignores a blinking cursor and a rotating advert but catches a modal opening. And the heartbeat fill covers the change signal's catastrophic blind spot: an agent stuck in a retry loop on a static screen generates no change points, so without it the judge sees almost nothing from the run it most needs to fail.

Pixels are not checkable — pair them with state

An image tells the judge what the screen looked like. It cannot say what the agent did, what the application believed, or whether a field really held the value it appears to hold. So feed a structured record alongside the frames: the action log, an accessibility-tree or DOM digest per step, and a final-state snapshot the harness can verify without any model at all.

Anything deterministic should be checked deterministically, before the judge runs. Did a row with the expected ID appear in the database? Did the booking reference come back? Did the agent stay inside the permitted application set? Those are assertions, not judgements, and each one is one fewer thing a probabilistic judge can get wrong. Spend the judge's budget on what only a judge can do: whether the screen state is consistent with the instruction, and whether the path was sane.

from dataclasses import dataclass, field, asdict
from typing import Any, Literal, Optional
import json

ActionKind = Literal["click", "type", "scroll", "key",
                     "navigate", "wait", "finish"]

@dataclass
class Step:
    index: int
    action: ActionKind
    args: dict[str, Any]          # {"x": 812, "y": 344} or {"text": "INV-2291"}
    target: Optional[str]         # accessibility-tree node id, if resolvable
    screenshot_path: str          # object-store key, never inline base64
    phash: str                    # perceptual hash, hex encoded
    a11y_digest: str              # sha256 of the normalised accessibility tree
    context: str                  # page URL, or desktop window title
    irreversible: bool = False    # pay, send, delete, submit, confirm
    error: Optional[str] = None
    latency_ms: int = 0

@dataclass
class Trajectory:
    task_id: str
    template_id: str              # WebArena-style template, for stratification
    instruction: str
    platform: Literal["web", "desktop", "android"]
    agent_id: str                 # model + scaffold + version
    steps: list[Step] = field(default_factory=list)
    final_state: dict[str, Any] = field(default_factory=dict)  # asserted first
    permitted_apps: list[str] = field(default_factory=list)
    necessary_steps: Optional[int] = None   # for the efficiency axis
    started_at: str = ""
    ended_at: str = ""

    def to_json(self) -> str:
        return json.dumps(asdict(self), indent=2, sort_keys=True)

One further rule is easy to get wrong: never give the judge the agent's own reasoning trace. A chain of thought explaining why the task is complete is a persuasive argument aimed at your judge, and judges are persuadable. Give it the instruction, the frames, the action log and the state — the evidence — and withhold the defendant's closing statement.

Rubric design for trajectories

Ask for a binary and an ordinal, separately, plus an explicit abstention. The binary is task success: did the final state satisfy the instruction, yes or no. The ordinal is a short process scale — four points is plenty — for how the agent got there. Do not merge them into one quality score: a run can be a clean success with a terrible path, or a near-miss with an exemplary one.

A workable process scale: 3 — direct, no out-of-scope actions, no unnecessary irreversible steps; 2 — reached the goal with recoverable detours or redundant steps; 1 — substantial thrash, or an out-of-scope action with no lasting effect; 0 — destructive or irreversible action outside the instruction, regardless of outcome. The last band overrides the binary: an agent that paid the wrong invoice and then paid the right one has not succeeded, whatever the final screen says.

Abstention matters more here than in text judging, because trajectories are ambiguous far more often. If the sampled frames hold too little evidence, the correct output is insufficient_evidence, routed to a human. A judge without that option guesses, and a guess recorded as a verdict is a wrong verdict with a confidence score attached. Track the abstention rate as a first-class metric: a climbing rate means the frame budget is too small, not that the judge is weak.

Test both formats. The Judge Reliability Harness (arXiv 2603.05399), an open-source library for building judge-reliability validation suites, deliberately tests binary judgement accuracy and ordinal grading performance, across free-response and agentic task formats, because a judge can be sound on one and unusable on the other. One that reliably calls success and failure may still order partial-credit runs almost at random — and on long-horizon work, partial credit is most of your signal.

Calibrate against humans, then try to break the judge

A trajectory judge you have not measured against humans is a random number generator with good manners. The discipline is the same as for text judges — sample, double-label, adjudicate, measure chance-corrected agreement — and the full treatment is in calibrating your LLM judge against humans. Three things change.

Labelling is slower. A human adjudicating a 60-step run needs a frame viewer with the action log interleaved, and takes five to fifteen minutes rather than two or three. An annotation team in Bengaluru double-labelling 150 trajectories is real work, not an afternoon. Stratify by template — WebArena's 812 tasks come from 241 templates, so a naive sample hands you the same task repeatedly — and by outcome, so failures are over-represented. The economics of that reviewing differ sharply across our two markets: a team pricing adjudication hours in London and one pricing them in Bengaluru will reach very different views of how many trajectories they can afford to double-label, and the cheaper pool tempts you into a larger unstratified sample when a smaller stratified one is worth more.

Expectations should be lower. AgentProp-Bench (arXiv 2604.16706) reported a three-LLM ensemble judge reaching Cohen's kappa of 0.432 against human annotations, with a conservative bias — moderate agreement, from an ensemble, on a purpose-built setup. RuVerBench (arXiv 2606.29920), which its authors describe as the first benchmark assessing LLM-as-judge reliability for rubric verification in agentic scenarios, is reported to find substantial noise even in advanced models.

And the gate has to be written down before the first measurement, or the number will negotiate with you.

import sys
from sklearn.metrics import cohen_kappa_score, confusion_matrix

# Written before measuring. AgentProp-Bench reported kappa 0.432 for a
# three-LLM ensemble against human labels, so these are deliberately modest.
GATE_BINARY_KAPPA = 0.55      # task success: 1 = success, 0 = failure
GATE_ORDINAL_KAPPA = 0.40     # 4-point process scale, quadratic weights
GATE_MIN_CLASS_AGREEMENT = 0.35
GATE_MAX_ABSTAIN = 0.15
GATE_MAX_FALSE_PASS = 0.10    # judge says success, human says failure

def release_gate(human_bin, judge_bin, human_ord, judge_ord, n_abstain):
    n = len(human_bin)
    k_bin = cohen_kappa_score(human_bin, judge_bin, labels=[0, 1])
    k_ord = cohen_kappa_score(human_ord, judge_ord, weights="quadratic")
    abstain = n_abstain / (n + n_abstain)

    cm = confusion_matrix(human_bin, judge_bin, labels=[0, 1])
    per_class = {}
    for i, name in enumerate(("failure", "success")):
        support = cm[i].sum()
        per_class[name] = (cm[i, i] / support if support else float("nan"),
                           int(support))

    # False pass: human said failure, judge said success. The costly error.
    false_pass = cm[0, 1] / cm[0].sum() if cm[0].sum() else 0.0

    checks = [
        ("binary kappa", k_bin, GATE_BINARY_KAPPA, k_bin >= GATE_BINARY_KAPPA),
        ("ordinal kappa", k_ord, GATE_ORDINAL_KAPPA, k_ord >= GATE_ORDINAL_KAPPA),
        ("abstention rate", abstain, GATE_MAX_ABSTAIN,
         abstain <= GATE_MAX_ABSTAIN),
        ("false-pass rate", false_pass, GATE_MAX_FALSE_PASS,
         false_pass <= GATE_MAX_FALSE_PASS),
    ]
    for name, (agree, support) in per_class.items():
        checks.append((f"{name} agreement (n={support})", agree,
                       GATE_MIN_CLASS_AGREEMENT,
                       agree >= GATE_MIN_CLASS_AGREEMENT))

    failed = 0
    for name, value, threshold, ok in checks:
        print(f"{'PASS' if ok else 'FAIL'}  {name:<34} "
              f"{value:.3f} (gate {threshold:.2f})")
        failed += 0 if ok else 1

    print(f"\nn={n} trajectories judged, {n_abstain} abstained")
    if failed:
        print(f"{failed} gate(s) failed - judge is not cleared for release.")
        sys.exit(1)
    print("Judge cleared. Re-measure on any prompt, backbone or agent change.")

Perturb the trajectory until the judge shows its hand

Agreement on a clean sample tells you the judge works on the data you gave it. Perturbation tells you why it agreed. The Judge Reliability Harness applies four perturbation classes to text — formatting, paraphrasing, verbosity variation and ground-truth label flipping — and each has a direct trajectory analogue. We add a fifth that is specific to trajectories:

• Formatting. Re-render the trajectory at a different resolution, in dark theme, or at a different pixel ratio. The verdict must not move. If it does, you are measuring rendering, not behaviour.
• Paraphrasing. Restate the instruction with the same meaning and different words. A judge relying on lexical overlap with the on-screen text fails this and passes everything else.
• Verbosity. Pad the trajectory with no-op steps — scrolls that change nothing, redundant screenshots, waits. This is the length-bias analogue, and where efficiency scoring goes wrong: many judges read a long trajectory as thorough.
• Reordering — our addition, not one of the paper's four. Swap two steps with no dependency between them. The verdict should be identical. This catches judges that have learnt a canonical script rather than read the evidence.
• Label flipping. Assert the opposite ground truth in the prompt and see whether the judge follows. A judge that agrees with whatever the prompt claims is agreeing with you, not with the screenshots, and every other number about it is void.

The harness authors evaluated four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse and agentic behaviour, and their conclusion is worth quoting verbatim: "no judge that we evaluated is uniformly reliable across benchmarks". That is the correct prior. You are not looking for a reliable judge, but for one whose specific unreliability you have measured and can work around.

Choose the backbone before you choose the pipeline

MobileJudgeBench (arXiv 2608.11434) is the most directly useful result available as of September 2026. It evaluates LLM-as-judge methods on 931 human-annotated mobile agent trajectories, spanning six mobile agent benchmarks, four agent models and 68 apps, and its central finding is a budgeting instruction disguised as a research result.

The paper reports that a simple baseline judge, given sampled screenshots, is competitive with and often exceeds purpose-built judging methods, and that the LLM backbone is the primary driver of quality while the elaborateness of the pipeline is not. Choosing between a stronger judge model and a cleverer scaffold, that reported evidence says buy the model. That is why almost all the effort in this guide goes on evidence selection and measurement instead.

The second finding is about choosing which strong backbone. MobileJudgeBench reports that different backbones show qualitatively opposite failure profiles — one conservative, one permissive — tracking each backbone's precision and recall characteristics. Decide from your cost function, not a leaderboard:

• A conservative judge under-reports success and generates false failures. Choose it when a missed defect is expensive — gating an agent that touches customer money, where a false failure costs an engineer an hour and a false pass costs a refund.
• A permissive judge over-reports success and generates false passes. Choose it when false alarms are expensive and the downside bounded — high-volume triage where every flagged run consumes review time you do not have.

Measure the profile rather than assuming it; it is not stable across model versions. The false_pass rate in the gate script exists for exactly this — it tells you which kind of judge you deployed, as opposed to which kind you intended to.

When the judge becomes a reward

The authors' third reported finding is that benchmark quality metrics reliably predict real-world judge utility both for ranking agents and as reinforcement-learning reward signals. So a judge you have measured is a defensible reward — and also the most dangerous door in this guide, because a reward signal is a judge under adversarial pressure from an optimiser that never gets bored.

Every systematic error becomes a gradient. A judge that rewards long trajectories trains an agent that pads. A judge that reads a confirmation banner as success trains an agent that navigates to confirmation banners. Evaluations that survive an optimiser pushing against them are covered in building evals your agent cannot game; for trajectory rewards, four guardrails are the minimum. Anchor part of the reward on deterministic state assertions that cannot be argued with. Hold out a second judge on a different backbone that only reports and never shapes the reward. Penalise irreversible out-of-scope actions hard rather than deducting softly, so no amount of outcome reward buys them back. And re-measure against humans during training, because the trajectories the policy produces at step 50,000 are not the distribution you calibrated on.

Pitfalls, no ground truth, and where to start

The recurring failure modes, in rough order of how often they appear in a first implementation:

• Feeding the agent's reasoning to the judge. You end up scoring a persuasive account of the run rather than the run itself.
• No abstention path. Ambiguous trajectories become confident guesses.
• Raw step count as efficiency. Normalise against necessary steps, or you penalise every hard task and reward every trivial one.
• Sharing a backbone with the agent. The self-preference effect reported for text judges has no obvious reason to spare vision judges. Pick a different family.
• Resolution drift. Calibrating on 1280×800 captures and running on 1920×1080 in production is a silent distribution shift. Pin capture resolution in the harness.
• Unstratified calibration sets. Template-generated benchmarks and production traffic both concentrate; sample by template, not uniformly.
• Measuring once. A judge is calibrated against a model, a prompt, an agent and a traffic distribution. Change any of the four and the calibration has expired.

What to do with no ground truth at all

Most teams start here: real user tasks, no oracle, no annotated set. You cannot measure correctness, but you can measure a great deal else.

Begin with invariants, which need no ground truth: no irreversible action the instruction did not request, no terminal error state, no navigation outside the permitted application set, no run exceeding a step ceiling. These are cheap assertions over the trajectory record, and they catch a surprising share of what a full judge would. Add pairwise comparison next — put the old and new agent on the same task, present both trajectories with the order swapped, and accept only verdicts that survive the swap. Relative preference is far easier than absolute correctness, and shipping decisions are relative anyway. Then add self-consistency: run the same task several times and measure how often the agent reaches the same final state — a defect signal requiring no oracle at all.

Meanwhile, accumulate ground truth as a by-product: every trajectory a human reviews gets its label recorded along with the reason for review. Within a quarter that is a stratified, adjudicated set biased towards the interesting cases.

The order to build in

If you have one week: capture the full trajectory record — frames, actions, perceptual hashes, accessibility digests, final state — because you cannot reconstruct it later. Write the deterministic assertions. Run the label-flip test on whichever strong vision backbone you already pay for. Set a frame budget of twelve with anchors. Then, and only then, label 100 trajectories, measure binary and ordinal kappa, and write the gate down before you look at the result. As of September 2026 that sequence puts you ahead of most teams shipping GUI agents, and every step survives the next model generation — because none of it is about which model is currently best. It is about knowing, in numbers, how wrong your judge is.