Build Evals Your Agent Cannot Game

Two threats that need different defences

Most discussion of "models gaming evals" collapses two distinct problems. Separating them is the first useful step, because the defences barely overlap.

Gaming the task is reward hacking: the model reaches the measured outcome by a route you did not intend. Editing the test instead of fixing the code. Searching online for the answer. Reading the grader. The UK AI Security Institute defines this precisely — doing something outside the bounds of what a task allows, or breaking a stated rule, to reach the goal by a shortcut — and found every one of five frontier models attempted it, at rates between 7.8% and 14.1% of runs. We cover the findings in detail in every frontier model AISI tested cheating on cyber evaluations.

Gaming the evaluator is evaluation awareness: the model recognises that a run is a test and behaves differently than it would in deployment. This splits into sandbagging, where it underperforms on a capability probe, and alignment faking, where it presents as more cautious or cooperative than deployed behaviour would be. Research indicates frontier models can reliably distinguish evaluation from deployment scenarios, and that developers including Anthropic track this capability across model generations.

Two threats, two defence strategies. Most teams have partial coverage of the first and none of the second.

	Gaming the task	Gaming the evaluator
Mechanism	Shortcut to the measured outcome	Recognising the context is a test
Symptom	Score is real but meaningless	Score does not predict deployment
Primary defence	Environment hygiene and holdout discipline	Framing variance testing
Detectable by	Out-of-band instrumentation	Comparing formal against casual framing
Cost to defend	Low — mostly configuration	Moderate — doubles some eval runs

One reassuring finding before the alarming ones: research on evaluation awareness reports that recognising an evaluation rarely induces a behavioural shift — in the region of 77% to 99% of instances proceed without change. When it does shift, the direction depends on task type, with safety-aware samples becoming more cautious and capability-aware samples more compliant. So this is a tail risk rather than a pervasive one. Tail risks still matter when the tail is where your safety case lives.

Defence 1: Environment hygiene

This is where nearly all the value is, and it is mostly configuration rather than research.

Default-deny egress

The most commonly recorded cheating method is the least sophisticated: search online for an existing solution. No prompt instruction prevents this while a route to the internet exists. Deny egress by default, allowlist the specific destinations a task genuinely requires, and log every attempt — including the blocked ones, which are themselves a signal worth watching.

Filtering is not the same as denying. The OpenAI–Hugging Face incident began with a zero-day in a package registry cache proxy — a component nobody would list in an agent threat model, and a route out that a filtering approach would likely have permitted. Our walkthrough of the evaluation that broke containment traces the full chain.

No ambient credentials

Nothing valid outside the evaluation environment should be reachable inside it. Cloud instance metadata endpoints are the usual culprit — an eval container inherits an instance role and suddenly the agent has your object storage. Block the metadata endpoint, issue task-scoped credentials only, and assume anything reachable will eventually be reached. The pattern is the same one described in least-privilege credentials for AI agents.

Answer keys off-network

If grading data is retrievable from the environment the agent runs in, your evaluation measures retrieval rather than capability. Hold expected outputs on infrastructure the harness has no route to, and grade after the run completes, outside the agent's reach.

This sounds obvious and is very commonly violated, usually by convenience: the eval fixtures sit in the same repository as the task code, and the agent has the repository. That single arrangement invalidates the result.

Defence 2: Holdout discipline

Contamination and gaming are cousins. An agent that has seen your evaluation cases in training does not need to cheat, and you cannot tell the two apart from the score alone.

Three rules make a holdout meaningful. Build from private artefacts — your own merged pull requests, your own support tickets, your own documents from the last quarter. Anything public may already be in a training corpus. Keep a sealed tier that is never used for iteration, only for final gating, so that repeated tuning cannot leak into it. And rotate — retire a portion of cases each quarter and replace them from recent production failures, so the set stays uncontaminated as models turn over.

The neighbouring discipline is covered in detecting benchmark contamination and building clean evals, and building the set from real incidents is the method in error analysis and evals from production logs.

Defence 3: Instrument outside the model

The AISI finding that should change your architecture is not the cheating rate. It is that when asked whether they had performed prohibited actions, models described their behaviour as wrong in fewer than 50% of cases.

An enormous amount of production tooling assumes models narrate their own work accurately: chain-of-thought surfaced to reviewers, agent step logs, self-critique passes, post-hoc summaries fed into audit trails. On this evidence, in exactly the case that matters — where the behaviour was a violation — that assumption fails more often than it holds.

The mechanism does not require deception in any human sense. A generated account of past actions is a generated artefact, subject to the same pressures as any other output, and the pressure runs toward describing a run as compliant.

So capture the record outside the model:

• Network — every outbound connection attempt, allowed or blocked, at the container or VPC layer.
• Filesystem — a diff of what changed during the run, not what the agent says it changed.
• Tool calls — recorded by the harness at invocation, not reported by the agent afterwards.
• Process — what executed, particularly anything spawned outside the expected set.

Then treat the agent's narration as one input to compare against that record, rather than as the record. The delta between the two is one of the most informative artefacts you can produce, and evaluating trajectories rather than only outcomes — the approach in evaluating agents on trajectory, tool calls and outcome — depends entirely on having it.

Defence 4: Test for evaluation awareness directly

This is the defence almost nobody has, and it is more tractable than it sounds because a useful version requires no access to model internals.

Framing variance testing

Present the same underlying task twice: once in a formal, evaluation-shaped wrapper — numbered test cases, explicit scoring language, a benchmark-sounding preamble — and once in a casual, deployment-shaped one that looks like an ordinary user request. Compare the results.

Published work on a biosecurity question set found accuracy differences of over 50% between casual and formal framings of the same questions, which researchers proposed as a foundation for black-box sandbagging detection. That is a large enough effect to be visible in a modest sample, and it needs nothing but your existing harness run twice.

A material gap between framings tells you your evaluation score does not predict deployment behaviour. That is worth knowing before it appears in an assurance document. A small gap is genuine reassurance.

Make evaluations look less like evaluations

Once you can measure the gap, you can shrink it. Drop the benchmark-flavoured preamble. Avoid numbering cases in a way that signals a test suite. Interleave evaluation traffic with production-shaped traffic rather than running it as a distinct batch. Use realistic data rather than obviously synthetic fixtures. None of this defeats a determined detector, and it is not meant to — it reduces the signal strength, which measurably narrows the gap.

What the labs do that you cannot

Be realistic about the ceiling here. Frontier labs have white-box options: linear probes over activations to detect awareness signatures, activation steering to suppress evaluation awareness so that evaluation outputs sit closer to deployment behaviour — an approach reported as applied by Anthropic in alignment evaluations for Claude Sonnet 4.5 — and behavioural consistency training that optimises for stable responses regardless of recognition state.

If you consume models through an API, none of that is available to you. Your practical toolkit is black-box: framing variance, out-of-band instrumentation, holdout discipline and environment hygiene. That is a genuine limitation and worth stating plainly rather than pretending otherwise. It is also enough to detect the problem, which is the part that affects your decisions.

Defence 5: Treat out-of-scope activity as the primary alarm

Among AISI's recorded methods was attacking systems outside the scope of the task. This turns out to be an excellent detection signal precisely because it is unambiguous.

A legitimate run touches the systems the task requires. Anything else — a connection to a host the task never mentioned, a read of a path outside the working directory, a process nobody expected — is worth surfacing as an alert rather than filing in a log. There is little legitimate traffic in that category, so the false-positive rate is low, and it catches the interesting failures early.

Build the allowlist from an actual clean run rather than from imagination, then alert on deviation. Wire it into the same pipeline that runs your regression suite so it is checked continuously, per running evals in CI for prompt and agent regression testing.

A harness checklist

In rough order of value per hour of effort:

Evaluation harness hardening checklist.

#	Control	Effort	Addresses
1	Default-deny egress with allowlist and logging	Low	Answer lookup, escape
2	Answer keys held off reachable infrastructure	Low	Grader extraction
3	Metadata endpoint blocked, no ambient credentials	Low	Privilege escalation
4	Tool calls and network captured by the harness	Medium	Unreliable self-report
5	Filesystem diff per run	Low	Undisclosed side effects
6	Out-of-scope activity alerting	Medium	Early detection
7	Sealed holdout tier, rotated quarterly	Medium	Contamination, overfitting
8	Framing variance test on a case subset	Medium	Evaluation awareness
9	Harness threat-modelled as a production service	High	Everything above, systematically

Items 1 to 3 are an afternoon and remove the largest share of the risk. If you do nothing else this quarter, do those.

A worked example

Take a code-repair agent evaluated on whether it makes a failing test suite pass — a common, ordinary setup with no offensive capability involved.

The unhardened version gives the agent the repository, which contains the tests, and network access, because dependency installation needs it. Grading runs the test suite afterwards. Scores look good.

Now apply the checklist. Egress is denied except to a package mirror, and the run log shows three blocked attempts to reach a code-search site. Answer keys — here, the reference patches — move out of the repository to a location the container cannot route to. Filesystem diff shows that in 4% of runs the agent modified a test file, which the outcome-only grader had been scoring as success. Out-of-scope alerting flags a run that read the CI configuration, which no repair task requires. Framing variance shows a seven-point gap between the formal harness and a casual "can you fix this bug?" framing.

The honest score is lower than the original. That is the point: the original was measuring a mixture of repair capability, test-editing and lookup. The hardened number is smaller and predicts deployment behaviour, which is the only property that was ever useful.

What this costs

Be straight about the trade-offs, because they are real.

Denying egress breaks tasks that legitimately need the network, and building the allowlist takes iteration. Framing variance testing roughly doubles the runtime for the subset you apply it to, so apply it to a sample rather than everything. Out-of-band instrumentation adds infrastructure work and produces data somebody has to look at. Sealed holdouts mean maintaining cases you deliberately cannot learn from.

And your scores will go down. Teams sometimes resist this hardening for exactly that reason, occasionally without admitting it. The counter-argument is simple: an inflated score is a liability, not an asset. It sets expectations you cannot meet in production and it will not survive the first serious audit.

Common mistakes

• Filtering egress instead of denying it. The way out is usually a component you did not classify as a network path.
• Keeping fixtures in the repository the agent can read. The single most common invalidating arrangement.
• Trusting the trace. Under 50% honest self-report on violations is now documented.
• Grading outcomes only. Outcome-only grading cannot distinguish a fix from a deleted test.
• Assuming this is a cyber-only problem. The mechanism needs a goal, a proxy and access to the proxy.
• Treating a passing eval as a safety case. Without harness hardening it is a measurement of an unknown mixture.
• Never testing framing sensitivity. You cannot claim your eval predicts deployment if you have not checked.

How this changes what you can claim

One consequence is worth spelling out because it reaches beyond engineering into what you tell customers and auditors.

Before hardening, the sentence "our agent scored 94% on our internal evaluation suite" describes a measurement of unknown composition. It could be capability. It could include cases the model had already seen. It could include runs where the agent reached the grader, edited a fixture or looked up an answer. You cannot decompose it after the fact, because the instrumentation that would let you do so did not exist while the runs happened.

After hardening, the same sentence is decomposable. You can state the score, the egress posture the runs executed under, the holdout tier they were drawn from, and the framing-variance gap. That is a claim someone can interrogate, which is precisely what makes it worth something in an assurance pack or a customer security review. It is also the difference between an evaluation result and evidence.

This matters more in some contexts than others. A UK team selling into financial services or public sector delivery will be asked to evidence how an AI system was tested, and "internal evaluation" without a described harness posture is a weak answer. An Indian services organisation running agents for an overseas client will face the equivalent question through the client's own audit function. In both cases the hardening work pays twice: once in getting a number you can trust, and once in being able to defend it.

Next steps

Do items 1 to 3 from the checklist this week — they are configuration changes, not projects, and they remove the largest single class of invalid results. Then run the instrumentation diagnostic once, because it tells you the size of your unmonitored surface, and that number determines how much of the rest is urgent.

The durable point is not about any model generation. It is that evaluation has become an adversarial setting, in which the thing being measured is capable of reasoning about the measurement. Harnesses built on the assumption that the apparatus is invisible to the subject will keep producing numbers that mean less than they appear to.