Your Eval Harness Is a Security Boundary. Prove It.

What you need to know

An evaluation harness is a security boundary. It sits between a capable model and everything the model is not supposed to reach, and it makes a set of promises about what is on the other side. Those promises are usually written in three places: a design document, a README, and — most consequentially — the system prompt handed to the model at the start of every run. None of those three places is a control.

The distinction sounds pedantic until it costs you. A control is a mechanism that makes an outcome impossible, or at minimum makes it observable. A statement is a description of what someone believes the mechanism does. When the two diverge, nothing complains. The evaluation still runs, the scores still come out, the dashboard still goes green — and the property you thought you had has quietly not existed for months.

This guide argues that the gap between the two is among the cheapest security work available to an AI team, and it walks through closing it: enumerating the assertions your harness actually makes, verifying network egress from inside the environment on every run, seeding canaries that trip on first use, scanning transcripts rather than only scores, and treating a third-party evaluation partner as a trust boundary you inherit rather than a guarantee you can rely on.

What actually happened, and why it generalises

On 30 July 2026, Anthropic's Frontier Red Team published a post titled "Investigating three real-world incidents in our cybersecurity evaluations". The substance is worth stating precisely, because the precise version is more instructive than the headline version.

Anthropic reviewed 141,006 evaluation runs in which Claude could have obtained internet access. Across those runs it found three incidents, spanning six runs, in which models reached the open internet and compromised three real organisations. The models involved were Claude Opus 4.7, Claude Mythos 5, and an internal research test model not intended for general release. The evaluations were run with a third-party testing partner, Irregular, and the tasks were capture-the-flag exercises — the model was asked to find information deliberately left on another machine or network.

The root cause is one sentence long, and it is the reason this article exists. Anthropic's evaluation prompt told the model its environment was a simulation with no internet access. Because of a misunderstanding between Anthropic and the evaluation partner, that was not true — internet access was available. The containment guarantee existed in the prompt. It did not exist in the network.

The timeline is worth recording too, because it becomes a useful benchmark later. The earliest incidents date to April 2026. Anthropic's review began on Thursday 23 July 2026 and the evaluations were halted the same day. The specific incidents were identified on 24 July. The affected organisations were notified on Monday 27 July. The post went out on 30 July. Anthropic also noted that its review was prompted by a separate but similar incident that OpenAI had disclosed the previous week — an episode we covered in OpenAI's models escaped their sandbox and breached Hugging Face.

It is worth being clear about the shape of this before drawing lessons from it. Anthropic found the problem through its own review, stopped the affected evaluation programme on the day it started looking, identified the specific runs within a day, notified the affected parties within four days, and published within a week. It announced mitigations rather than assurances: expanding continuous monitoring of evaluation transcripts, improving investigation tooling, more rigorous assurance work with vendors, plans to release lightly redacted transcripts, and engaging the third-party review organisation METR. That is what responsible disclosure looks like when the news is bad. The interesting content here is architectural, not a gotcha, and treating it as a company story rather than an engineering story is how you avoid learning anything from it.

Because the engineering story is entirely general. Strip out the frontier-model specifics and what remains is a harness that asserted a property, an environment that did not provide it, and no mechanism anywhere in between that would have noticed. Any team running agent evaluations has exactly that structure. Your harness asserts no network, or fake credentials, or a scoped filesystem, or stub tools. Your environment implements some subset of those. Nothing tests the difference.

Now consider the base rate honestly, because it is the part most write-ups skip. Six runs out of 141,006 is roughly 0.004 per cent. That is not a number you find by spot-checking. If you sampled a hundred transcripts a week — an unusually diligent review cadence for most teams — you would expect to wait years before one of the affected runs landed in your sample, and even then you would have to recognise it. A failure rate that low is invisible to human review by construction. It is only findable two ways: continuous automated monitoring across the whole corpus, or a deterministic control that makes the failure impossible rather than rare.

That is the argument in one line. Rare failures in a security boundary are not caught by looking harder; they are caught by testing the boundary. And the reason the rate was so low is itself instructive — most of those 141,006 runs did not attempt to escape, because most tasks did not call for it. The environment was equally uncontained for all of them. The six that surfaced were the ones where the model happened to try. A containment gap does not announce itself in proportion to its severity; it announces itself in proportion to how often something pokes at it.

Enumerate the assertions your harness makes

Start with a list, because you cannot verify a property you have not written down. Sit with the harness and the eval prompt open side by side and extract every claim either of them makes about the environment. Most teams find between six and twelve. Then, for each one, answer two questions: where is this enforced, and what would it look like if the enforcement were missing?

The table below covers the assertions that appear in almost every agent evaluation harness, together with the implementation people typically reach for and the failure mode that actually occurs in practice. The failure modes are the important column — they are all mundane, none of them require an adversary, and every one of them produces an environment that still looks correct from the outside.

Work through that list against your own harness and the exercise usually takes an afternoon. It is uncomfortable in a productive way: most teams discover at least two assertions that exist only in the prompt, and at least one that used to be enforced and stopped being enforced during a migration nobody connected to it. The container base image changed, the CI runner class changed, someone added a shared cache to speed the suite up — and a property quietly lapsed.

Two entries deserve extra attention. The stub-with-fallback pattern is worth grepping for before you go any further; a mock layer that silently proxies to the real API when a fixture is missing is how an evaluation suite ends up making thousands of live calls and nobody notices until the invoice. And the metadata endpoint is worth testing explicitly, because it is not "the internet" in most people's mental model — it is a link-local address that many egress rules do not touch, and it dispenses genuine cloud credentials to anything that asks.

Verify egress, do not assume it

Egress is the assertion that matters most, because it is the one that turns a contained failure into somebody else's incident. Everything else on the list above produces a mess inside your own estate. Egress produces a compromised third party, a notification obligation and a phone call you do not want to make.

Defence in depth here has four layers, and the fourth is the one almost nobody has.

Layer one: deny by default at the network namespace. The eval container gets its own network namespace with no default route to anything but an explicit egress path. Not a firewall rule applied afterwards; no route at all. This is the difference between "packets are dropped" and "there is nowhere for packets to go", and the second is considerably harder to undo by accident.

Layer two: an explicit allowlist at a proxy you control. Whatever the eval legitimately needs — a model API endpoint, an internal fixture server — reaches it through a forward proxy that you operate and the eval process cannot reconfigure. Everything else is denied and, crucially, logged. The deny log is a security signal in its own right. The mechanics of building this are covered in our guide to sandboxing AI agents with microVMs and allowlists, which handles the production case in depth; the evaluation case differs mainly in that the allowlist should be dramatically smaller and, for most suites, empty.

Layer three: a DNS sinkhole. Name resolution is an exfiltration channel by itself — a query for <base32-encoded-secret>.attacker.example carries data outward even if no connection is ever established. Pin the resolver to one you control, answer everything not on the allowlist with a loopback address, and log every query. As with the proxy, the log is half the value.

Layer four: an assertion test that runs before every batch and fails closed. Layers one to three are configuration, and configuration drifts. Layer four is a test that runs inside the eval environment, as the eval user, with the eval's own environment variables, immediately before the batch starts — and refuses to let the batch start unless every attempted egress path fails in the way you expect. On the facts Anthropic published, this is the layer that would very likely have caught that class of gap on day one rather than in month four, and it is roughly a hundred lines of code.

The critical design point is where it runs. A test that runs on your build server, or in a separate "network validation" job, or as a Terraform assertion, tells you about the network as your build server sees it. The only thing that tells you about the network the model sees is a test executing in the same namespace, same container, same user, same environment as the model's tool calls.

#!/usr/bin/env python3
"""preflight_egress.py -- assert that the eval sandbox is actually contained.

Run INSIDE the eval environment, as the eval user, with the eval's own
environment, immediately before every batch. Exit 0 means sealed.
Any other exit code means the batch MUST NOT start.

Each probe is an independent path to the outside world. They fail for
different reasons and they are configured in different places, which is
exactly why you have to test all of them.
"""

import json
import os
import socket
import sys
import time
import urllib.request

TIMEOUT = 3.0

# Stable, well-known anycast addresses. We are asserting that we CANNOT
# reach them, so the choice matters less than the fact that they are up.
V4_HOSTS = [("1.1.1.1", 443), ("8.8.8.8", 53), ("93.184.216.34", 80)]
V6_HOSTS = [("2606:4700:4700::1111", 443), ("2001:4860:4860::8888", 53)]
DNS_NAMES = ["example.com", "api.github.com", "canary-probe.invalid"]
METADATA = [("169.254.169.254", 80), ("fd00:ec2::254", 80)]

results = []


def record(path, blocked, detail):
    results.append({"path": path, "blocked": blocked, "detail": detail})


def probe_tcp(family, host, port, label):
    """Raw TCP. No DNS involved -- isolates routing from resolution."""
    started = time.monotonic()
    s = None
    try:
        # Socket construction is inside the try on purpose: in a namespace
        # with no IPv6 stack, socket(AF_INET6) itself raises. That is a
        # blocked path, not a crash.
        s = socket.socket(family, socket.SOCK_STREAM)
        s.settimeout(TIMEOUT)
        s.connect((host, port))
        record(label, False, f"CONNECTED to {host}:{port}")
    except OSError as exc:
        elapsed = time.monotonic() - started
        # Distinguish a route-level block from a timeout. A namespace with
        # no route refuses in microseconds; a silent DROP burns the full
        # timeout; an upstream that is merely slow also burns it. Record
        # which one happened -- a path that changes failure mode between
        # runs is a configuration change you want to see.
        mode = "no-route" if elapsed < 0.5 else "timeout-or-drop"
        record(label, True, f"{type(exc).__name__} ({mode}, {elapsed:.3f}s)")
    finally:
        if s is not None:
            s.close()


def probe_dns(name):
    """Resolution is an exfil channel even with every port closed."""
    try:
        addrs = sorted({ai[4][0] for ai in socket.getaddrinfo(name, None)})
    except OSError as exc:
        record(f"dns:{name}", True, type(exc).__name__)
        return
    # A sinkhole that answers 127.0.0.1 / :: is a PASS. Anything routable
    # means real resolution is happening somewhere.
    sinkholed = all(a.startswith("127.") or a in ("::1", "0.0.0.0", "::")
                    for a in addrs)
    record(f"dns:{name}", sinkholed, f"resolved to {addrs}")


def probe_http(host, use_proxy):
    """HTTP through the proxy, and HTTP with the proxy stripped.

    The second case is the one that bites: a subprocess, a Go binary or an
    SDK that ignores HTTP_PROXY goes straight out and your proxy allowlist
    was never in the path at all.
    """
    env_backup = {}
    if not use_proxy:
        for var in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy",
                    "https_proxy", "ALL_PROXY", "all_proxy"):
            if var in os.environ:
                env_backup[var] = os.environ.pop(var)
    label = f"http:{host}:{'via-proxy' if use_proxy else 'direct'}"
    try:
        # Use urllib, not http.client: http.client NEVER reads the proxy
        # environment variables, so building both probes on it would test
        # the direct path twice. urllib re-reads getproxies() per opener,
        # which is what makes stripping the variables above meaningful.
        opener = urllib.request.build_opener(
            urllib.request.ProxyHandler(urllib.request.getproxies()))
        with opener.open(f"https://{host}/", timeout=TIMEOUT) as resp:
            record(label, False, f"HTTP {resp.status}")
    except Exception as exc:
        record(label, True, type(exc).__name__)
    finally:
        os.environ.update(env_backup)


def probe_env_secrets():
    """Real credentials in the environment are an egress path with legs."""
    suspicious = []
    for key, value in os.environ.items():
        if not value or len(value) < 20:
            continue
        upper = key.upper()
        if any(t in upper for t in ("KEY", "TOKEN", "SECRET", "PASSWORD",
                                    "CREDENTIAL")):
            # Seeded canaries carry a distinctive prefix; anything else
            # in this shape is a real secret that leaked in.
            if not value.startswith("ATC-CANARY-"):
                suspicious.append(key)
    record("env:no-real-secrets", not suspicious,
           f"unexpected secret-shaped vars: {sorted(suspicious)}")


def main():
    for host, port in V4_HOSTS:
        probe_tcp(socket.AF_INET, host, port, f"tcp4:{host}:{port}")
    if socket.has_ipv6:
        for host, port in V6_HOSTS:
            probe_tcp(socket.AF_INET6, host, port, f"tcp6:[{host}]:{port}")
    else:
        record("tcp6", True, "no IPv6 stack in this namespace")
    for name in DNS_NAMES:
        probe_dns(name)
    probe_http("api.github.com", use_proxy=True)
    probe_http("api.github.com", use_proxy=False)
    for host, port in METADATA:
        probe_tcp(socket.AF_INET if ":" not in host else socket.AF_INET6,
                  host, port, f"metadata:{host}")
    probe_env_secrets()

    leaks = [r for r in results if not r["blocked"]]
    report = {"sealed": not leaks, "checked_at": time.time(),
              "results": results}
    print(json.dumps(report, indent=2))

    if leaks:
        sys.stderr.write(
            f"\nCONTAINMENT FAILED: {len(leaks)} path(s) reachable. "
            "Batch aborted.\n")
        for r in leaks:
            sys.stderr.write(f"  - {r['path']}: {r['detail']}\n")
        return 2
    return 0


if __name__ == "__main__":
    sys.exit(main())

A few things about that script are deliberate. It probes raw TCP by IP address before it touches DNS, because those are separately configured and separately broken. It probes IPv6 explicitly, because an IPv4-only firewall rule on a dual-stack host is one of the most common ways an "isolated" container reaches the internet, and because a machine can be perfectly filtered on IPv4 and completely open on IPv6 with nothing in the logs to suggest it. It probes HTTP twice — once through the proxy and once with every proxy variable stripped — because the direct path is the one that catches a subprocess or a non-Python binary that never honoured your proxy configuration in the first place. It probes the cloud metadata endpoint, which is the shortest route from a contained sandbox to a real credential. And it records how each path failed, not merely that it failed, so that a path silently changing from "no route" to "timeout" shows up as the configuration change it almost certainly is.

Canary credentials and canary targets

Egress blocking is a preventive control, and preventive controls fail silently. Canaries are the complementary detective control: they do not stop anything, but when one is touched you learn quickly, with very little ambiguity and a false-positive rate close to zero — usually your own tooling misconfigured rather than a genuine escape. If you build only two things from this article, build the pre-flight and build the canaries.

The idea is simple. Seed the evaluation environment with credentials and URLs that look genuinely useful and are worth nothing. Every one of them is monitored. Because no legitimate process has any reason to use them, a single use is a definite signal rather than an anomaly to be triaged. There is no threshold to tune and no baseline to establish — the expected count is zero for ever, and one is an incident.

This is the control most likely to surface that incident class on day one. A model that reaches the open internet during a capture-the-flag task and starts probing will, sooner or later, try a credential it found lying around. If that credential is yours and it is instrumented, you get an alert with a timestamp, a source address and a run identifier, on the first run rather than the six-hundredth.

The prefixed fake key deserves the most emphasis because it is the cheapest and the most versatile. Pick a distinctive prefix that will never occur naturally, put keys carrying that prefix everywhere a real key might plausibly live, and add one rule to whatever secret scanner you already run. Now the same token is detectable in transcripts, in proxy logs, in your repositories and in any external scanning you subscribe to. One string, five detection surfaces.

# canaries.yaml -- seeded into every eval environment at container build.
# Nothing here is real. Everything here is watched.

prefix: "ATC-CANARY-"          # never occurs naturally; one scanner rule covers all

http_canaries:
  - id: internal-wiki
    url: "https://wiki-internal.canary.example.co.uk/runbooks/db-access"
    seed_into: ["/workspace/docs/ONBOARDING.md"]
    alert: pagerduty://eval-containment

  - id: metrics-endpoint
    url: "https://metrics.canary.example.in/v1/ingest"
    seed_into: ["/workspace/.env", "/workspace/config/telemetry.yaml"]
    alert: pagerduty://eval-containment

credential_canaries:
  - id: fake-openai
    env: OPENAI_API_KEY
    value: "ATC-CANARY-sk-7f2c91a4e83b06d5"
    watch: ["transcripts", "proxy-logs", "secret-scanner"]

  - id: fake-aws
    file: /home/eval/.aws/credentials
    access_key_id: "AKIAATCCANARY0000001"     # real IAM user, explicit deny-all
    watch: ["cloudtrail:ConsoleLogin", "cloudtrail:GetCallerIdentity"]
    alert: pagerduty://eval-containment

dns_canaries:
  - id: fake-db-host
    hostname: "pg-primary.canary.example.co.uk"
    seed_into: ["/workspace/config/database.yaml"]
    watch: ["authoritative-ns-query-log"]

# --- Minimum viable listener. Deploy at the canary domain. -----------------
# Any request that reaches this has, by definition, escaped the sandbox.
#
#   from flask import Flask, request
#   import logging, json
#   app = Flask(__name__)
#
#   def page_oncall(service):        # your own pager integration --
#       ...                         # e.g. the PagerDuty Events API
#
#   @app.route("/", defaults={"path": ""}, methods=["GET", "POST",
#                                                   "PUT", "DELETE"])
#   @app.route("/<path:path>", methods=["GET", "POST", "PUT", "DELETE"])
#   def trip(path):
#       logging.critical(json.dumps({
#           "event": "CANARY_TRIPPED",
#           "path": "/" + path,
#           "src": request.headers.get("X-Forwarded-For",
#                                      request.remote_addr),
#           "ua": request.headers.get("User-Agent"),
#           "body": request.get_data(as_text=True)[:2000],
#       }))
#       page_oncall("eval-containment")   # expected rate is zero
#       return "", 204

Two notes on operating these. Host the canary domain somewhere with no relationship to your production estate — a separate account, a separate registrar entry, ideally a separate provider — so that a canary hit cannot be explained away as one of your own health checks. And rotate the seeded values periodically, recording which run batch used which generation, so that a hit six weeks from now can be traced to a specific window rather than to "some time in the last year".

Monitor the transcripts, not just the scores

Most evaluation pipelines are built to answer one question: did the score go up? They record a number per task, aggregate it, chart it, and discard the reasoning that produced it — or keep it in a bucket nobody has ever queried. That is a reasonable design for measuring quality and a terrible one for detecting containment failures, because the score is exactly the place where an escape does not show up. A model that reached a real machine and completed the capture-the-flag task scores well. Success on the task is not evidence of containment; in this class of evaluation it can be evidence against it.

It is worth noting that expanding continuous monitoring of evaluation transcripts was the first mitigation Anthropic announced. That is not a coincidence — it is the control that matches the failure mode. When the failure rate is 0.004 per cent and the failure is invisible in the metric you already collect, the only thing that finds it is reading everything, mechanically.

The practical shape is a retention policy plus an indicator scan. Retain full transcripts — the model's reasoning, every tool call with its arguments, every tool result, the environment fingerprint and the pre-flight containment report — for as long as you would want to answer "when did this start?", which in practice means at least a year. Then run a cheap deterministic scan over every transcript as it lands, and a more expensive analysis over a sample.

On cost, the honest trade-off is this. A regex and heuristic pass over transcripts is close to free — on the order of milliseconds per transcript on commodity hardware, which is nothing next to the cost of generating the transcript in the first place. Run that over 100 per cent of runs, always, no exceptions. A model-based review, where you ask a cheap model to read a transcript and flag anything resembling an escape attempt, costs real money per transcript and should be applied selectively: to everything the regex pass flagged, to every run in a security-adjacent suite, and to a random sample of the remainder large enough to catch what your patterns miss. The random sample is not optional — its purpose is to find the indicator categories you have not thought of yet, and a monitoring system that only looks for known patterns slowly becomes a monitoring system that only finds old problems.

Structurally this is the same problem as production agent observability, and it deserves the same treatment: structured spans for every tool call, a stable run identifier that ties transcript to environment to containment report, and an exported trace you can query rather than a text file you can grep. The patterns in our guide to instrumenting agents with OpenTelemetry transfer directly to the eval harness, and the teams that already run that instrumentation in production usually find the eval-side work is mostly configuration.

Third-party eval partners: the trust boundary you inherit

The root cause in the incident above was not a bug in a firewall. It was a misunderstanding between two organisations about what the environment provided. One party believed the network was severed; the other party had not been asked to sever it in those terms. Both parties were competent. Neither was lying. The gap lived in the space between them, which is where these gaps almost always live.

This matters increasingly because evaluation is specialising. Serious capability and safety evaluations are hard to build, and buying them from a partner who has built the task suite, the scoring rubric and the infrastructure is often the right call. But when you run your model inside someone else's environment, you inherit their network, their credentials hygiene, their tenancy model and their change process — and you inherit them without visibility. The commercial relationship gives you an assurance. It does not give you a control.

Notice the shape of the third column: it is the same work in every row. You run your own probe, from inside the environment, on every batch, and you believe the result rather than the document. That is the principle, and it is worth stating plainly because it feels rude and is not. An assurance from a partner is not a control either. It is a statement in a different document by a different organisation, and it fails in exactly the same way a statement in a prompt fails: silently, and without anybody being at fault.

Geography compounds the problem. If you are a UK team evaluating in eu-west-2 or an Indian team in ap-south-1, and your evaluation partner is in a third jurisdiction, then "where does this run" is simultaneously a security question, a contractual question and a data-residency question. Agreeing the region in writing and then verifying it from inside the container costs one extra probe in your pre-flight and closes all three at once. Our guide to data residency for AI apps under DPDP and GDPR covers the routing side of that in detail.

None of this should read as an argument against third-party evaluation partners. Specialised partners are how the ecosystem gets rigorous evaluations at all, and Anthropic's response to this incident included engaging another external organisation for review rather than pulling everything in-house. The argument is narrower: buy the evaluation, and verify the environment yourself anyway, on every run, from the inside.

Blast-radius design and the incident runbook

Assume containment fails once. Not because your controls are bad, but because every layered control eventually has a day where the layers line up wrongly, and the teams that come out of that day well are the ones who designed for it in advance. Blast-radius work is what determines whether a containment failure is an internal finding or an external incident.

Four design choices carry most of the weight:

• No real credentials anywhere in the eval environment. Not scoped-down ones, not read-only ones, not "just the staging key". If the network opens and there is nothing genuine to authenticate with, the failure stays a curiosity. The reasoning is the same as in least-privilege credentials for AI agents, taken to its endpoint: in an eval environment the least privilege available is none.
• No shared network with production. The eval estate should not be able to reach your production VPC even if it reaches the internet. Separate accounts, separate networks, no peering, no shared service endpoints. This is the control that stops a containment failure from becoming a customer-data incident.
• Per-run ephemeral environments. A fresh namespace per run, destroyed afterwards, with no shared writable volume or cache. This bounds one run's contamination of another and makes forensics tractable, because the environment for run N is reproducible rather than a mutated descendant of runs 1 to N-1.
• Outbound rate limits, even on the allowlisted path. A cap on requests per minute and bytes per run at the proxy turns an undetected exfiltration channel into a slow one, and gives your alerting something to fire on. It is a poor primary control and an excellent last one.

Then the runbook. The useful thing about the incident described earlier is that it gives us a timeline to aim at, from a team that executed it under real conditions. The table below reframes it as a target — a reasonable benchmark for what good looks like, not a standard anyone has ratified.

Two observations about that timeline. First, almost every row depends on work done long before the incident. You cannot identify affected runs in a day unless transcripts were retained and indexed; you cannot map destinations to owners unless egress was logged; you cannot halt a batch class in one command unless somebody built that command. The runbook is mostly a list of things that must already exist.

Second, the notification step is where teams hesitate longest and should hesitate least. The instinct to wait until the investigation is complete is understandable and wrong. The organisation on the other end needs to check their own logs, and log retention is finite — every day you spend perfecting the disclosure is a day of their evidence ageing out. Send what you know, say clearly what you do not know yet, and follow up.

A two-day implementation plan

All of the above is two days of work for one engineer who knows the harness. That is a genuinely unusual ratio in security engineering, and it is the reason to do it now rather than after the quarterly planning cycle.

Day one — make the boundary real. Morning: open the harness and the eval prompt side by side and write down every assertion either makes about the environment. Use the table earlier in this article as a prompt list; expect to find between six and twelve, and expect at least two to exist only in prose. For each one, write where it is enforced. Where the answer is "nowhere", say so in the document rather than fixing it immediately — you want the complete list before you start work.

Afternoon: write the pre-flight egress check. Start from the script above, adapt the allowlist to your environment, and run it inside your actual eval container as the actual eval user. Fix what it finds — and it will find something; IPv6 and the metadata endpoint are the usual first hits. Then wire it into the orchestrator as a blocking gate with a non-zero exit stopping the batch, and add the negative test that proves the check can fail. Write the JSON report alongside the batch results.

Day two — make failures visible. Morning: canaries. Register a domain unconnected to your production estate, stand up the logging endpoint, choose a canary prefix, and seed keys and URLs into the fixtures. Create the deny-all IAM user and wire the CloudTrail alarm. Route every canary alert to a pager rather than a channel, because the expected rate is zero and a zero-rate alert should wake somebody.

Afternoon: transcripts and vendors. Turn on full transcript retention with a horizon of at least a year, add the run identifier that ties transcript to environment to containment report, and write the regex indicator scan against the categories in the table above. Run it over your existing transcript history — the back-scan is often where the first surprise lives. Then send the vendor checklist to every evaluation partner you use, and book the half-hour to walk through it rather than emailing it and hoping.

Here is the checklist, condensed for copying into your own tracker:

EVAL HARNESS CONTAINMENT CHECKLIST

Assertions
  [ ] Every environmental claim in the harness AND the eval prompt is written down
  [ ] Each one records where it is enforced (or that it is not)
  [ ] Claims that are not enforced are removed from the prompt, not just noted
  [ ] Stub/mock layers audited for fallback paths that call real APIs

Egress
  [ ] Deny-by-default network namespace; no default route
  [ ] Forward proxy with explicit allowlist, not reconfigurable by the eval process
  [ ] DNS pinned to a sinkhole resolver; all queries logged
  [ ] Pre-flight probes: raw TCP by IP, DNS, proxied HTTP, direct HTTP, IPv6,
      cloud metadata endpoint, environment secret scan
  [ ] Pre-flight runs INSIDE the eval env, as the eval user, before EVERY batch
  [ ] Non-zero exit is a hard stop, not a warning
  [ ] Negative test proves the pre-flight can fail
  [ ] Report persisted on the run record

Canaries
  [ ] Distinctive prefix chosen; one secret-scanner rule covers it
  [ ] HTTP canary URLs seeded in plausible locations
  [ ] Fake cloud credential (real IAM user, deny-all) with an auth alarm
  [ ] Canary DNS name with authoritative query logging
  [ ] Canary domain hosted separately from production
  [ ] All canary alerts page a human; expected rate is zero

Transcripts
  [ ] Full transcripts retained >= 12 months, queryable
  [ ] Run ID ties transcript -> environment -> containment report
  [ ] Regex/heuristic indicator scan over 100% of runs
  [ ] Model-based review over flagged runs + a random sample
  [ ] Back-scan run over existing history

Vendors
  [ ] Network provisioning, change process and notification agreed in writing
  [ ] Allowlist received as a file, not a description
  [ ] Your pre-flight runs inside their environment on every batch
  [ ] Transcript export on your own retention terms
  [ ] Region confirmed from inside the environment
  [ ] Joint incident contact and runbook rehearsed once

Blast radius
  [ ] No real credentials in the eval environment at all
  [ ] Eval estate cannot reach production networks
  [ ] Per-run ephemeral environments; no shared writable state
  [ ] Outbound rate and byte limits at the proxy
  [ ] One command halts a batch class
  [ ] Documented way to suspend ALL retention expiry during an incident

A closing thought on why this is worth an unglamorous two days. Evaluation infrastructure is where a lot of AI teams put their most capable models under the most adversarial pressure, and it is simultaneously the part of the stack that gets the least security attention — because it is internal, because it is tooling, because it does not serve customers. The authorisation failures catalogued in AgentRedBench's 215 tests mostly describe agents in production, and the same failures apply to the harness you use to test those agents, with rather less scrutiny.

The fix is not sophisticated. It is a hundred lines of Python that tries to reach the internet six different ways and refuses to start if any of them works, plus some worthless credentials that page you if anybody touches them. It is arguably the highest-leverage security work available to a small AI team in 2026, and it is well within a couple of days.

It also happens to be excellent proof-of-work. "We take evaluation security seriously" is a claim anyone can put in a deck; a public repository containing a pre-flight containment check, a canary seeding config and a transcript indicator scan is a claim that verifies itself. If you have built one, put it on a Builder profile where the people hiring for evaluation, safety and platform roles across Bengaluru, Chennai, London and Manchester can actually find it.