Testing Agent Reliability: pass^k, Perturbation and Fault Injection

What you need to know

• Accuracy is a single-sample estimate. Reliability is a distribution. One number cannot describe the other.
• Consistency is measured with pass^k — the probability that all k independent runs of the same task succeed. It falls as k grows, which is exactly what an unsupervised agent experiences.
• Robustness is measured by perturbing inputs into semantically equivalent variants and watching the success rate move. Users do not speak in your canonical phrasing.
• Fault tolerance is measured by injecting tool and API failures deliberately. Rate limiting is the most damaging class and the one you will meet first.
• All three fit in CI if you gate on the aggregate metric against a recorded baseline rather than on individual runs.

As of August 2026 this framing is becoming standard in the literature. ReliabilityBench (arXiv 2601.06112) proposes exactly these three axes with the notation used throughout this guide — pass^k for consistency, perturbation intensity ε for robustness, fault injection rate λ for fault tolerance. A parallel line of work on the science of agent reliability adds two further dimensions, predictability and safety, which are covered briefly at the end. Note that ReliabilityBench is a single-author preprint evaluating a specific model and scaffold set, so treat its individual figures as illustrative rather than as constants. The method is the durable part.

Pre-requisites

Before any of this is worth doing, you need three things in place:

1. A deterministic grader. If you cannot decide programmatically whether a run succeeded, you cannot run the same task fifty times. An LLM judge counts, provided it is itself stable — see our guide to LLM-as-judge evals in production for calibrating that.
2. A task set with real outcomes. Twenty to fifty tasks drawn from actual usage beats two hundred synthetic ones. Our guide to building evals from production logs covers sourcing them.
3. A tool layer you can intercept. Fault injection requires that every outbound call goes through something you control. If your agent calls SDKs directly, wrap them first.

Why one number is misleading: the arithmetic

Start with the simplest possible correction. Suppose your agent succeeds on a task 90% of the time. Now ask what happens when it runs that task unsupervised, five times, and every one has to be right.

pass^k for a task with independent per-run success probability p. Each cell is p to the power k.

Per-run success p	pass^3	pass^5	pass^10
0.80	0.512	0.328	0.107
0.90	0.729	0.590	0.349
0.95	0.857	0.774	0.599
0.99	0.970	0.951	0.904

A 90% agent gets everything right across ten runs about a third of the time. That is the same agent, the same tasks, and a very different impression of whether it is ready to ship.

The same arithmetic applies within a single long task. A twenty-step agent trajectory where each step independently succeeds 99% of the time completes cleanly about 82% of the time. This is why long-horizon agents feel so much less reliable than their per-step numbers suggest, and why adding steps to a workflow is never free.

pass@k and pass^k are not the same metric

This trips people up constantly, so state it plainly. pass@k asks whether at least one of k attempts succeeded. It comes from code generation, where a human or a test suite selects the best candidate, and it rises as k grows. pass^k asks whether all k attempts succeeded, and it falls as k grows.

If your agent runs unsupervised, nobody is picking the best of five. Every run ships. pass@k is measuring a workflow you do not have.

Axis 1 — Consistency

The harness for this is short. Run each task k times with independent sampling, grade each run, and report both the mean success rate and the all-pass rate.

from dataclasses import dataclass
from statistics import mean

@dataclass
class TaskResult:
    task_id: str
    runs: list[bool]

    @property
    def mean_success(self) -> float:
        return mean(1.0 if r else 0.0 for r in self.runs)

    @property
    def all_pass(self) -> bool:
        return all(self.runs)

    @property
    def is_flaky(self) -> bool:
        # Passed at least once and failed at least once.
        return any(self.runs) and not all(self.runs)


def run_consistency(tasks, agent, grader, k=5) -> list[TaskResult]:
    results = []
    for task in tasks:
        runs = []
        for _ in range(k):
            # Fresh session per run. No shared memory, no cached state.
            output = agent.run(task.prompt, session=agent.new_session())
            runs.append(grader(task, output))
        results.append(TaskResult(task.task_id, runs))
    return results


def report(results, k):
    n = len(results)
    print(f"tasks={n}  k={k}")
    print(f"mean success   : {mean(r.mean_success for r in results):.3f}")
    print(f"pass^{k} (all runs): {sum(r.all_pass for r in results) / n:.3f}")
    print(f"flaky tasks    : {sum(r.is_flaky for r in results)} / {n}")
    for r in sorted(results, key=lambda r: r.mean_success)[:5]:
        print(f"  worst: {r.task_id}  {r.runs}")

Three outputs matter, and the third is the one people skip.

• Mean success — the number you were already reporting.
• pass^k — the number that predicts unsupervised behaviour.
• The flaky task list — tasks that pass sometimes and fail sometimes. This list is a work queue. A task that fails every time is a capability gap; a task that fails intermittently is usually an ambiguity, a race, or a tool that returns inconsistently, and those are fixable this week.

Axis 2 — Robustness to perturbation

Your eval set was written by you. It phrases every request the way you think about the problem. Real users paraphrase, reorder constraints, add irrelevant background, make typos, correct themselves mid-sentence, and — in both Indian and British contexts — code-switch or use regional phrasing that never appears in a canonical test set.

Robustness testing applies semantically equivalent perturbations at an intensity ε and measures how far success drops. ReliabilityBench reports success falling from 96.9% at ε=0 to 88.1% at ε=0.2 in its own setup — roughly a nine-point drop from paraphrasing alone, on models and scaffolds specific to that study.

The perturbation classes worth having

Perturbation classes, ordered by how often they appear in real traffic.

Class	Example transformation	What it tests
Paraphrase	"Book me a flight to Delhi on Friday" → "I need to get to Delhi, Friday works"	Intent extraction independent of surface form
Constraint reorder	Move the budget limit from the end of the request to the middle	Whether attention to constraints is position-dependent
Irrelevant context	Prepend two sentences of unrelated background	Distractor resistance
Self-correction	"Book Tuesday — sorry, I meant Wednesday"	Whether the superseded instruction is dropped
Typos and casing	Realistic keyboard-adjacent errors, lowercase throughout	Tokenisation sensitivity
Register and locale	Formal British phrasing vs casual Indian English; "cheque" vs "check"	Whether the agent generalises across your actual user base

That last row is the one teams serving both India and the UK consistently under-test. If your eval set is written in one register, you have measured the agent's performance for one half of your users.

Generating the set without poisoning the measurement

Use a model to generate candidate perturbations, then freeze them. The critical rule: the perturbation set must be a fixed fixture, verified once, and checked into the repository.

# Generate ONCE, review, commit. Do not regenerate per run.
#
#   python -m evalkit.perturb \
#       --tasks evals/tasks.jsonl \
#       --classes paraphrase,reorder,distractor,selfcorrect,typo,register \
#       --intensity 0.2 \
#       --out evals/perturbations.jsonl
#
# Then a human reviews a sample for semantic equivalence
# and the file becomes an immutable fixture.

def run_robustness(tasks, perturbations, agent, grader, k=3):
    baseline = run_consistency(tasks, agent, grader, k=k)
    scores = {"baseline": mean(r.mean_success for r in baseline)}

    for cls, variants in perturbations.items():
        perturbed = run_consistency(variants, agent, grader, k=k)
        scores[cls] = mean(r.mean_success for r in perturbed)

    for cls, s in scores.items():
        delta = s - scores["baseline"]
        flag = "  <-- investigate" if delta < -0.05 else ""
        print(f"{cls:14s} {s:.3f}  ({delta:+.3f}){flag}")
    return scores

If perturbations regenerate on every run, your robustness metric moves whenever the generator model changes, and you will spend a fortnight debugging an agent regression that was a generator regression. This is the same discipline that makes golden sets useful, and the same failure mode that makes them useless when they drift.

Axis 3 — Fault tolerance

Clean benchmarks assume every tool call returns valid data promptly. Production does not. Timeouts, rate limits, partial responses, malformed JSON, schema changes and outright 500s are routine, and an agent's response to them is almost never tested before it happens live.

This is chaos engineering applied to the tool layer. Inject faults at a controlled rate λ and measure how much of the success rate survives.

Fault classes to inject, and the agent behaviour each one exposes.

Fault	How to inject	Failure it exposes
Rate limit	HTTP 429, with and without Retry-After	Immediate retry loops that burn the budget on a wall
Timeout	Delay past the client deadline, then succeed	Duplicate side effects when the retry also lands
Transient 500	Fail n times, then succeed	Whether retry-with-backoff exists at all
Partial response	Truncate the payload mid-object	Parsing that assumes well-formed input
Schema drift	Rename a field, add an unexpected one	Brittle field access; silent misreads
Semantically wrong data	Valid schema, implausible values	Whether anything sanity-checks tool output

Rate limiting deserves to be first. ReliabilityBench found it the most damaging fault type in its evaluation, and the mechanism is intuitive: an agent that retries a 429 immediately consumes its remaining turns against a limit that has not reset, converting a recoverable delay into a total failure.

import random, json, time

class FaultInjector:
    """Wrap the tool layer. Inject faults at rate lam."""

    def __init__(self, inner, lam=0.1, classes=None, seed=0):
        self.inner = inner
        self.lam = lam
        self.classes = classes or ["rate_limit", "timeout", "transient_500",
                                   "partial", "schema_drift"]
        self.rng = random.Random(seed)   # seeded: reproducible chaos
        self.log = []

    def call(self, tool_name, **kwargs):
        if self.rng.random() < self.lam:
            fault = self.rng.choice(self.classes)
            self.log.append((tool_name, fault))
            return self._fault(fault, tool_name, kwargs)
        return self.inner.call(tool_name, **kwargs)

    def _fault(self, fault, tool_name, kwargs):
        if fault == "rate_limit":
            return {"status": 429, "headers": {"Retry-After": "8"},
                    "body": "rate limit exceeded"}
        if fault == "timeout":
            time.sleep(self.inner.deadline + 0.1)
            return self.inner.call(tool_name, **kwargs)
        if fault == "transient_500":
            return {"status": 500, "body": "internal error"}
        if fault == "partial":
            good = json.dumps(self.inner.call(tool_name, **kwargs))
            return {"status": 200, "body": good[: len(good) // 2]}
        if fault == "schema_drift":
            out = self.inner.call(tool_name, **kwargs)
            if isinstance(out, dict) and out:
                first = next(iter(out))
                out[f"{first}_v2"] = out.pop(first)
            return out
        raise ValueError(fault)

Seed the injector. Reproducible chaos is the entire point — an unseeded fault schedule means a failing CI run cannot be reproduced locally, and you will start ignoring the signal within a fortnight.

Putting it in CI without creating a flaky build

The objection to all of this is cost and flakiness. Both are manageable with three rules.

Gate on the metric, never on the run

A single failed run must never fail the build — the whole premise is that runs are stochastic. Gate on pass^k against a committed baseline with a tolerance band.

# evals/baseline.json  (committed artefact)
# {"pass_k": 0.78, "k": 5, "robustness_min": 0.71, "lambda_01": 0.69}

TOL = 0.05

def gate(current, baseline, tol=TOL):
    failures = []
    for key, base in baseline.items():
        if key == "k":
            continue
        got = current[key]
        if got < base - tol:
            failures.append(f"{key}: {got:.3f} < {base:.3f} - {tol}")
    if failures:
        raise SystemExit("reliability regression:\n  " + "\n  ".join(failures))
    print("reliability gate passed")

Split the tiers by cost

A three-tier schedule that keeps per-commit cost bounded.

Tier	When	Scope
Smoke	Every commit	10 tasks, k=3, λ=0 — catches gross breakage in minutes
Full	Nightly, and on every prompt or model change	All tasks, k=5, all perturbation classes, λ=0.1
Stress	Weekly and pre-release	k=10, λ sweep to 0.3, compound faults

Any change to a system prompt, a tool schema or a model version should trigger the full tier regardless of schedule. Those are the three inputs that move reliability most, and a model swap in particular can hold accuracy steady while consistency collapses — which is exactly the scenario our guide to shadow and canary deploys for model upgrades is designed to catch before users do.

Report a scorecard, not a number

RELIABILITY  agent=support-triage  model=  tasks=42

  consistency   mean 0.913   pass^5 0.786   flaky 9/42
  robustness    paraphrase 0.902  reorder 0.887  distractor 0.841
                selfcorrect 0.792  typo 0.898   register 0.864
  faults        lam=0.0 0.913   lam=0.1 0.694   lam=0.3 0.402

  baseline delta   pass^5 +0.011   distractor -0.062  <-- regression

Read across, not down. In that example the headline accuracy is healthy and unchanged, but distractor robustness has dropped six points and the fault curve falls off a cliff between λ=0.1 and λ=0.3. Neither is visible in a single accuracy number, and both would surface in production within a week.

The two dimensions this guide does not cover

Work on the science of agent reliability (arXiv 2602.16666) decomposes the property into four dimensions rather than three: consistency, robustness, predictability — whether the agent's confidence is calibrated — and safety — whether failures are bounded in severity when they do occur.

Both are worth building towards. Predictability matters because an agent that knows when it is unsure can escalate, and one that is uniformly confident cannot. Safety matters because your reliability numbers describe how often things go wrong, not how bad it is when they do, and those are independent properties — a 99% reliable agent whose 1% failure deletes production data is worse than a 90% agent whose failures are inert.

Common mistakes

1. Reporting mean success as though it were reliability. The single most common error, and the reason this guide exists.
2. Reusing sessions across repeated runs. Inflates consistency, silently.
3. Regenerating perturbations each run. Turns a fixed measurement into a moving one.
4. Unseeded fault injection. Produces failures nobody can reproduce, which trains the team to ignore them.
5. Testing faults only in isolation. Real incidents are compound — a rate limit during a retry storm while a downstream service is slow.
6. Building the harness and never wiring it to CI. A reliability suite run manually before releases is a reliability suite run twice.

Next steps

Build it in this order, because each stage is useful before the next one exists. First, add k=5 repetition to the eval suite you already have and record pass^5 as your baseline — that is an afternoon's work and it will change what you think your agent's quality is. Second, add a perturbation fixture, starting with paraphrase and distractor, which are the two highest-yield classes. Third, wrap your tool layer and inject rate limits, which is the fault you will actually meet.

From there, the natural companions are trajectory-level grading, covered in evaluating agents on trajectory, tool calls and outcome, and wiring the whole thing into a pipeline, covered in running evals in CI. When something does break in production, incident-response runbooks for LLM applications covers the response side.

One closing observation. Reliability work is unglamorous, rarely demoed, and almost never in a portfolio — which is precisely why a published account of a reliability harness you built, with the scorecard and the regressions it caught, stands out so sharply to anyone hiring. Evaluation literacy is now one of the first things interviewers probe for, and most candidates can only describe accuracy.

Reference material: ReliabilityBench (arXiv 2601.06112) and Towards a Science of AI Agent Reliability (arXiv 2602.16666). Figures quoted are as of August 2026.