Mutation Testing for Agent-Written Code

Ask a coding agent to add tests to a module and it will do it quickly and in a form that looks entirely convincing. The suite runs green. The coverage number goes up. Somebody screenshots the percentage for the sprint review. The question none of that answers is the only one that matters: if the code under test were wrong, would any of these tests have noticed?

That is not a rhetorical flourish. Researchers studying LLM-generated tests on the HumanEval-Java benchmark recorded a case where the generated tests achieved 100% line and branch coverage yet scored only 4% on mutation testing. The suite executed effectively every line and branch while being blind to almost every fault that could be injected, missing corner cases such as leap-year date handling. Perfect coverage. Near-zero fault-detection power. The same test files.

This guide is about measuring and fixing that gap. Mutation testing is the measurement; property-based testing is a large part of the fix. Both predate the current generation of coding agents by decades, which is precisely why they are worth the effort: techniques that were stable in 2005 will still be stable in 2030, whereas your agent's model version will have turned over a dozen times.

It sits alongside the other quality layers this site covers. AI code review in CI asks whether a change is sound; putting evals in CI asks whether a prompt or agent has regressed; the QA discipline that separates demos from products covers the whole pipeline. This one is narrower and more awkward: does the test suite guarding all of that detect faults at all?

One warning before the mechanics, because a lot of writing on this subject skips it. Mutation testing is slow — the most computationally expensive thing you can reasonably do to a codebase — and pretending otherwise is how teams end up switching it off in week three. Half this article is about making it cheap enough to keep.

The coverage lie: how 100% and 4% coexist

Coverage instrumentation does exactly one thing: it marks a line, branch or condition as visited when a test causes it to execute. That is the whole mechanism. It makes no claim about whether the value that line produced was ever inspected. A test file containing no assertions whatsoever will drive coverage to 100% as reliably as a rigorous one, provided it calls the functions. So will a suite that only asserts the code did not throw.

Once you internalise that, the HumanEval-Java result becomes almost predictable. The generated tests called the function with inputs the implementation obviously handled, asserted the values it obviously returned, and never probed the region where it might be wrong. Every line ran. Almost nothing was verified.

Metric	What it actually records	What it proves about fault detection	Cheapest way to hit 100%
Line coverage	This line executed during at least one test	Nothing. An assertion-free test achieves it	Call every function once and assert nothing
Branch coverage	Both outcomes of each decision point occurred	Slightly more: control flow was exercised in both directions, but never that either outcome was checked	One input per side of each if, still asserting nothing
Condition / MC-DC	Each sub-condition independently affected the outcome	Genuinely stronger on compound predicates; still silent on assertions	Hard to fake, but expensive and rare outside avionics and medical software
Mutation score	Share of deliberately introduced faults that made a test fail	Directly measures fault-detection power — the only one of the four that does	Cannot be faked without writing assertions that genuinely discriminate

Branch coverage deserves a closer look, because it is the metric most teams believe is rigorous. Consider a leap-year predicate: year % 4 == 0 and (year % 100 != 0 or year % 400 == 0). Test the years 2023 and 2024 and both outcomes of that decision occur, so branch coverage reports it fully covered. But 2024 short-circuits the inner or on its first operand, and 2023 short-circuits the outer and before the parenthesis is evaluated at all. The century rule — the part encoding the difficult knowledge — was never exercised, and no coverage tool in normal configuration will say so.

What mutation testing actually does

Mutation testing inverts the question. Instead of asking what your tests touched, it deliberately breaks the code and asks whether your tests complained.

The tool parses your source, applies a catalogue of small syntactic changes called mutation operators, and produces one mutant per change — a copy of the program with exactly one fault injected. It then runs your suite against each mutant. If some test fails, the mutant is killed. If every test still passes, the mutant survived, and you have located a fault your tests would ship. The mutation score is the proportion killed. Coverage answers "did this run?"; mutation score answers "would you have noticed?".

Operator class	Change applied	Example: original → mutant
Relational operator replacement	Swap a comparison for a neighbouring one	if (n >= limit) → if (n > limit)
Arithmetic operator replacement	Swap an arithmetic operator	total = price * qty → total = price / qty
Logical connector replacement	Swap and for or and vice versa	a and b → a or b
Conditional boundary / negation	Invert or short-circuit a condition	if (is_active) → if (not is_active), or the whole condition forced to True
Return value replacement	Replace a returned value with a default or neighbour	return 29 → return 28; return xs → return []
Statement removal	Delete a call whose effect should be observable	audit_log.write(event) → removed entirely

Take the leap-year case as a worked example, because it is the exact class of bug the HumanEval-Java study surfaced.

# billing/calendar.py — the function under test.

def days_in_month(year: int, month: int) -> int:
    if month == 2:
        if year % 4 == 0 and (year % 100 != 0 or year % 400 == 0):
            return 29
        return 28
    if month in (4, 6, 9, 11):
        return 30
    return 31


# Four of the mutants a tool would generate from the February branch:
#
#   M1  relational          year % 4 == 0     ->  year % 4 != 0
#   M2  logical connector   ... and ...       ->  ... or ...
#   M3  condition removal   year % 100 != 0   ->  True
#   M4  return value        return 29         ->  return 28

Now the test suite a coding agent typically produces when pointed at that function, and what mutation testing says about it.

# test_calendar_weak.py — reaches 100% line AND 100% branch coverage.

def test_february_leap():        assert days_in_month(2024, 2) == 29
def test_february_common():      assert days_in_month(2023, 2) == 28
def test_thirty_day_month():     assert days_in_month(2024, 4) == 30
def test_thirty_one_day_month(): assert days_in_month(2024, 1) == 31

# Mutation result: M1 killed (2024 -> 28, expected 29).
#                  M2 killed (2023 -> 29, expected 28).
#                  M4 killed (2024 -> 28, expected 29).
#                  M3 SURVIVES. No test passes a century year, so forcing
#                  `year % 100 != 0` to True changes nothing observable.


# test_calendar_strong.py — two examples chosen because a mutant survived,
# not because the implementation suggested them.

def test_century_not_divisible_by_400(): assert days_in_month(1900, 2) == 28
def test_century_divisible_by_400():     assert days_in_month(2000, 2) == 29

# M3 now killed: under the mutant, 1900 returns 29 and the assertion fails.
# Coverage did not move. Fault-detection power did.

That last comment is the entire argument of this article compressed into two lines. The strengthened suite has identical coverage to the weak one and is materially better. No coverage-derived metric can express the difference; the mutation score can.

Equivalent mutants: the tax you must plan for

The technique has one genuine, unavoidable flaw. Some mutants are semantically identical to the original — they change the source without changing observable behaviour, so no test can possibly kill them. Replacing i < n with i != n in a loop that increments by one; changing an initial value that is unconditionally overwritten; mutating inside an unreachable branch. These are equivalent mutants, and detecting them in general is undecidable.

Two practical consequences. Your mutation score has a ceiling below 100% that nobody can compute, so treating 100% as the target is a category error. And every survived mutant costs a human a minute or two of judgement to classify as "real gap" or "equivalent, ignore" — triage time that is the real tax on the technique, more than the CPU time, and the reason the rollout below starts with one module rather than a repository.

Why agent-written tests fail in this specific way

There is a structural reason agent-written suites score badly here, and it is not carelessness. It is that an agent writes tests against the code that exists, not against the specification the code was meant to satisfy. Given a function, the model infers intent from the implementation, then asserts that the implementation does what the implementation does. That produces characterisation tests: excellent for pinning behaviour during a refactor, close to worthless as a correctness check.

The consequence is uncomfortable. Had days_in_month omitted the century rule, an agent reading that code would have written assert days_in_month(1900, 2) == 29, enshrining the bug in a passing test. The suite is not merely failing to catch the fault; it is defending it against future correction. This is why spec-driven development with coding agents matters more for tests than for implementation code: an agent given the specification can write tests that disagree with the code, and an agent given only the code cannot.

What agents actually produce is now reasonably well documented. The study "Testing with AI Agents: An Empirical Study of Test Generation Frequency, Quality, and Coverage" (arXiv:2603.13724) analysed 2,232 commits containing test-related changes from the AIDev dataset across 10 TypeScript repositories using Vitest, measuring coverage across 531 test-adding commits in three of those projects. Its findings, all attributable to that paper:

Dimension	AI-authored tests	Human-authored tests	What it implies for mutation score
Share of test-adding commits	16.4% overall; 83–100% in small projects (5–28 contributors)	Dominant in large projects — AI share only 1.9–14.4% at 608+ contributors	Small teams are already relying on agent-written suites almost entirely
Size (effective lines of code)	Median 12.0; maximum 87 — a narrow band	Median 11.0; extreme outliers up to 699 eLoC	Consistent, but consistency is not discrimination
Assertion density	Median 2.0 assertions per test	Median 1.0 assertion per test	More assertions, but the paper flags Assertion Roulette risk
Cyclomatic complexity	Median 1.0; mean 1.09; rarely above 2–3	Median 1.0; mean 1.31	Linear, straight-line tests explore fewer awkward paths
Statement coverage delta	Liam +0.072; AppKit +0.030; Helper 0.000	Liam −0.090; AppKit −0.001; Helper 0.000	Small and project-dependent; not a quality verdict either way

Two honest caveats. First, on branch coverage the same paper reports AppKit at AI +0.183 against human +0.008 — a real and favourable difference for the AI-authored tests, so the picture is not uniformly negative. Second, and more importantly: that study explicitly states that mutation testing and long-term quality were not assessed. It measured frequency, structural quality and coverage, not fault-detection power, and any writing that implies otherwise is over-reading it. The link between linear, low-complexity tests and weak mutation scores is a reasoned inference from the mechanism, not a finding of that paper.

A different route suggests purpose-built generation can do better than a general-purpose agent — with the heavy caveat that the source is a vendor reporting on its own product. Diffblue published a report describing Diffblue Cover reaching 81% average line coverage and 61% mutation coverage autonomously across 8 Java projects, against 32% line coverage achieved by a senior developer using an AI coding agent. Read it as a vendor's own benchmark of its own tool, not independent evidence. The structurally interesting part is the choice to report mutation coverage at all — a tacit admission from inside the market that line coverage does not settle the question.

None of this argues against letting agents write tests — given the volume they now produce, refusing their help is not a real option, and the same asymmetry drives everything from legacy migrations with coding agents to routine feature work. It argues for a verification layer underneath. Stack Overflow's 2025 developer survey found only 32.7% of developers trust the accuracy of AI output. That scepticism is rational and almost entirely unmeasured — mutation testing is the cheapest way to replace it with a number a build log can carry.

Property-based testing as the complement

Mutation testing is diagnostic. It tells you a mutant survived at line 47 and therefore that your suite is weak there. It does not tell you what test to write, and inventing a killing input is exactly the work an agent is bad at, because the input you need is by definition one the implementation did not suggest.

Property-based testing attacks the same weakness from the other side. Rather than enumerating examples, you state a property that must hold for all inputs, and the framework generates hundreds of cases, including the awkward ones: zero, empty, maximum, negative, Unicode, century years. When it finds a failure it shrinks the counter-example to the smallest input that still fails — usually the bug in its clearest form.

Property pattern	Shape of the assertion	Good fit for
Round trip	decode(encode(x)) == x	Serialisers, cursors, parsers, compression, ID encoding
Invariant	A stated fact holds after any operation	Balances never negative; sorted output stays sorted; totals reconcile
Independent oracle	Result matches a slower, obviously-correct computation	Date and interval maths, pricing, tax and VAT/GST rules
Metamorphic	A known input change produces a known output change	Search ranking, discounts, currency conversion, idempotency

The independent-oracle pattern is the one that kills the leap-year class of bug outright, because it removes your implementation from the loop entirely.

# test_calendar_property.py — Hypothesis (Python), as of 2026.
from datetime import date
from hypothesis import given, strategies as st
from billing.calendar import days_in_month

@given(
    year=st.integers(min_value=1, max_value=9998),
    month=st.integers(min_value=1, max_value=12),
)
def test_matches_independent_calendar_arithmetic(year, month):
    """The number of days in a month is the distance to the first of the
    next month. The standard library computes that without our leap rule,
    so it is a genuine oracle rather than a restatement of the code."""
    first = date(year, month, 1)
    next_first = (date(year + 1, 1, 1) if month == 12
                  else date(year, month + 1, 1))
    assert days_in_month(year, month) == (next_first - first).days


// cursor.property.test.ts — fast-check (TypeScript), as of 2026.
// Round-trip property: whatever we encode must decode back identically.
import fc from 'fast-check';
import { encodeCursor, decodeCursor } from '../src/pagination';

test('pagination cursors round-trip', () => {
  fc.assert(fc.property(
    fc.record({ id: fc.uuid(), createdAt: fc.date({ min: new Date(0) }) }),
    (page) => {
      expect(decodeCursor(encodeCursor(page))).toEqual(page);
    },
  ));
});

Run that Python property against the M3 mutant and it fails within a few dozen generated cases, because the generator has no reason to avoid 1700, 1800 or 1900. That is the whole value: the framework is not reading your implementation, so it is not biased by it.

Agents are good at writing property-based tests once you name the property; what they are poor at is choosing which property matters, because that requires the specification. State the property yourself and let the agent write the generator and the plumbing — getting that instruction to stick is covered in AGENTS.md and CLAUDE.md files that actually steer agents.

Wiring it into CI without destroying build times

Here is the arithmetic that decides whether any of this survives contact with your pipeline. A run generates roughly one mutant per few lines of mutable code, and each mutant requires executing a test suite. A thousand mutants against a two-minute suite is thirty-three hours of naive compute. Nobody pays that on every pull request. A four-person team in Pune on a free-tier runner and a forty-person team in Leeds on self-hosted runners hit the same wall from opposite directions — one runs out of minutes, the other runs out of the afternoon — and both switch the job off if it is not scoped.

Every practical deployment therefore combines four reductions. Coverage-based test selection is the big one and modern tools do it automatically: a mutant on line 47 needs only the tests that execute line 47. Diff scoping mutates only files the branch changed, which on a normal pull request is a handful. Sampling runs a random subset of mutants for an estimate rather than an exact score. And deferral moves the exhaustive sweep to a nightly job that blocks nobody.

Run strategy	Scope	Where it belongs	Honest limitation
Diff-scoped (incremental)	Only files changed against the base branch, reusing cached results elsewhere	Every pull request; target a few minutes	Blind to suites weakened by changes elsewhere; cache can go stale after refactors
Sampled	A random percentage of all mutants	Merge to main, or large PRs touching many files	An estimate with variance — never gate a tight threshold on a sampled score
Full sweep	Every mutant in the gated modules	Nightly or weekly, off-peak, nobody waiting	Expensive; results arrive after the change has already merged
Module-gated	Full run, but only over designated critical packages	The pragmatic default for most teams	Says nothing at all about the rest of the codebase, and should not pretend to

# .github/workflows/mutation.yml — diff-scoped on PRs, full sweep nightly.
# Flags shown are illustrative of Stryker as of 2026; check current docs.
name: mutation

on:
  pull_request:
  schedule:
    - cron: "0 21 * * *"     # 21:00 UTC — 02:30 IST, 22:00 BST. Off-peak both.

jobs:
  diff-scoped:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    timeout-minutes: 20            # a hard ceiling, so it can never hold a PR
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }   # full history so the merge base resolves
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci

      - name: Restore incremental mutation state
        uses: actions/cache@v4
        with:
          path: reports/mutation/stryker-incremental.json
          key: stryker-incremental-${{ github.base_ref }}

      - name: Mutate only what this branch changed
        run: npx stryker run --incremental --since=origin/${{ github.base_ref }}

      - name: Enforce the ratchet
        run: ./scripts/mutation-ratchet.sh

  full-sweep:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    timeout-minutes: 180
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }
      - run: npm ci
      - run: npx stryker run --concurrency 4    # report only; never blocks

Note what the nightly job does not do: fail anything. Its output is a trend line and a list of survived mutants for triage. A job that wakes people up is a job people delete.

Language	Mutation testing (as of 2026)	Property-based testing (as of 2026)
JavaScript / TypeScript	Stryker	fast-check
Java / JVM	PIT (Pitest)	jqwik
Python	mutmut, cosmic-ray	Hypothesis
.NET	Stryker (.NET edition)	Ports of the property-based approach exist; check maturity before adopting

Treat that table as a starting point for a search, not a recommendation, and re-check it before adopting anything: names in this space are unusually stable, but flags, defaults and maintenance status all drift. The general principle from our comparison of Claude Code, Cursor and Codex — that the methodology should survive the vendor swap — holds doubly for testing infrastructure you intend to keep for years.

Choosing a gate that does not become theatre

A mutation score in a build log is interesting. A mutation score wired to a merge decision is a policy, and policies get gamed. Four rules keep the gate honest.

Ratchet, never a constant. Do not pick 80% because it sounds rigorous. Measure what your chosen module scores, commit that number to a file, and fail the build only when a change drops below it. When the score improves durably the floor rises — in a commit, visible in review, with a name attached. A ratchet starting at 55% and climbing three points a quarter delivers more real safety than an aspirational 85% gate the team routes around. It also handles equivalent mutants gracefully: whatever the unknowable ceiling is, you are measuring movement towards it.

#!/usr/bin/env bash
# scripts/mutation-ratchet.sh
# The floor only ever rises, and only in a commit a human can see.
set -euo pipefail

FLOOR=$(cat .mutation-floor)                                  # committed
SCORE=$(jq -r '.mutationScore' reports/mutation/report.json)

# 0.5 points of slack absorbs ordinary run-to-run noise, nothing more.
awk -v s="$SCORE" -v f="$FLOOR" 'BEGIN { exit !(s + 0.5 >= f) }' || {
  echo "FAIL: mutation score ${SCORE} is below the floor ${FLOOR}"
  echo "      a survived mutant means a fault this suite would ship"
  exit 1
}

# Raise the floor only with two points of headroom, so a lucky run does
# not lock in a threshold the next honest run cannot meet.
NEW=$(awk -v s="$SCORE" 'BEGIN { printf "%d", int(s - 2) }')
if [ "$NEW" -gt "$FLOOR" ]; then
  echo "$NEW" > .mutation-floor
  echo "NOTE: floor raised ${FLOOR} -> ${NEW}. Commit .mutation-floor."
fi

Exclude honestly, in the open. You will need exclusions — generated code, thin adapters, equivalent mutants classified by hand. Put every one in a version-controlled config with a written reason and a date, exactly as with a linter suppression. An exclusion list nobody reads becomes a place to hide failures, and the first survived mutant somebody quietly excludes is the moment the gate becomes theatre.

Assume the metric will be optimised, possibly by a machine. Tell an agent to raise the mutation score and it will succeed, sometimes in ways you did not want: assertions on private internals that make refactoring painful, whole-object snapshots that kill every mutant while expressing no intent, or a dozen assertions crammed into one test. That last one has empirical support — the AIDev study found AI tests carried a median of 2.0 assertions against 1.0 for humans, and named Assertion Roulette as the specific risk that trade-off carries.

Gate one thing well. A mutation gate over the payments module, the permissions layer or the tax calculator is credible and affordable. One over the whole repository is neither, and will be turned off. A named subset is more honest than a repository-wide number averaging a rigorous core with a directory of scripts.

A pragmatic rollout in seven steps

1. Pick one module where a bug costs money. Billing, permissions, date and interval logic, anything computing a total someone will be charged. Small enough to run in minutes, important enough to justify the effort.
2. Get a baseline before you change anything. Run the tool locally over that module and write the number down. Expect it to be lower than your coverage figure suggests, and expect that to be uncomfortable. That discomfort is the finding.
3. Triage the survivors once, by hand. Sort them into real gaps, equivalent mutants and genuinely-untested-but-unimportant. Budget an afternoon. This pass teaches you more about your suite than any dashboard, and you only do it in full once.
4. Kill the top ten survivors with real tests. Start with mutants on branches that touch money, permissions or data deletion. For each, ask what input distinguishes the mutant from the original — that input is your missing test case.
5. Add two or three properties over the same module. Use the pattern table above: a round trip, an invariant, an independent oracle. Properties tend to kill clusters of mutants at once, and they keep working as the implementation changes.
6. Wire diff-scoped mutation into pull requests with a hard timeout. Report only at first — no failing builds for the first two weeks, so the team sees the numbers before the numbers can block them.
7. Commit the floor and turn the ratchet on. Add the nightly full sweep as a trend line at the same time, and review survived mutants monthly rather than daily. When the module has held its floor for a quarter, pick a second module and repeat.

The same measurement discipline applies as in benchmarking coding agents on your own repository: a private, repository-specific number you trust beats a public score you cannot audit. The contamination logic from why public eval scores lie has a direct analogue — a metric the system under test has effectively memorised stops measuring anything.

When not to use it

Mutation testing is a poor fit far more often than its advocates admit, and knowing where to stop is what stops it becoming a ritual.

Prototypes and throwaway code. If the code will be deleted or rewritten within a month, verification effort spent on it evaporates with it. Prototypes need to be easy to throw away, not well tested.

Glue and configuration. Adapters that map one shape to another, dependency wiring, thin controllers that call one service and return its result. Mutants here are overwhelmingly either trivially killed or equivalent, and the triage cost dominates any insight.

Code dominated by I/O. When most of a function's runtime is a network call or database round trip, every mutant pays that cost again. Extract the pure decision logic and mutate that — the right move anyway — or leave the module out of the gate and say so.

Generated code. Client libraries generated from a schema, migration scaffolding, ORM boilerplate. Test the generator or the schema instead. Mutating generated output measures the generator's determinism, which you were not worried about.

Anywhere the equivalent-mutant tax exceeds the benefit. The honest general rule. If a module consistently produces mostly-equivalent survivors, you are paying an afternoon of triage for no new information. Drop it, record why in the exclusions file with a date, and revisit in six months.

What makes it worth the trouble everywhere else is that it is the only routinely available measurement answering the question agents have made urgent. Code volume has gone up, the fraction any human reads has gone down, and the tests guarding it are increasingly written by the same class of system that wrote the code — the AIDev study found agents already authoring 83–100% of test-adding commits in small projects, meaning those with 5–28 contributors. In that setting a coverage percentage is a comfort blanket, and 100% coverage next to a 4% mutation score is what a comfort blanket looks like when somebody finally measures it. The number you want has existed since the 1970s. It is just expensive, and now worth paying for.