Debug With a Coding Agent: Reproduce, Bisect, Fix, Prove

What you need to know

• Generating a fix and establishing a cause are different skills, and agents have one of them. A plausible patch at the point of failure is what a language model is good at. Locating where the behaviour changed is not.
• Three documented agent failure modes are debugging failures. Ineffective backtracking, poor resource awareness and instruction drift, all named in the 2026 case-study paper on open-ended AI research (arXiv 2607.27191).
• Every stage ends with a gate, not a judgement. A deterministic failing test. A named commit or version boundary. A stated causal chain. A revert check that goes red.
• Bisection is the highest-value agent task in the loop. Mechanical, verifiable, and structurally immune to a favoured hypothesis: the next commit to test is chosen by the last result, not by argument.
• One green run is weak evidence, and nobody can tell you the base rate. Agent performance varies substantially across runs (arXiv 2608.13417), and no public dataset measures agent debugging success at all.

Why agents generate fixes and fail to find causes

Watch an unstructured debugging session and the shape is always the same. You paste an exception. The agent reads the trace, finds the line that threw and proposes a change there: a guard clause, a coerced type, a retry. The error stops appearing. The agent tells you, with total composure, that the issue is resolved.

Sometimes it is. Often a symptom has simply been suppressed at the place the program noticed a problem, which is rarely the place it acquired one. A None arriving from a misparsed configuration file four modules upstream can be null-checked into silence at the point of use, and that null check will look like perfectly correct code. The bug is now quieter and harder to find.

This is not a shortcoming you can prompt away; it is downstream of how these systems behave over long tasks. The 2026 study asking whether AI agents can conduct open-ended research (arXiv 2607.27191) gave frontier agents six days and thousands of dollars of compute per case study. The engineering all got done without human help; what the agents could not do was make substantial progress on the research questions. The paper names five recurring failure modes — poor judgement about the bar for publishable research, uncreative responses to shortcomings in research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift — and the last three describe a bad debugging session with uncomfortable precision.

Ineffective backtracking is the agent that keeps refining its caching theory rather than abandoning it, because abandoning it means discarding an hour of reasoning it can still see. Poor resource awareness is the session spending forty minutes instrumenting a subsystem a two-minute bisection would have exonerated. Instruction drift is the agent asked why a settlement total is short that is now refactoring a currency helper it disliked on the way past.

A companion evaluation published on 13 August 2026 (arXiv 2608.13417) adds that current agents operate more like engineering optimisers than autonomous researchers, and that performance varies substantially across runs. An optimiser is exactly what you want driving a bisection and exactly what you do not want deciding when an investigation is finished. So the loop below does not try to make the agent reason better; it removes the decisions agents make badly and replaces each with a checkable artefact.

Stage	Entry condition	Agent may do	Exit gate
1. Reproduce	A report, a trace or an observed misbehaviour.	Read code and logs, write a failing test, pin the environment. No code edits.	A test failing on the current commit, for one reason, every run.
2. Bisect	The stage-1 test, runnable from a clean checkout.	Write the predicate, drive git bisect run, sweep versions, flags, inputs. No fixes.	A named commit, input, version or flag boundary. Not a hypothesis.
3. Fix	A named boundary plus a one-sentence causal chain.	Edit production code — the smallest change addressing the named cause.	A diff whose every line traces to the causal chain.
4. Prove	A candidate fix and the original reproduction.	Run the suite, run the revert check, write the regression test.	Green with the fix, red without it, no other test moved.

The gates are the whole design. Each is binary, checkable by someone who was not in the session, and impossible to satisfy with a persuasive paragraph. A drifting agent cannot drift past a gate, only fail to reach one.

Stage one: reproduce, and edit nothing

No work begins until there is a deterministic failing test. This is the stage teams skip, and skipping it poisons everything downstream: you cannot bisect without a predicate, and you cannot prove a fix without something that was red before it.

The rule that makes the stage work is a prohibition: the agent may not propose or apply a fix during stage one. State it explicitly, and again when the agent offers one anyway, because it will, and it will be certain. Note the observation as a stage-three candidate and carry on. A short debugging-protocol section in your repository instructions file makes this stick across sessions; see the guide to steering agents with AGENTS.md.

Determinism means removing everything the code can read that is not its input. Seed every random source. Freeze the clock rather than mocking one call site, because month-end and daylight-saving transitions are where date bugs live and a partially frozen clock reproduces them only sometimes. Pin timezone and locale explicitly: a suite that passes in Asia/Kolkata and fails in Europe/London has taught you something already. Commit the fixture and never let CI regenerate it.

# tests/test_repro_settlement_rounding.py
# The single failing test the whole session hangs on. It must fail on the current
# commit, for one reason, on every run — no network, no wall clock, no ambient state.
import random

import pytest
import time_machine                       # freezes the clock, including C-level calls

from ledger import settle_batch
from tests.helpers import load_fixture

SEED = 20260825
FIXTURE = "tests/fixtures/batch_4417.json"    # committed; never regenerated by CI
MONTH_END = "2026-03-31T23:59:58+00:00"       # the boundary the report pointed at


@pytest.fixture(autouse=True)
def deterministic_environment(monkeypatch):
    """Pin everything the code under test could read that is not its input."""
    random.seed(SEED)
    monkeypatch.setenv("TZ", "UTC")
    monkeypatch.setenv("LC_ALL", "C")
    # Feature flags are ambient state. Absent by default; set explicitly per test.
    monkeypatch.delenv("LEDGER_FEATURE_FLAGS", raising=False)
    # PYTHONHASHSEED must be set before the interpreter starts, so export it in
    # the runner -- PYTHONHASHSEED=20260825 pytest ... -- not here.


@time_machine.travel(MONTH_END, tick=False)
def test_settlement_credits_gross_minus_fees():
    batch = load_fixture(FIXTURE)
    result = settle_batch(batch, currency="INR")
    # One assertion, on the invariant that is actually broken.
    assert result.credited_total == batch.gross_total - result.fees_total


@pytest.mark.parametrize("attempt", range(20))
def test_reproduction_is_not_flaky(attempt):
    """Twenty identical runs, expected red twenty times. If this ever comes back
    mixed, stop: you are debugging non-determinism, not the bug you came for.
    Delete this test once the reproduction has proved stable."""
    batch = load_fixture(FIXTURE)
    with time_machine.travel(MONTH_END, tick=False):
        result = settle_batch(batch, currency="INR")
    assert result.credited_total == batch.gross_total - result.fees_total

When a bug will not reproduce locally, the gap between environments is the task — and it is good agent work, because it is a list rather than an insight. Walk the differences one at a time: operating system and architecture, dependency versions resolved from a lock file rather than a range, timezone and locale, feature flag state, database contents and migration level, concurrency, memory limits. A team in Bengaluru chasing a failure that appears only on a European deployment usually finds it in the third or fourth item.

Stage two: bisect to a boundary, do not guess at a cause

Stage two answers one question with a fact: where did the behaviour change? Not why. Where. A named commit, a dependency version, a specific input, a flag flip. Hypotheses are not admissible at this gate.

git bisect run is the central technique and the reason this stage suits an agent so well. Give it a known-bad revision, a known-good revision and a predicate that exits 0 for good and 1 for bad, and it binary-searches the history. The agent's job is the predicate. It cannot skip to the commit it suspects, weight evidence towards its existing theory, or end the search early: the procedure decides the next step, and every step is independently re-runnable. Against ineffective backtracking, a search that cannot be argued with is a strong structural defence.

#!/usr/bin/env bash
# /var/tmp/repro/predicate.sh -- the entire hypothesis, expressed as an exit code.
# Keep this file and the reproduction OUTSIDE the repository: bisect checks out
# each revision, which would delete or revert anything living inside the tree.
#
# git bisect run contract:
#   0        -> good  (bug absent at this revision)
#   1..124   -> bad   (bug present)
#   125      -> skip  (this revision cannot be judged)
#   >127     -> abort the whole run
set -u

REPRO="/var/tmp/repro/test_repro_settlement_rounding.py"

# Bring dependencies to the state THIS revision declares. If that fails, the
# revision is unjudgeable -- skip it rather than blaming it for the bug.
if ! uv sync --frozen --quiet; then
  exit 125
fi

PYTHONHASHSEED=20260825 uv run pytest "$REPRO" -x -q --no-header
status=$?

case "$status" in
  0) exit 0 ;;    # tests passed  -> good
  1) exit 1 ;;    # tests failed  -> bad
  *) exit 125 ;;  # collection error, import error, missing module -> unjudgeable
esac

Driving it is four lines, and the fourth is the one people forget:

git bisect start
git bisect bad  HEAD                      # the bug is here
git bisect good v4.2.0                    # a release you are CERTAIN predates it
git bisect run /var/tmp/repro/predicate.sh

# When it stops, record the evidence before you touch anything:
git bisect log > /var/tmp/repro/bisect-4417.log
git show --stat "$(git rev-parse refs/bisect/bad)"
git bisect reset

Three details separate a bisection that produces a fact from one that produces a plausible-looking hash. The 125 exit code is not optional: revisions where dependencies will not install must be skipped rather than judged, or the search converges on infrastructure churn. The known-good revision must be verified rather than assumed, because a bisection anchored to a bad "good" is guaranteed to mislead. And the predicate must exit 1, never 125, for the ordinary failure case; one that skips every bad revision terminates cleanly and tells you nothing.

Plenty of bugs are not in your commit history at all, and the same discipline applies to every other dimension. Sweep dependency versions by pinning one library at a time and re-running the predicate. Bisect the input: halve the failing payload, keep the failing half, repeat, until you hold the minimal record that triggers it. Bisect feature flags one at a time from the known-bad configuration, and configuration by eliminating differences against a working environment.

The gate is the one agents most want to negotiate, so restate it: stage two ends with a boundary, not a theory. "The behaviour changes at a3f19c2" passes; "it is probably the connection pool change" does not.

Stage three: the smallest change that addresses the named cause

Only now may production code be edited, and the first thing stage three produces is not a diff. It is one sentence: commit a3f19c2 changed X, which means Y, which produces the observed Z. The stage-two boundary at one end, the stage-one symptom at the other, stated before the agent touches anything.

It is a cheap and remarkably effective filter. An agent that has located a cause writes that sentence immediately. An agent that has pattern-matched a plausible repair produces something with a gap in the middle — a chain starting at the symptom, gesturing at a mechanism, never mentioning the bisected commit. When the sentence will not connect both ends, return to stage two.

Then hold the diff to the chain. Every changed line should trace back to the stated mechanism; anything else is separate work, belonging to its own branch. This is where instruction drift becomes an unreviewable pull request: the fix is four lines and the diff is four hundred, because the session wandered through a tidy-up on the way. Nobody can see a subtle four-line fix inside four hundred lines of reformatting, so the drift does not merely add noise, it defeats the review.

Fixing the symptom is not always wrong, but it must be a decision rather than an accident. Where the cause sits in a third-party library, a documented containment at the boundary is legitimate engineering — provided the commit message names the cause it contains. A guard clause carrying a3f19c2 and the upstream issue is a different artefact from an unexplained one.

Stage four: prove it, and the revert check is the gate

Four things must be true before a debugging session is finished. The reproduction passes with the fix applied. It fails again with the fix removed. No other test changed state. And the fix ships with a regression test that would have caught this bug when it was introduced.

The second is the gate, and the one nearly everybody skips. A test that goes green after a change proves only that the change and the green result co-occur — a warmed cache, a regenerated fixture or a dependency resolving differently produces the same result. Removing the fix and watching the reproduction go red is the cheap experiment that closes the loop.

#!/usr/bin/env bash
# /var/tmp/repro/prove.sh -- four checks, one exit code. Run before opening the PR.
set -euo pipefail

REPRO="/var/tmp/repro/test_repro_settlement_rounding.py"
FIX="${1:-HEAD}"                      # the commit containing the fix
export PYTHONHASHSEED=20260825

echo "== 1/4 reproduction passes WITH the fix =="
uv run pytest "$REPRO" -x -q

echo "== 2/4 full suite: nothing else moved =="
uv run pytest -q

echo "== 3/4 revert check: reproduction FAILS without the fix =="
WORKTREE="$(mktemp -d)/without-fix"
git worktree add --quiet --detach "$WORKTREE" "${FIX}^"
trap 'git worktree remove --force "$WORKTREE" >/dev/null 2>&1 || true' EXIT
( cd "$WORKTREE" && uv sync --frozen --quiet )    # must succeed; set -e aborts if not

set +e
( cd "$WORKTREE" && uv run pytest "$REPRO" -x -q )
without_fix=$?
set -e

if [ "$without_fix" -eq 0 ]; then
  echo "FAILED: the reproduction passes without your fix."
  echo "        Your change is not what makes this test green. Return to stage 2."
  exit 1
fi
echo "   ok -- red without the fix, green with it"

echo "== 4/4 the fix commit carries a test =="
git diff --name-only "${FIX}^" "$FIX" | grep -qE '(^|/)tests?/' || {
  echo "FAILED: the fix commit touches no test file."
  exit 1
}

echo "all four checks passed"

The regression test is not the reproduction. The reproduction was built to fail on one commit with one fixture; the regression test runs for the next five years, phrased as the invariant rather than the incident. "Settlement credits gross minus fees, at any instant, in any timezone" survives a refactor. "The March 2026 batch totals 4,417" does not — and the invariant version is also better raw material if you ever build a private benchmark harness from your own repository history.

Then tie the stage back to variance. Agent performance varies substantially across runs, so a single green result from any stochastic process is weak evidence. For a deterministic reproduction, one green plus one red revert check is genuinely strong: both directions have been demonstrated. Introduce anything stochastic — a model in the loop, concurrency, a timing-dependent path — and repeat until a mixed result would have appeared. The AI code review and CI quality gates guide covers where to draw the blocking line; mutation testing for agent-written code answers whether your regression test asserts anything at all.

Context hygiene during a long debug session

A debugging session accumulates something a code-writing session does not: a growing pile of disproved theories. By the fifth exchange the context holds three abandoned hypotheses, the reasoning that supported them, two red herrings from unrelated log lines and a file the agent read for reasons nobody remembers. All of it is still visible to the model, and none of it is marked false.

That is poor resource awareness and ineffective backtracking in practical form: an agent whose context holds an elaborate case for a caching problem keeps finding the caching problem, because the strongest signal available says caching. Every turn spent near a dead theory makes leaving it less likely.

The countermeasure is unglamorous: reset the session and re-seed with only the established facts — the things that passed a gate. The reproduction command and its output, the bisected commit and its diff, the causal sentence if you have one, the eliminated dimensions. Not the narrative, not the theories, not the transcript. A good re-seed fits in a short paragraph; if yours does not, you are carrying reasoning rather than facts.

Do it at the seams: after stage one, so bisection starts from a clean predicate; after stage two, so the fix is written against a named commit; and whenever the agent restates a theory you already disproved. The mechanics of what survives a reset are covered in the context engineering and compaction guide; the debugging-specific rule is that gate artefacts survive and reasoning does not.

Instruction drift responds to the same treatment. The stage-one prohibition decays with distance, and restating it at each reset costs one line. Keep a scratch file outside the model's context holding the gates and their status, and re-seed from that rather than from your memory of the conversation. One London fintech team keeps it as a five-line markdown block in the incident channel: the facts live where the session cannot quietly rewrite them.

Common pitfalls

Five failures account for most wasted debugging sessions, and all five are procedural.

Letting the agent write the failing test and the fix in the same turn. The two artefacts are then correlated by construction, and all you have learned is that the agent can build a matched pair — not that the test captures the reported bug. Split the turns, and inspect the failing test before any fix exists.

Accepting "I have fixed it" without a revert check. Confidence in the report is uncorrelated with correctness, and everything that makes a green result appear by accident produces exactly the same confident report.

Bisecting with a flaky predicate. Prove the predicate is stable on the known-bad revision before starting, and remember git bisect skip exists for revisions that genuinely cannot be judged.

Letting the scope widen mid-session. "While I was in there" reads as diligence and behaves as instruction drift: the diff grows, the review degrades, and if the fix turns out to be wrong you can no longer revert it cleanly, because three unrelated improvements are wrapped around it.

Losing the reproduction when the branch moves. The failing test lives in the working tree, the branch gets rebased or the bisection checks out something else, and the one artefact the session depends on is gone. Keep the reproduction and the predicate outside the repository.

When not to reach for an agent

Two categories of bug are worse with an agent, and recognising them early saves more time than any technique in this article.

The first is the bug whose cause is in your own head rather than in the code. The system behaves exactly as written; your model of what it should do is wrong. A rate limiter that looks broken because you misread the window semantics, a query returning "wrong" rows because you misunderstood the join, a queue delivering "duplicates" that were always at-least-once by design. An agent handed a report of misbehaviour will find a way to change the behaviour — and now the code no longer matches the specification either. The signal is a report of the form "this is wrong" with no invariant attached. Read the specification first.

The second is the bug where reproduction costs more than the fix. A one-character typo in a log message needs neither a deterministic test nor a bisection. This loop is priced for bugs where being wrong is expensive: data corruption, money, authentication, anything intermittent, anything a customer noticed. Applied to a typo it is ceremony, and ceremony applied indiscriminately gets abandoned wholesale.

Where this leaves you

Nothing in this loop is new. Deterministic reproduction, binary search over history, minimal fixes, regression tests — ordinary engineering practice, predating every model you have used. What has changed is who needs the structure. A human debugger who guesses gets bored, notices the circles and backs out. An agent that guesses produces fluent, confident output at the same rate whether it found the cause or invented one, and will not back out on its own.

As of August 2026, the vendor-reported figures on the Qwen3.8-27B model card — Terminal Bench 2.1 at 73.0, SWE-bench Pro at 61.7 — make the modest point that even a strong current model fails a meaningful share of realistic software-engineering tasks. Nobody publishes an equivalent number for debugging, because no public dataset measures it. That absence is the argument for the gates: when the base rate is unknowable, build a process whose output you can check without it.

Start with the revert check. It is one script, it takes a minute, and it will change what you believe about how many of your recent fixes actually fixed anything.