Does Your AGENTS.md Do Anything? 288 Runs Say Maybe Not

What the study found

• Three context strategies were compared. No context file at all; the repository's real AGENTS.md injected into the system prompt every turn; and a topic-organised wiki left in the workspace for the agent to retrieve on demand.
• Two frontier agents were tested. Claude Code and Codex, run against the same task set on the same repositories.
• The correctness difference was not measurable. The paper states that context strategy "does not measurably move correctness on either agent", with the effect bounded to no more than 10 to 15 percentage points by equivalence testing.
• Adding the real AGENTS.md converted no near-miss failure into a pass. On either agent. The tasks that failed, failed for reasons a context file did not address.
• One effect did reach significance. On Claude, the selective retrieval strategy reduced cache-creation tokens at p=0.012 — which the author attributes to injection mechanics rather than any gain in capability.

The paper is Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories, by Prakhar Khatri, posted to arXiv in July 2026. The design is the part worth paying attention to: 17 real tasks drawn from three genuine open-source repositories, each run three times, for 288 evaluated runs. That is a small study by machine-learning standards and a serious one by the standards of most claims made about agent tooling, which are usually anecdote plus a screenshot.

The result matters because the convention it interrogates has spread extremely fast. In under two years, a file at the root of the repository telling the agent about the project has gone from a Claude Code affordance to something close to an industry default, with AGENTS.md as the vendor-neutral name. Teams across Bengaluru, Hyderabad, London and Manchester now maintain these files, review them in pull requests, and argue about their contents. Very little of that practice has been tested.

The numbers, in full

Here is the whole result set. It is short, which is itself informative.

Read across the Claude rows and the context file looks marginally helpful: one extra pass out of 45, in both context conditions. Read across the Codex rows and it looks marginally harmful: one fewer pass with the file always on, three fewer with the wiki. Neither pattern survives contact with the sample size. A single task flipping in either direction moves these figures by more than the gap between the conditions, which is precisely why the author frames the finding as an absence of a measurable effect rather than as a direction.

The limitation that changes the reading

The most useful sentence in the paper is a limitation, not a result. The author notes that with 15 to 17 tasks and three repeats, the minimum detectable effect is well above 30 percentage points — meaning an effect of around 10 points would be entirely invisible to this design.

That reframes everything. A 10-point improvement in agent pass rate would be an enormous practical win. If your agent completes just over half its tasks unaided and a well-written context file lifted that to just under two-thirds, no engineering manager in Pune or Bristol would call that marginal. This study could not have seen such an effect. What it can tell you, with reasonable confidence, is that context files are not delivering the very large gains that the enthusiasm around them sometimes implies.

This is the difference between "we found no effect" and "there is no effect", and it is the distinction that separates careful reading of a paper from a viral summary of one. The same discipline applies to any benchmark claim you encounter; we set out the general method in our guide to benchmarking coding agents on your own repository, which remains the only evaluation that answers the question you actually care about.

Why the agents failed

The paper's substantive claim is about the shape of the failures rather than their count. Agents failed, the author argues, because of implementation skill gaps rather than missing repository knowledge. The supporting evidence is that no near-miss failure was converted into a pass by adding the real AGENTS.md, on either agent.

That fits a pattern many teams will recognise from their own logs. When an agent botches a change, the transcript rarely shows it confused about where the module lives or which test runner the project uses. It shows the agent producing code that is plausible, syntactically fine, and wrong about the logic — an off-by-one in a boundary condition, a misunderstanding of what a function is supposed to return in the failure case, a change that satisfies the immediate test and breaks an invariant three files away. A document explaining the project's directory structure does not help with any of that.

There is a supporting signal here too: task difficulty correlated across the two agents at a Spearman rho of 0.75. The tasks that were hard for Claude Code were largely the same tasks that were hard for Codex. If difficulty were mostly a function of how well each agent had been briefed on the codebase, you would expect more divergence, since the two systems handle context quite differently.

What was actually in the files

The three repositories were pdm, a Python package manager; firebase-admin-python, a cloud SDK; and opshin, a domain-specific compiler. Their real AGENTS.md files ran to 477, 1,236 and 248 words respectively.

Those lengths are worth sitting with, because they are short. A 248-word file is a page of notes. A 1,236-word file is still comfortably under the length that many teams' context files have drifted to after a year of additions. If your repository's file is now four thousand words of architecture description, this study did not test anything resembling it — and the general principle that longer context dilutes attention gives you reason to suspect that scaling up would not have helped.

The selective condition carries an explicit confound that the author flags. For pdm and firebase-admin-python, the wiki corpus was 10 and 18 times larger than the corresponding AGENTS.md. So the comparison between "always on" and "selective" is not a clean test of delivery mechanism; it also varies how much material existed. That matters most for the Codex selective row, the worst-performing cell in the table, which is exactly where you would want the cleanest design.

What a builder should do on Monday

The practical guidance is narrower than either the enthusiasts or the sceptics would like.

Keep the file, but change what you expect from it

A context file is documentation with a specific reader. It is a good place for the things that are true, stable and awkward to infer: the build and test commands, the commit message convention, which directories are generated and must not be edited by hand, the fact that the project uses one particular async library and not the obvious alternative. None of that is what this study measured, because none of it shows up as a pass or fail on a benchmark task. It shows up as the agent not wasting a turn asking, and not opening a pull request in the wrong format.

What you should stop expecting is that a better-written file will lift your agent's success rate on hard work. The evidence for that proposition is currently thin, and the mechanism the paper proposes — that failures come from implementation ability, not briefing — is a plausible explanation for why.

Watch the length, and prefer retrieval for bulk

The single significant efficiency result points at a real operational consideration: a large block injected into the system prompt on every turn has a cost, and moving that material into files the agent can fetch when relevant changes the cache profile. If your context file has grown past a page or two, splitting the reference material into retrievable documents and keeping only the critical rules in the always-injected file is a reasonable move on cost grounds, even though the study gives you no reason to expect a correctness gain from it. Teams juggling several repositories will find the same tension covered in our note on multi-repository context strategy.

Put your effort where the failures are

If the paper's central claim holds — that agents fail on implementation rather than orientation — then the leverage sits elsewhere: in tighter task decomposition, in tests that catch the specific class of error your agent makes, and in review discipline for agent-written changes. That last one is doing more work than most teams admit, and the techniques for checking whether your tests would actually catch an agent's mistakes are covered in our guide to mutation testing for agent-written code.

For teams in India and the UK running lean, this is a mildly encouraging finding. The thing you were told to invest in — careful, extensive, continuously-maintained context documentation — appears to be less load-bearing than advertised. That is time you can redirect towards evaluation harnesses and test coverage, both of which have a clearer relationship to whether the work an agent produces is correct.

Where this leaves the convention

One underpowered study on three Python repositories does not settle a question. What it does is establish a baseline of evidence where previously there was almost none, and it sets a standard that the next study will have to beat: a controlled ablation, multiple agents, real repositories, repeated runs, and an honest statement of what the design could not detect.

The most likely resolution is that context files matter, but conditionally and modestly — more for unusual conventions than for common ones, more for repositories where the obvious inference is wrong, and more for the parts of the interaction that benchmarks do not score. That is a less satisfying conclusion than either "AGENTS.md is essential" or "AGENTS.md is theatre", and it is the one the current evidence supports. Our existing walkthrough of steering coding agents with AGENTS.md and CLAUDE.md still holds on the mechanics; what has changed is how much you should expect from them. More research coverage sits in our AI research section.