What the leaderboard actually says — and what to do with it
Open any coding-model tracker this week and the top of the table looks decisive. Per third-party trackers as of 1 June 2026, the SWE-bench Verified leaderboard reads Claude Mythos Preview at 93.9%, Claude Opus 4.8 at 88.6%, and Claude Opus 4.7 (Adaptive) at 87.6%. OpenAI stopped self-reporting Verified in February 2026, so GPT-5.5 only appears on independent trackers, where it lands around 88.7%. Read at a glance, that is a near-perfect machine that fixes nine in ten real bugs.
Read properly, it is not. A benchmark percentage is the most compressed claim in machine learning: one number standing in for a model, a scaffold, a dataset, a grading harness, and a thousand decisions you cannot see from the chart. For a builder in Bengaluru choosing a default coding model, or a team in Manchester defending a vendor decision to a CFO, the skill that matters is not picking the highest row. It is knowing what the row leaves out. Here is how to read one without being misled.
| Model | SWE-bench Verified | Source / status |
|---|---|---|
| Claude Mythos Preview | 93.9% | Preview entry — not generally available |
| GPT-5.5 | ~88.7% | Third-party estimate (OpenAI no longer self-reports) |
| Claude Opus 4.8 | 88.6% | Generally available |
| Claude Opus 4.7 (Adaptive) | 87.6% | Generally available |
Figures per third-party trackers / the SWE-bench Verified leaderboard as of 1 June 2026. Not lab-official. SWE-bench Lite, a separate split, is led by Claude Opus 4.6 at roughly 62.7 — a reminder that the dataset split changes the ranking.
Never pick a coding model on the headline percentage alone. The gap between 88.6% and 88.7% is noise — well inside run-to-run variance and scaffold differences. Treating it as "GPT-5.5 beats Opus 4.8" is the single most common leaderboard-reading error, and it has signed off more bad vendor decisions than any other.
What SWE-bench Verified actually measures
SWE-bench Verified is a curated set of 500 real GitHub issues drawn from popular open-source Python projects. Each task hands the model a repository at a specific commit plus an issue description, and asks it to produce a patch. The patch is then run against the project's own hidden test suite. If the tests pass, the task counts as resolved. The headline number is pass@1: one attempt, graded automatically, no human in the loop and no partial credit.
Three things follow from that definition. First, it is real software, not a quiz — the model has to navigate an unfamiliar codebase, find the right files and write a patch that compiles and passes tests. That is genuinely hard and genuinely useful to measure. Second, "Verified" means OpenAI and the SWE-bench team hand-screened the tasks in 2024 to remove broken or under-specified ones, which is why it became the default coding benchmark. Third, and most important, it is heavily Python and heavily skewed towards a handful of large, well-tested repositories. Your TypeScript monorepo, your Go services, your legacy Java — none of that is in the set.
A leaderboard row is a model plus a scaffold
This is the point most readers miss. SWE-bench does not test a raw model. It tests a model wrapped in an agent harness — the scaffold — that decides how files are retrieved into context, how many turns the agent gets, whether it can run tests and read the output, how failures are retried, and what the system prompt says. Change the scaffold and the same model can move several points in either direction.
That is why open-source scaffolds now post numbers within touching distance of closed commercial agents on the same underlying model. It is also why two leaderboards can disagree about the same model by a wide margin: one is reporting the lab's best in-house harness, the other a community runner. When you read a row, read it as model + scaffold, never the model alone. If your production setup uses a different agent loop — and it will — the public number does not transfer cleanly to your stack. We unpacked this gap in detail in our piece on why agentic reasoning agents still score 38% on harder, scaffold-sensitive tests.
When comparing two models, hold the scaffold constant. Either compare rows that use the same harness, or run both candidates through your agent loop on the same tasks. A clean within-harness comparison is worth more than any headline figure, because it isolates the variable you actually control: the model.
Contamination — why OpenAI walked away
In February 2026 OpenAI stopped self-reporting SWE-bench Verified, and the reasons are a masterclass in how benchmarks decay. The 500 tasks come from public GitHub repositories. Any model trained on GitHub data after mid-2024 has very likely seen those exact issues — and their merged fix commits — during pre-training. The benchmark stopped measuring "can the model solve this bug" and started measuring "did the model memorise this bug's solution". OpenAI also found that a large share of the failing test cases were themselves flawed, so a model could be penalised for being right.
The tell is the gap between contamination-resistant and contamination-exposed benchmarks. Models scoring around 80% on Verified have dropped to far lower scores on fresher, held-out sets such as SWE-bench Pro. When a number falls off a cliff the moment the tasks are unseen, the original number was measuring exposure, not capability. This is why third-party verified runs matter: an independent party re-running the harness on the stated model, ideally with some held-out or refreshed tasks, is far more trustworthy than a lab's own marketing figure — and why a preview-only entry near the top of a table should be read with particular caution.
What the number does not tell you
Even a clean, uncontaminated, same-scaffold score is silent on most of what decides whether a model is right for production. Before you commit, price these in:
- Cost per task. Two models a fraction of a point apart on Verified can differ several-fold in token spend per resolved issue. Opus 4.8 at $5 per million input tokens is the cheapest model within 10% of the leader — that is a real, usable signal the percentage column hides.
- Latency. An agent that takes four minutes per fix is a different product from one that takes forty seconds, whether your developers sit in Pune or Bristol.
- Token burn and reliability. A model that solves a task in three turns half the time and twelve turns the other half has a variance problem that a single pass@1 number erases entirely.
- Generalisation to your repo. Verified is Python-heavy and built from a narrow set of large projects. Your stack's language, framework and house conventions are simply not represented.
- Run-to-run variance. Re-run the same model on the same tasks and the score wobbles. Treat sub-one-point differences between rows as a tie, not a ranking.
The billing point is no longer academic. With GitHub Copilot moving to AI-credits, usage-based billing on 1 June 2026, the cost-per-task column has become the column that lands on your invoice — we walk through the new pricing maths in our GitHub Copilot usage-based billing cost guide. A model two points lower on a leaderboard but half the token cost can be the correct production choice by a wide margin.
Every article here is written by a Verified Builder. Want your name on the next one?
AI Tech Connect lists AI engineers, founders and researchers across India and the UK — and the people hiring browse it to find them. Adding your profile is free.
Become a Verified Builder →The other five benchmarks, and when each one matters
SWE-bench Verified answers exactly one question: can the model fix a bug in a Python repository. If your product is an agent that does anything else, you need a wider panel. Six benchmarks cover most production agent decisions in 2026, and the trick is matching the benchmark to your workload rather than reading whichever one is trending.
- GAIA — general assistant tasks that need reasoning plus tool use. Read this if you are building a broad assistant rather than a narrow coder.
- SWE-bench Verified — real GitHub bug-fixes across 500 verified tasks. The right lens only if code-editing is your core loop.
- OSWorld — computer-use on a real desktop. The one to watch if your agent clicks, types and drives applications.
- Tau²-Bench — a tool-using agent talking to a simulated user while obeying a policy, across retail, airline and telecom domains. This is your benchmark if policy adherence and customer-facing conversation matter.
- WebArena — multi-step tasks inside a real browser. Relevant for any agent that navigates web apps to get work done.
- METR HCAST / Time Horizons — the longest task duration at which a model succeeds half the time. The closest thing to a measure of how much autonomy you can trust the agent with before a human steps in.
A team building a customer-support agent in Hyderabad should weight Tau²-Bench far above SWE-bench; a London fintech automating internal browser workflows should care about WebArena and OSWorld. Picking the model with the best coding score for a non-coding job is the same error as the headline-percentage trap, one level up.
A 2026 study showed that several major agent benchmarks — including Verified, WebArena and OSWorld — could be gamed to near-perfect scores without solving any task, through leaked answers, unsanitised evaluation code and prompt-injectable LLM judges. The lesson is not that benchmarks are worthless; it is that no single public score is load-bearing on its own.
How to actually evaluate for your own stack
The honest answer to "which coding model is best" is "best for what, measured how". Public leaderboards are a starting filter, not a decision. The decision comes from a small private evaluation set you build yourself, and it is less work than it sounds.
- Mine your own closed issues. Pull 20 to 30 GitHub issues from your repositories that are already resolved, where you know the correct fix and have tests that prove it. This is your private, contamination-free SWE-bench in miniature — and it is in your languages and frameworks.
- Run candidates through your real harness. Take two or three shortlisted models and run them through the exact agent loop you ship, not a benchmark scaffold. This holds the scaffold constant and measures the thing you control.
- Grade on your tests, then on cost. Score pass@1 on your suite, then record tokens, latency and turn count per task. Rank on resolved-per-rupee or resolved-per-pound, not on resolution alone.
- Re-run quarterly. Models, prices and your own codebase all move. A private eval you refresh every quarter is the only leaderboard that stays true for your team.
If you want a worked example of how the same model scores differently across splits and scaffolds, our breakdown of Claude Opus 4.8's dynamic workflows and parallel subagents shows how harness design alone changes real-world outcomes. The leaderboard is where you start the conversation. Your own closed issues are where you end it.