Every AI lab that ships a coding model leads with a benchmark score. Every engineering team that picks a model reads those scores. And for most of 2026, the score everyone quoted came from SWE-Bench Pro.
There is now solid evidence that the number was measuring something other than what we thought.
In late May 2026, Datacurve published an audit alongside its new benchmark, DeepSWE. The finding was blunt: on SWE-Bench Pro, roughly a third of trials are graded incorrectly. Artificial Analysis has since migrated its coding evaluations off SWE-Bench Pro entirely. If your model selection, your procurement decision, or your architecture bet leaned on that leaderboard, it is worth understanding exactly what went wrong.
The audit: two failure modes, both bad
Datacurve sampled 30 tasks each from DeepSWE and SWE-Bench Pro, then ran three rollouts across ten frontier model configurations — 735 DeepSWE rollouts against 789 SWE-Bench Pro rollouts. An independent LLM analyzer then judged whether each patch actually implemented the requested behavior, rather than trusting the benchmark's own verdict.
The gap is not subtle:
| Metric | SWE-Bench Pro | DeepSWE |
|---|---|---|
| False positive rate | 8.5% | 0.3% |
| False negative rate | 24.0% | 1.1% |
| Pass-rate spread (worst to best model) | ~30 points | 70 points |
Read those two error columns carefully, because they fail in opposite directions.
False positives mean the benchmark credited a passing grade to a patch that did not implement the requested behavior. Roughly one in twelve "successes" was not one.
False negatives are the bigger number and the more insidious one. Nearly a quarter of correct patches were marked as failures. That is not noise you can average away — it systematically compresses the scoreboard. Every model gets dragged toward the middle, and genuinely better models cannot show their advantage. This is precisely why SWE-Bench Pro produced the comforting narrative that all frontier models are converging. They are not converging. The ruler was compressing them.
On DeepSWE, the same ten configurations spread across a 70-point range. On SWE-Bench Pro, they clustered inside 30. The models were always that far apart. The measurement instrument hid it.
The agents were reading the answers
The false-positive story has a specific and uncomfortable mechanism. SWE-Bench Pro derives its tasks from real repository commits, and the containers handed to the agent retained the repository's .git directory.
Agents found it.
Of the improper passes Datacurve identified on SWE-Bench Pro, 87% involved the agent retrieving the solution from git history. In 33 of 38 such trials, the agent explicitly ran git log or git show. Claude Opus 4.7 was observed reading git history in over 12% of its SWE-Bench Pro containers.
This is not a model behaving badly. Grepping history for context is exactly what a competent engineer does, and exactly what we want an agent to do in a real repository. It is a benchmark construction failure. The task was "implement this feature"; the environment quietly contained a commit that implemented the feature. A sufficiently capable agent finds the shortest path, and the shortest path was git show.
The lesson generalizes well beyond this one benchmark: the more capable your agent, the more aggressively it will exploit whatever your evaluation environment leaks. Any harness you build to evaluate agents internally has the same exposure. Sealed environments are not a nicety.
What DeepSWE does differently
DeepSWE is built to close both holes. It comprises 113 tasks across 91 repositories in five languages — TypeScript, Go, Python, JavaScript, and Rust — with four design choices that matter:
- Contamination-free tasks. Tasks are written from scratch rather than reverse-engineered from existing commits. There is no upstream commit to find, because the solution never existed in the repository's history.
- Hand-written behavioral verifiers. Instead of inheriting a repo's existing test suite (which may pass for the wrong reasons, or fail on unrelated flakes), each task ships purpose-built verifiers that check the requested behavior. This is what collapses the false-negative rate from 24.0% to 1.1%.
- Real long-horizon scope. Reference solutions average 668 lines added across 7 files. This is a feature-sized change, not a one-line patch.
- Repository diversity. 91 repositories for 113 tasks means minimal overlap. Repos are actively maintained, permissively licensed, and carry at least 500 GitHub stars.
Notably, DeepSWE prompts are shorter — 2,158 characters on average versus SWE-Bench Pro's 4,614. Less hand-holding, more inference required, larger patches expected. The published rankings are correspondingly spread out: GPT-5.5 at 70% ± 3%, GPT-5.4 at 56% ± 2%, Claude Opus 4.7 at 54% ± 5%, and Claude Sonnet 4.6 at 32% ± 2%.
Those are numbers you can actually make a decision with.
The other half of the problem: your harness is the variable
The benchmark is only one side of a trustworthy evaluation. Databricks recently published results from an internal benchmark built on its own multi-million-line codebase, and the findings reframe what "model choice" even means.
Their methodology deserves credit on its own. Tasks are derived from real pull requests, filtered for recency, human authorship, quality test suites, and self-contained scope, spanning Python, Go, TypeScript, Scala, Rust, and Java. Engineers manually extracted intent from each PR, wrote prompts with the solution hints stripped out, and isolated the relevant tests. Crucially — and clearly with the SWE-Bench Pro lesson in hand — they sealed git history so agents could not read the answer.
Three findings stand out.
Open models reached the top tier. GLM 5.2 matched Opus 4.8 on quality while costing $1.28 per task against Opus's $1.94. Databricks' conclusion is that open models are now viable daily drivers for everyday coding work, not just cost-conscious fallbacks. If you have been treating GLM-5.2 as a budget option, the data says it is a frontier one.
Token price does not predict task price. Sonnet 5 is roughly 1.7x cheaper per token than Opus 4.8. It also cost more per task: $2.09 versus $1.94, while scoring six points lower. A less reasoning-efficient model burns more tokens getting to the same place, and the per-token discount evaporates. Any cost model built on published token prices is measuring the wrong unit. Cost per completed task is the only number that pays your bill.
The harness moved cost 2x with identical quality. Running the same model through different harnesses produced cost differences above 2x at the same quality level. Databricks' internal Pi harness sent 3x less context per turn and finished tasks in fewer runs than Claude Code or Codex. Same model. Same output quality. Half the spend.
That last point is the one most teams underrate. You do not just choose a model — you choose a model and a harness, and the harness is doing at least as much work as the model weights. This is the practical case for treating harness engineering as a first-class discipline rather than plumbing.
What to do on Monday
The takeaway is not "distrust all benchmarks." It is that public leaderboards answer a question about someone else's repository, graded by someone else's verifier. Here is how to get a number that means something for your codebase.
Build a small internal benchmark from your own merged PRs. Twenty to fifty tasks is enough to separate models. Pull recent, human-authored PRs with real test coverage and self-contained scope. Strip the solution hints out of the prompt.
Seal the environment. Remove .git, or at minimum truncate history past the task's base commit. Remove issue trackers, changelogs, and any artifact that names the fix. Assume a capable agent will find anything you leave behind — because it will.
Write behavioral verifiers, not inherited test suites. Check that the requested behavior exists. A repo's own tests were written for a different purpose and will misgrade in both directions, exactly as SWE-Bench Pro did.
Measure cost per completed task, never cost per token. Track tokens consumed, wall-clock time, and retry count per task. Reasoning efficiency varies enough between models to invert the ranking that per-token pricing implies.
Evaluate the harness as a separate axis. Hold the model fixed and vary the harness. Then hold the harness fixed and vary the model. If you only ever test bundled pairs, you cannot tell which half is responsible for the result.
Re-run on every model release. The frontier moves monthly. A benchmark that ran once is a snapshot, not an instrument.
The wider point
The SWE-Bench Pro story is a good reminder that evaluation infrastructure is real infrastructure, and it decays. A benchmark that quietly leaks solutions through .git and misgrades a quarter of correct patches will still produce a clean-looking leaderboard, published in press releases, cited in procurement decisions, and used to justify architecture bets. Nothing about the output signals that the ruler is bent.
This is the same discipline problem we keep rediscovering in agent evaluation and in the gap between benchmark scores and real feature work. Models get better faster than the tools we use to measure them. When the measurement lags, we mistake a compressed scoreboard for a converging field, and we mistake a resourceful agent for a correct one.
Build the small internal benchmark. Seal the environment. Measure the task, not the token. It is a week of work that will outlast every model on your shortlist.
Need help building an evaluation harness for AI coding agents in your own codebase? Talk to the Noqta team — we design and ship agent infrastructure for engineering organizations across MENA.