DeepSWE vs SWE-Bench Pro: What a 24% False Negative Estimate Means for Coding Benchmarks

Two days ago, Datacurve published DeepSWE, a new benchmark for coding agents. Most of the attention went to the leaderboard, where GPT-5.5 leads by a wide margin.

The more interesting part sits lower down the post. The authors went back to SWE-Bench Pro, a benchmark many of us use when comparing models, and tried to measure how often its verifiers give the wrong verdict.

Their estimate: on about a third of sampled trials, the verifier disagreed with an independent review of the patch.

That number needs caveats, and I will get to them. The question behind it applies to anyone choosing a model or testing an agent: when your test says pass, how do you know?

What is DeepSWE?

DeepSWE is a long-horizon software engineering benchmark of 113 tasks across 91 active open-source repositories, covering TypeScript, Go, Python, JavaScript and Rust.

Datacurve says the tasks are written from scratch, rather than adapted from existing commits, and are never merged upstream. Each one has a prompt, an executable verifier and a reference solution. The verifiers extend each repository's test infrastructure and assert through public APIs and observable behaviour, so that any valid implementation can pass. They run three times during authoring, and flaky ones get revised.

The task sizes differ a lot from SWE-Bench Pro:

Source: Datacurve, May 26, 2026. All figures are the authors' own.

A shorter prompt and a much larger solution means the agent has to work out more for itself. That is closer to how senior engineers receive work.

The verifier audit

The authors argue that SWE-Bench Pro's verifiers are inherited from merged pull request test suites, and so they may misgrade solutions that differ from the original patch.

To test this, they drew 30 random tasks from each benchmark and ran 3 rollouts per task across frontier agent configurations. The post says 10 configurations in one place and 9 in another. An LLM judge then read each trajectory with the task, reference solution and verifier output, and gave its own verdict on whether the patch implemented the requested behaviour.

Datacurve notes that this puts disagreement on SWE-Bench Pro at about 32% of trials, which tells us the rates are shares of all reviewed trials.

That allows a small piece of arithmetic of my own. If those are shares of all trials, then the benchmark's pass rate on this sample sits about 15.5 points below what the judge would have given (24.0 minus 8.5). False negatives outnumber false positives by nearly three to one, so the errors do not cancel out. If you read a SWE-Bench Pro score of 50 as "half of these patches work", this audit suggests that reading is too low, and that some of the patches behind it are wrong as well.

Why I would treat these numbers with care

Three caveats matter.

The judge is a model. The audit's ground truth is an LLM judge, which the authors acknowledge can be wrong. These are estimates of disagreement with a judge. They are not measured truth.

The authors are auditing a rival. Datacurve built the benchmark that wins this comparison, and its own tasks were designed with this exact failure in mind. I have no reason to doubt the method, but I would like to see someone without a benchmark to promote repeat it.

The sample is modest. Thirty tasks from each benchmark is a reasonable start. It is not a full census.

The direction of the finding is plausible regardless. If a test suite came from one specific patch, I would expect it to reject some correct patches that made different internal choices. Anyone who has reviewed a change that broke tests while doing the right thing has seen this first-hand.

What the DeepSWE leaderboard shows

Source: Datacurve, leaderboard snapshot of May 26, 2026.

All models ran on mini-swe-agent, a model-agnostic harness with one bash tool and a shared prompt. The authors chose it to isolate model capability, and report that in a pilot of 10 SWE-Bench Pro tasks it matched or beat the native harnesses for each model tested. Ten tasks is a small pilot.

This has a direct consequence for buyers. The score describes a model in one harness, and your team will probably use it through a different one. Claude Code, Codex CLI and Cursor all add their own context management and tool behaviour, so the same model can perform differently in each.

The authors also report that pass rates for the compared models span 70 points on DeepSWE, against 30 on SWE-Bench Pro. The table above is a selection, and its own range is 65 points. A wider spread makes differences easier to see, though it does not prove the scale is measuring the right thing. It may only mean the tasks are harder.

As for a 70% score looking like a ceiling, I would not get attached. I look forward to rereading this table in six months.

What this means for your own evaluation suite

Few of us run public benchmarks. Many of us run private test suites, and those have the same weakness in a smaller form.

When I built the evaluation harness for Career Rafeeq, an AI SaaS product, I used 24 hand-curated real job descriptions with assertion-based ground truth and six kinds of assertion, with several runs per case aggregated statistically. A prompt edit or model swap has to pass before merge. The assertions deserve more thought than the runner: they have to test what the user needs without demanding one particular wording.

The audit gives a useful way to think about that trade-off. An assertion that is too narrow produces false negatives: a better output fails, and you reject a good change. An assertion that is too loose produces false positives: a wrong output passes, and it reaches users. Most suites drift towards the first problem, because people copy the output they saw when the test was written.

Some steps I would take:

• Assert behaviour, not text. Check facts, structure and outcomes through the interface a user would see.
• Audit your own verifier. Take 30 failed cases and 30 passed cases, and have a person review them. This could take an afternoon, and you get your own false positive and false negative estimates.
• Run cases more than once. Single-run results on a stochastic system mix signal with noise.
• Record the harness, model, settings and date with every score. A number without those four fields can't be compared to anything.
• Keep an LLM judge advisory. The DeepSWE audit shows a judge is useful for finding disagreements. A person or deterministic check should settle them.

Should you trust DeepSWE?

I would add it to the list of benchmarks I check, and I would not replace the rest with it.

The design decisions are good ones: behavioural verifiers, implementation-agnostic checks, longer tasks and a stated method. The verifier audit is the most useful thing a benchmark publisher has released in a while, even with its conflict of interest.

But the honest position is unchanged. A public leaderboard tells you which models deserve a trial. Your own repository, harness and acceptance criteria tell you which one to use.

Sources

• Huang, Lee, Tng and Ge, Datacurve, DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks, May 26, 2026

Written by Eslam Hamed, a Technical Lead and AI Solutions Architect working on model evaluation, agentic AI, and cloud-based software systems.