astradevlabsastradevlabs
← All posts
AI News4 min

The SWE-Bench Pro Scoreboard Is a Mess: 6 Coding Models Ranked (and Why the Numbers Do Not Agree)

AI News

One benchmark, one name, and at least four different "top" scores depending on who ran the eval. SWE-Bench Pro has become the coding number every model launch quotes, and in the last two weeks it has quietly turned into a mess.

Here is the setup. SWE-Bench Pro hands a model a real GitHub issue plus the repo as it existed when the issue was filed, and asks for a patch. Success is binary: the patch has to make the failing tests pass without breaking anything else. The full set is 1,865 tasks across 41 professional repositories, and the 731-task public subset is drawn from strong copyleft (GPL) projects on purpose, so the code is a legal headache to train on and the benchmark resists contamination.

That design is good. The problem is that "80% on SWE-Bench Pro" and "59% on SWE-Bench Pro" are both being reported this month for the same era of models, because vendors run different subsets with different scaffolding. Here are six models everyone is arguing about, ranked by their loudest reported number, with the asterisk each one deserves.

1. Claude Mythos 5 / Fable 5 — 80.3%

Anthropic launched Fable 5 (and the restricted Mythos 5 tier) on June 9 at $10/$50 per million tokens, headlining 80.3% on SWE-Bench Pro versus GPT-5.5's 58.6%. It is the number the whole leaderboard is now measured against.

The asterisk: that 80.3% was produced with Anthropic's own scaffolding, not a neutral harness, and independent evaluators have openly contested it. Treat it as a vendor ceiling, not a settled result.

2. Sakana Fugu Ultra — 73.7%

Fugu Ultra sits second on the vendor-aggregate view at 73.7%. Impressive on paper, and worth watching, but it lands in the same bucket as Fable 5: a strong self-reported figure that has not been reproduced on a standardized public run.

3. Claude Opus 4.8 — 69.2%

Opus 4.8 is the pragmatic answer to "what actually leads right now." On the llm-stats vendor aggregate it is the top active model at 69.2%, and unlike the two above it, it has been the stable reference point through the June launch firehose.

When Fable 5 briefly went offline under an export-control order, Opus 4.8 was the model most teams fell back to. "Top active" is often the number that matters more than "top."

4. Claude Sonnet 5 — 63.2%

Shipped June 30 as the most agentic Sonnet yet, at 63.2% on SWE-Bench Pro. The interesting part is the price-to-score ratio: on knowledge work it edges ahead of Opus 4.8, and for coding-agent loops where you burn a lot of output tokens, a mid-60s score at Sonnet pricing beats a low-80s score you can't afford to run all day.

5. GLM-5.2 — 62.1%

The open-weight standout. Zhipu AI's GLM-5.2 is a 744B-parameter mixture-of-experts model with a 1M-token context, released June 16 under an MIT license with no regional limits. It scores 62.1 on SWE-Bench Pro, ahead of GPT-5.5's 58.6.

If you need to self-host or you care about license terms more than the last few points of accuracy, this is the one to try first. It is the only model on this list you can run without asking anyone's permission.

6. GPT-5.5 — 58.6% (and the standardized picture)

GPT-5.5's 58.6% is the number everyone else benchmarks against, and it is close to the only figure produced on a genuinely neutral harness: on Scale's standardized public set, the June 28 leader was GPT-5.4 (xHigh) at 59.1%.

Sit with that for a second. On the contamination-resistant public set that nobody's marketing team controls, the frontier is hovering around 59% — not 80%.

Why the numbers don't line up

Three things are being quietly swapped between reports:

  1. Different subsets. The full 1,865-task set, the 731-task public set, and various vendor slices are all called "SWE-Bench Pro." They are not the same test.
  2. Scaffolding. A model wrapped in a vendor's best agent harness, with retries and custom tools, scores far higher than the same model on a plain neutral runner.
  3. Attempt budget. pass@1 and best-of-k are both reported as "the score." More attempts, higher number.

If a launch post quotes a SWE-Bench Pro figure without naming the subset, the harness, and the attempt budget, it is a marketing number, not a measurement.

What to actually do with these numbers

  • Compare models only within a single harness — Scale's public leaderboard is the cleanest cross-vendor read right now.
  • Weight the standardized ~59% frontier over the ~80% headlines when you plan capacity and cost.
  • For your own stack, replay 15–20 of your real merged PRs as issues and score them yourself. Your repo is the only leaderboard that pays your bills.

References