SWE-Bench Pro
Shares rank 3 of 4 coding benchmarks by recommenders
Recommended for testing AI coding models on longer tasks in real codebases, and seen as less contaminated than older benchmarks. Critics say many tasks are underdetermined or mismarked, so scores may be unreliable.
Recommended for testing AI coding models on longer tasks in real codebases, and seen as less contaminated than older benchmarks. Critics say many tasks are underdetermined or mismarked, so scores may be unreliable.
11 people gave an opinion
- recommend it
- 5
- criticize it
- 6
- approval share
- 45%
Recommended in 4 of 13 months
- November 2025: not recommended
- December 2025: not recommended
- January 2026: not recommended
- February 2026: not recommended
- March 2026: recommended
- April 2026: recommended
- May 2026: recommended
- June 2026: not recommended
- July 2026: recommended
- August 2026: not recommended
- September 2026: not recommended
- October 2026: not recommended
- November 2026: not recommended (month in progress)
Why it doesn’t carry the badge
- 5 people recommend itNeeds 10 or more, not met.
- 0.8 recommenders for every criticNeeds 2 to 1 or better, not met.
- Recommended in 4 of 13 monthsNeeds 4 or more, met.
The same three rules apply to every product, and the badge can’t be bought. How the badge works
What people say about it
The reasons people give, with the number of comments that give each one. Open a reason to read what they said.
Why people recommend it
Each mark is one comment
Longer tasks in real codebases
“They cover these longer horizon tasks that need more than just writing a bit of code in one file.”
Comment from 27 Mar 2026 in “$500 GPU outperforms Claude Sonnet on coding benchmarks” “it directly tests model on REAL codebases”
Comment from 5 May 2026 in “Lessons for Agentic Coding: What should we do when code is cheap?”
What people warn about
Each mark is one comment
Tasks underdetermined or mismarked
“their analysis demolished the validity of SWE-Bench Pro, objectively: it is being mismarked.”
Comment from 28 May 2026 in “Claude Opus 4.8” “15.0% of the 728 public tasks are underdetermined”
Comment from 26 Jul 2026 in “Claude Opus 5” Gamed and flawed, untrustworthy
“SWE Bench Pro is also gamed and shouldn't be trusted.”
Comment from 9 Jul 2026 in “GPT-5.6” “Now SWE-Bench Pro is shown to have flaws.”
Comment from 8 Jul 2026 in “Separating signal from noise in coding evaluations” “SWE-Bench pro is pretty much useless now”
Comment from 9 Jul 2026 in “GPT-5.6”
Other coding benchmarks people recommend
The three most recommended other coding benchmarks, with the same counts for each, so you can compare like for like. The tick on each bar marks the 2 to 1 line the badge needs.
| Product | Recommend | Criticize | Approval | Months | Owned | Badge |
|---|---|---|---|---|---|---|
| FrontierCode | 14 | 4 | 78% | 5 | — | Approved |
| Terminal-Bench | 12 | 11 | 52% | 10 | — | — |
| SWE-Bench ProThis page | 5 | 6 | 45% | 4 | — | — |
| SWE-bench | 5 | 17 | 23% | 2 | — | — |
What people said
All 11 comments that mention it, newest first. Each links to the original comment.
“15.0% of the 728 public tasks are underdetermined”
“SWE Bench Pro is also gamed and shouldn't be trusted.”
“SWE-Bench pro is pretty much useless now”
“But that leaves ~70% un-broken, which seems pretty good to me.”
“Now SWE-Bench Pro is shown to have flaws.”
“their analysis demolished the validity of SWE-Bench Pro, objectively: it is being mismarked.”
“it directly tests model on REAL codebases”
“empirically seems to suffer less from contamination issues”
“OpenAI recommended switching to SWE-bench Pro as a better (but still imperfect) benchmark”
“They cover these longer horizon tasks that need more than just writing a bit of code in one file.”
“the page gives impression that the language, category boxes are selectable”
Counts come from 1,795,640 Hacker News comments, November 2025 to November 2026. How we count. Report a mistake.