SWE-bench
Shares rank 3 of 4 coding benchmarks by recommenders
Benchmark of AI coding agents on real GitHub issues, scored by automated tests. Critics say it is saturated and its public data invites gaming.
Benchmark of AI coding agents on real GitHub issues, scored by automated tests. Critics say it is saturated and its public data invites gaming.
22 people gave an opinion
- recommend it
- 5
- criticize it
- 17
- approval share
- 23%
Recommended in 2 of 13 months
- November 2025: not recommended
- December 2025: not recommended
- January 2026: not recommended
- February 2026: not recommended
- March 2026: recommended
- April 2026: recommended
- May 2026: not recommended
- June 2026: not recommended
- July 2026: not recommended
- August 2026: not recommended
- September 2026: not recommended
- October 2026: not recommended
- November 2026: not recommended (month in progress)
Why it doesn’t carry the badge
- 5 people recommend itNeeds 10 or more, not met.
- 0.2 recommenders for every criticNeeds 2 to 1 or better, not met.
- Recommended in 2 of 13 monthsNeeds 4 or more, not met.
The same three rules apply to every product, and the badge can’t be bought. How the badge works
What people say about it
The reasons people give, with the number of comments that give each one. Open a reason to read what they said.
Why people recommend it
Each mark is one comment
Scores patches with automated tests
“software engineering has a ton of tradition and infrastructure for making and using automated tests”
Comment from 26 Apr 2026 in “Why SWE-bench Verified no longer measures frontier coding capabilities” “The evaluation harness only needs the final patch.”
Comment from 13 Apr 2026 on Hacker News
What people warn about
Each mark is one comment
Public data invites training and gaming
“There is absolutely no reason to care about a benchmark whose dataset has been publicly available for a while.”
Comment from 12 Apr 2026 in “How We Broke Top AI Agent Benchmarks: And What Comes Next” “It's based on existing GitHub PRs and Issues, the full dataset is on HuggingFace and is one year old now.”
Comment from 11 Apr 2026 in “How We Broke Top AI Agent Benchmarks: And What Comes Next” “the SWE bench examples are particularly horrible as those are just publicly available historical PRs”
Comment from 18 Jun 2026 in “Local Qwen isn't a worse Opus, it's a different tool” - Comment from 27 Mar 2026 in “$500 GPU outperforms Claude Sonnet on coding benchmarks”
- Comment from 30 Jan 2026 in “Show HN: Arenas suck, here's why we just added one to Windsurf”
Saturated and outdated benchmark
“benchmarks like SWEbench are all saturated and gamed/trained on”
Comment from 30 Jan 2026 in “Show HN: Arenas suck, here's why we just added one to Windsurf” “SWE-bench series just aren't that great by today's standard”
Comment from 9 Jul 2026 in “GPT-5.6” “SWE-Bench has done its part and we need a new benchmark”
Comment from 30 Mar 2026 in “SWE-bench will hit 90% this year” - Comment from 19 Nov 2025 in “Gemini 3 Pro Model Card [pdf]”
Flawed and exploitable by design
“Designing benchmarks resistant to adversarial attempts to exploit the benchmark software is just something no one was thinking about when they created SWE-bench.”
Comment from 12 Apr 2026 in “How We Broke Top AI Agent Benchmarks: And What Comes Next” “Didn't we all know from the start that all of SWE-Bench was flawed?”
Comment from 8 Jul 2026 in “Separating signal from noise in coding evaluations” “Frontier model developers do not consider SWE-bench to be reliable.”
Comment from 11 Apr 2026 in “How We Broke Top AI Agent Benchmarks: And What Comes Next” - Comment from 28 May 2026 in “Claude Opus 4.8”
Tests miss hard, valuable work
“SWE-bench scores well in the narrow task of making tests pass, which means models get good at exactly that.”
Comment from 12 Mar 2026 in “Many SWE-bench-Passing PRs would not be merged” “commonly used benchmarks like SWE-bench are broken as they are not testing the right / valuable things”
Comment from 12 Mar 2026 in “Many SWE-bench-Passing PRs would not be merged” “they are definitely useful but they miss the things that are hard to encode in tests”
Comment from 11 Mar 2026 in “Many SWE-bench-Passing PRs would not be merged”
Other coding benchmarks people recommend
The three most recommended other coding benchmarks, with the same counts for each, so you can compare like for like. The tick on each bar marks the 2 to 1 line the badge needs.
| Product | Recommend | Criticize | Approval | Months | Owned | Badge |
|---|---|---|---|---|---|---|
| FrontierCode | 14 | 4 | 78% | 5 | — | Approved |
| Terminal-Bench | 12 | 11 | 52% | 10 | — | — |
| SWE-benchThis page | 5 | 17 | 23% | 2 | — | — |
| SWE-Bench Pro | 5 | 6 | 45% | 4 | — | — |
What people said
All 23 comments that mention it, newest first. Each links to the original comment.
“SWE-bench series just aren't that great by today's standard”
“SWE-bench Verified was created to fix the problems of SWE-bench.”
“Didn't we all know from the start that all of SWE-Bench was flawed?”
“the SWE bench examples are particularly horrible as those are just publicly available historical PRs”
“SWE-Bench has serious flaws, even the big boys say so”
“the scrutiny is a byproduct of the adoption and success of the benchmark”
“SWE-bench, which is derived from real units of labor”
“software engineering has a ton of tradition and infrastructure for making and using automated tests”
“The evaluation harness only needs the final patch.”
“There is absolutely no reason to care about a benchmark whose dataset has been publicly available for a while.”
“Designing benchmarks resistant to adversarial attempts to exploit the benchmark software is just something no one was thinking about when they created SWE-bench.”
“Frontier model developers do not consider SWE-bench to be reliable.”
“It's based on existing GitHub PRs and Issues, the full dataset is on HuggingFace and is one year old now.”
“SWE-Bench has done its part and we need a new benchmark”
“they are already being gamed hard so the developers can quote high numbers”
“SWE-Bench is a relatively decent benchmark and perf here is continually improving”
“they limit it to organization only”
“commonly used benchmarks like SWE-bench are broken as they are not testing the right / valuable things”
“SWE-bench scores well in the narrow task of making tests pass, which means models get good at exactly that.”
“they are definitely useful but they miss the things that are hard to encode in tests”
“it's not a great benchmark anymore... starting with it being python / django primarily”
“benchmarks like SWEbench are all saturated and gamed/trained on”
“swebench is (1) terrible and (2) saturated”
Counts come from 1,795,640 Hacker News comments, November 2025 to November 2026. How we count. Report a mistake.