# SWE-bench

5 Hacker News commenters recommended SWE-bench and 17 criticized it, November 2025 to November 2026. It was recommended in 2 of 13 months.

**Verdict:** Benchmark of AI coding agents on real GitHub issues, scored by automated tests. Critics say it is saturated and its public data invites gaming.

## Why people recommend it

- Scores patches with automated tests (2 comments): "software engineering has a ton of tradition and infrastructure for making and using automated tests" ([source](https://news.ycombinator.com/item?id=47912734)) "The evaluation harness only needs the final patch." ([source](https://news.ycombinator.com/item?id=47757106))

## What people warn about

- Public data invites training and gaming (5 comments): "There is absolutely no reason to care about a benchmark whose dataset has been publicly available for a while." ([source](https://news.ycombinator.com/item?id=47736312)) "It's based on existing GitHub PRs and Issues, the full dataset is on HuggingFace and is one year old now." ([source](https://news.ycombinator.com/item?id=47733559)) "the SWE bench examples are particularly horrible as those are just publicly available historical PRs" ([source](https://news.ycombinator.com/item?id=48584529))
- Saturated and outdated benchmark (4 comments): "benchmarks like SWEbench are all saturated and gamed/trained on" ([source](https://news.ycombinator.com/item?id=46829925)) "SWE-bench series just aren't that great by today's standard" ([source](https://news.ycombinator.com/item?id=48849777)) "SWE-Bench has done its part and we need a new benchmark" ([source](https://news.ycombinator.com/item?id=47568907))
- Flawed and exploitable by design (4 comments): "Designing benchmarks resistant to adversarial attempts to exploit the benchmark software is just something no one was thinking about when they created SWE-bench." ([source](https://news.ycombinator.com/item?id=47735387)) "Didn't we all know from the start that all of SWE-Bench was flawed?" ([source](https://news.ycombinator.com/item?id=48837694)) "Frontier model developers do not consider SWE-bench to be reliable." ([source](https://news.ycombinator.com/item?id=47733799))
- Tests miss hard, valuable work (3 comments): "SWE-bench scores well in the narrow task of making tests pass, which means models get good at exactly that." ([source](https://news.ycombinator.com/item?id=47344364)) "commonly used benchmarks like SWE-bench are broken as they are not testing the right / valuable things" ([source](https://news.ycombinator.com/item?id=47344709)) "they are definitely useful but they miss the things that are hard to encode in tests" ([source](https://news.ycombinator.com/item?id=47344152))

## Alternatives hackers recommend

- [Terminal-Bench](https://hackerapproved.com/p/terminal-bench.md): 12 recommend
- [SWE-Bench Pro](https://hackerapproved.com/p/swe-bench-pro.md): 5 recommend

## Cite as

Hacker Approved, "SWE-bench", https://hackerapproved.com/p/swe-bench/ (counts from 1,795,640 Hacker News comments, November 2025 to November 2026).