# SWE-Bench Pro

5 Hacker News commenters recommended SWE-Bench Pro and 6 criticized it, November 2025 to November 2026. It was recommended in 4 of 13 months.

**Verdict:** Recommended for testing AI coding models on longer tasks in real codebases, and seen as less contaminated than older benchmarks. Critics say many tasks are underdetermined or mismarked, so scores may be unreliable.

## Why people recommend it

- Longer tasks in real codebases (2 comments): "They cover these longer horizon tasks that need more than just writing a bit of code in one file." ([source](https://news.ycombinator.com/item?id=47540332)) "it directly tests model on REAL codebases" ([source](https://news.ycombinator.com/item?id=48023928))

## What people warn about

- Tasks underdetermined or mismarked (2 comments): "their analysis demolished the validity of SWE-Bench Pro, objectively: it is being mismarked." ([source](https://news.ycombinator.com/item?id=48314354)) "15.0% of the 728 public tasks are underdetermined" ([source](https://news.ycombinator.com/item?id=49055101))
- Gamed and flawed, untrustworthy (3 comments): "SWE Bench Pro is also gamed and shouldn't be trusted." ([source](https://news.ycombinator.com/item?id=48851818)) "Now SWE-Bench Pro is shown to have flaws." ([source](https://news.ycombinator.com/item?id=48837785)) "SWE-Bench pro is pretty much useless now" ([source](https://news.ycombinator.com/item?id=48849363))

## Alternatives hackers recommend

- [Terminal-Bench](https://hackerapproved.com/p/terminal-bench.md): 12 recommend
- [SWE-bench](https://hackerapproved.com/p/swe-bench.md): 5 recommend

## Cite as

Hacker Approved, "SWE-Bench Pro", https://hackerapproved.com/p/swe-bench-pro/ (counts from 1,795,640 Hacker News comments, November 2025 to November 2026).