Promptfoo
Hacker ApprovedLLM evaluation tool in Developer tools
Open-source tool for testing prompts and model outputs with test cases run against any model. One critic is unhappy and finds it not worth investing in, given how fast things move.
Open-source tool for testing prompts and model outputs with test cases run against any model. One critic is unhappy and finds it not worth investing in, given how fast things move.
13 people gave an opinion
- recommend it
- 11
- criticize it
- 2
- approval share
- 85%
Recommended in 8 of 13 months
- November 2025: recommended
- December 2025: recommended
- January 2026: recommended
- February 2026: recommended
- March 2026: recommended
- April 2026: recommended
- May 2026: not recommended
- June 2026: not recommended
- July 2026: recommended
- August 2026: recommended
- September 2026: not recommended
- October 2026: not recommended
- November 2026: not recommended (month in progress)
Why it carries the badge
- 11 people recommend itNeeds 10 or more, met.
- 5.5 recommenders for every criticNeeds 2 to 1 or better, met.
- Recommended in 8 of 13 monthsNeeds 4 or more, met.
The same three rules apply to every product, and the badge can’t be bought. How the badge works
What people say about it
The reasons people give, with the number of comments that give each one. Open a reason to read what they said.
Why people recommend it
Each mark is one comment
Best open-source tool for evals
“Promptfoo has been the best open-source tool for it.”
Comment from 9 Mar 2026 in “Promptfoo Is Joining OpenAI” “There are tools like promptfoo designed for this.”
Comment from 14 Aug 2026 in “Qwen3.8-27B” “You've built an industry standard with Promptfoo.”
Comment from 9 Mar 2026 in “Promptfoo Is Joining OpenAI” Tests model outputs well
“Think of it like pytest but for prompts - you define test cases, run evals against any model”
Comment from 14 Dec 2025 in “Ask HN: What Are You Working On? (December 2025)” “my experience with Promptfoo is great at testing model outputs”
Comment from 10 Mar 2026 in “OpenAI Acquires Promptfoo” Evaluates new models and prompt changes
“These help when evaluating new models/versions and when modifying the system prompt to guide the model.”
Comment from 26 Apr 2026 in “Why SWE-bench Verified no longer measures frontier coding capabilities” “Think of it like pytest but for prompts - you define test cases, run evals against any model”
Comment from 14 Dec 2025 in “Ask HN: What Are You Working On? (December 2025)”
What people said
12 of the 16 comments that mention it, newest first. Each links to the original comment.
“their evals is not so good”
“the biggest thing I've done recently is start to be extremely critical about the output”
“There are tools like promptfoo designed for this.”
“it was exactly, what I needed, very much still a niche thing hardly anyone was using”
“These help when evaluating new models/versions and when modifying the system prompt to guide the model.”
“Definitely not happy with it, but everything is moving too fast to feel like it's worth investing in.”
“my experience with Promptfoo is great at testing model outputs”
“Promptfoo has been the best open-source tool for it.”
“I’ve been using it from the very beginning and it’s been great!”
“You've built an industry standard with Promptfoo.”
“Technically no programming required for an eval creator, just yaml”
“Think of it like pytest but for prompts - you define test cases, run evals against any model”
Counts come from 1,795,640 Hacker News comments, November 2025 to November 2026. How we count. Report a mistake.