ninfer
Ranked 5 of 27 inference engines by recommenders
Free local inference engine that people report running Qwen models at high token speeds on single consumer GPUs, often faster than llama.cpp. Critics note one benchmark slower than vLLM.
Free local inference engine that people report running Qwen models at high token speeds on single consumer GPUs, often faster than llama.cpp. Critics note one benchmark slower than vLLM.
18 people gave an opinion
- recommend it
- 16
- criticize it
- 2
- approval share
- 89%
Recommended in 3 of 13 months
- November 2025: not recommended
- December 2025: not recommended
- January 2026: not recommended
- February 2026: not recommended
- March 2026: not recommended
- April 2026: not recommended
- May 2026: not recommended
- June 2026: not recommended
- July 2026: not recommended
- August 2026: recommended
- September 2026: recommended
- October 2026: recommended
- November 2026: not recommended (month in progress)
Why it doesn’t carry the badge
- 16 people recommend itNeeds 10 or more, met.
- 8 recommenders for every criticNeeds 2 to 1 or better, met.
- Recommended in 3 of 13 monthsNeeds 4 or more, not met.
The same three rules apply to every product, and the badge can’t be bought. How the badge works
What people say about it
The reasons people give, with the number of comments that give each one. Open a reason to read what they said.
Why people recommend it
Each mark is one comment
Very fast token generation on GPUs
“Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer”
Comment from 17 Aug 2026 in “Qwen3.8-27B at 256K on a 24GB RTX PRO 4000 SFF (432 GB/s): 50 tok/s with MTP” “I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast”
Comment from 3 Sep 2026 in “Qwen 3.8 27B available on Cerebras at 1500 tokens/s” “gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup”
Comment from 14 Aug 2026 in “Qwen 3.8 27B is out: open weights, best local dense model yet” - Comment from 23 Aug 2026 in “Why your local LLM feels dumber than it is”
- Comment from 25 Aug 2026 in “Qwen 3.8-Flash-Next releasing tomorrow (125B a6B)”
- Comment from 21 Sep 2026 in “M5 Ultra Mac Studio Review”
- Comment from 21 Aug 2026 in “A self hosted AI software factory”
- Comment from 29 Aug 2026 in “Run Qwen3.8 27B locally: real numbers from my Mac Studio”
- Comment from 29 Aug 2026 in “Run Qwen3.8 27B locally: real numbers from my Mac Studio”
- Comment from 15 Aug 2026 in “Qwen 3.8 27B is out: open weights, best local dense model yet”
Works as advertised
“really cool and works as advertised”
Comment from 29 Aug 2026 in “Run Qwen3.8 27B locally: real numbers from my Mac Studio” “which it does outstandingly well”
Comment from 5 Oct 2026 in “Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s” Faster than llama.cpp
“it is impressive. The place that it's doing better at than llama-cpp seems to really be late in the context window.”
Comment from 4 Sep 2026 in “Apple caught off guard by AI demand for Mac Mini and Mac Studio” “gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup”
Comment from 14 Aug 2026 in “Qwen 3.8 27B is out: open weights, best local dense model yet” Good for single-GPU setups
“Definitely worth looking at if you have only a single 5090 is ninfer”
Comment from 3 Oct 2026 in “From the creator of Redis; run LLM locally with ds4” “ninfer-3090 is worth checking out if you use just one GPU.”
Comment from 26 Oct 2026 in “Anthropic S-1”
Other inference engines people recommend
The four most recommended other inference engines, with the same counts for each, so you can compare like for like. The tick on each bar marks the 2 to 1 line the badge needs.
| Product | Recommend | Criticize | Approval | Months | Owned | Badge |
|---|---|---|---|---|---|---|
| vLLM | 112 | 56 | 67% | 12 | — | Approved |
| Strata | 31 | 15 | 67% | 5 | — | Approved |
| DwarfStar 4 | 26 | 2 | 93% | 6 | — | Approved |
| SGLang | 19 | 13 | 59% | 8 | — | — |
| ninferThis page | 16 | 2 | 89% | 3 | — | — |
What people said
22 of the 25 comments that mention it, newest first. Each links to the original comment.
“ninfer-3090 is worth checking out if you use just one GPU.”
“you can try my fork to have a JEV compatible decisions endpoint with qwen-3.8-27b”
“which it does outstandingly well”
“Definitely worth looking at if you have only a single 5090 is ninfer”
“You can get over 200 tps with ninfer using NVFP4 and MTP.”
“it is impressive. The place that it's doing better at than llama-cpp seems to really be late in the context window.”
“I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast”
“kv caching is not perfect for ninfer”
“really cool and works as advertised”
“you can get 1200 of prefill at 80 of decode”
“Ninfer is the real deal. 160 tokens/sec.”
“I haven't tried ninfer yet.”
“I forked ninfer with the obsession to squeeze everything out of this card for this model”
“I'm definitely bookmarking this. Thanks!”
“can pull ~180 TPS on 5090 with 3.8, and 500 (!) with 3.6 35B-A3B”
“Using ninfer on 5090 ... is also kind of cool to get a local cerebras experience”
“With an RTX5090 card and ninfer, you can get ~800 TPS token generation”
“I can run Qwen 3.8 27B at ~180 TPS with ninfer”
“And on my "own" "quick" benchmark, it's slower than vllm.”
“Folks over on /r/localllama have been reporting wild prefill/token gen speeds with ninfer”
“testing it now and it flies!”
“gets me ~138 tokens/second, roughly double what I get with a naive llama.cpp setup”
Counts come from 1,795,640 Hacker News comments, November 2025 to November 2026. How we count. Report a mistake.