llama.cpp

What hackers pick instead

• OllamaRecommended for running open models locally with one command, and for its cheap, generous Ollama Cloud plan; critics cite poor defaults, slow new-model support, and locked-in model storage.350recommend
• LM StudioAn easy way to download and run local LLMs, with a model browser, hardware-fit hints and a headless API server, though critics object to it being closed source.217recommend
• DS4 by antirezRuns DeepSeek V4 Flash on home hardware like Macs, with well-made quants and quality some find close to Claude, though it's slow and its reasoning can derail.109recommend
• Unsloth StudioAn open-source local AI app with web search and tool calling, from the makers of fast-arriving, high-quality quantized models, though installation and Python setup are hard, especially on macOS.94recommend
• oMLXRecommended for fast local models on Apple Silicon, with KV caching that keeps long agent sessions quick. Critics find it slower than llama.cpp in some tests and sometimes unstable.62recommend

Every quote

The 40 most detailed of 853 comments. Each links to the original.

1. The official llama.cpp ROCm build detected the card correctly, but kept crashing with 0xC0000005 while loading the model, so I switched to the Windows Vulkan build of llama.cpp instead, which worked reliably.

   Mixed · Sep 2026 · in “Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses”

2. Since Llama.cpp/llama-server recently added support for the Anthropic messages API, running Claude Code with open-weight local models is now very easy.

   Recommends · Feb 2026 · in “Claude Code: connect to a local model when your quota runs out”

3. llama.cpp is designed for partial offloading, the most important part of the model will be loaded into the GPU and the rest on system ram

   Recommends · Feb 2026 · in “Unsloth Dynamic 2.0 GGUFs”

4. I just run cmake and build with no special options. Same thing with llama-server, I just specify the model and use the built-in web UI.

   Recommends · Aug 2026 · in “Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with Llama.cpp”

5. ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.

   Recommends · Aug 2026 · in “llama.cpp”

6. It's not as bad as you might think to compile llama.cpp for your target architecture and spin up an OpenAI compatible API endpoint.

   Recommends · Mar 2026 · in “Can I Run AI locally?”

7. I get in the habit of building llama.cpp from upstream head and checking for updated quantizations right before I start using it

   Recommends · Apr 2026 · in “April 2026 TLDR Setup for Ollama and Gemma 4 26B on a Mac mini”

8. You can try to offload the experts on CPU with llama.cpp (--cpu-moe) and that should give you quite the extra context space

   Recommends · Apr 2026 · in “Qwen3.6-35B-A3B: Agentic Coding Power, Now Open to All”

9. llama.cpp having 2 different settings.json (and Google lying about where 1 goes) made it far too difficulty to figure out.

   Mixed · Aug 2026 · in “Why your local LLM feels dumber than it is”

10. just dropping down to llama-cpp's GGUF model support and asking for default settings has provided faster inference speeds

    Recommends · May 2026 · in “Qwen3.7-Max: The Agent Frontier”

11. it's not difficult to make llama-server available to your laptop over just about any form of internet connection and VPN

    Recommends · Apr 2026 · in “Running Local LLMs Offline on a Ten-Hour Flight”

12. That allows you to force a certain number of experts to be kept in system memory while everything else is kept in VRAM.

    Recommends · Dec 2025 · in “I got an Nvidia GH200 server for €7.5k on Reddit and converted it to a desktop”

13. with llama-cpp and offloading non-active experts (from MOE architecture) to cpu RAM, you can easily run 50 tok / s

    Recommends · Jun 2026 · in “GateGPT: 56k tokens per second Transformer (KV cache) on FPGA at 80 MHz”

14. A better software when you're used to it all is llama.cpp as it's usually a bit faster and more frequently updated

    Recommends · Jun 2026 · in “Ask HN: How close are we to local LLMs being useful? What's the impact?”

15. compiled for CUDA backend, but there are also prebuilt binaries and instructions for other backends in the README

    Recommends · Jan 2026 · in “GLM-4.7-Flash”

16. I found the llama.cpp build process to work on the first try on my machine, and it only takes a couple minutes

    Recommends · Jun 2026 · in “Gemma 4 12B: A unified, encoder-free multimodal model”

17. I'm hitting OOM fairly often right now, it turns out that the sliding window attention context is huge and llama.cpp wants to keep lots of context snapshots.

    Criticizes · Apr 2026 · in “Claude Opus 4.7”

18. the llama.cpp implementation for this model is still quite half-baked due to missing the support for DSA sparse attention mechanism

    Criticizes · Jun 2026 · in “Unsloth GLM-5.2 – How to Run Locally”

19. llama.cpp has backed itself into a bit of a corner architecturally by trying to support all models on all possible backends

    Criticizes · Aug 2026 · in “GLM-5.3: Frontier Coding with Emergent Cyber Capabilities”

20. It is unlikely that llama.cpp will include any time soon code capable of reading SSDs at the speed limited by hardware.

    Criticizes · Oct 2026 · in “Janas: Run LLMs larger than RAM by streaming experts from SSD”

21. the experience is not pleasant, low pcie (gpu ram) utilization, low gpu utilization, and really low tokens per second

    Criticizes · Mar 2026 · in “Flash-Moe: Running a 397B Parameter Model on a Mac with 48GB RAM”

22. finding out how to download and install models was a pain in Llama.cpp and it was also _much_ slower than Ollama.

    Criticizes · Apr 2026 · in “Stop Using Ollama”

23. the bottleneck is the 48 hours of guessing llama.cpp flags and chat template bugs before the ecosystem catches up

    Criticizes · Apr 2026 · in “Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model”

24. There's also a regression in llama.cpp wrt. Step Flash, where quantisation is getting worse KLD and Perplexity

    Criticizes · Jun 2026 · in “Ask HN: Has anyone replaced Claude/GPT with a local model for daily coding?”

25. A big problem with the existing engines like llama or sd is that they don’t support optimal graph compilation.

    Criticizes · Aug 2026 · in “Why we write our own C and C++ inference engines”

26. The performance of gemma-4-31B was affected by tokenizer bugs in many of the existing backends, like llama.cpp

    Criticizes · Apr 2026 · in “I ran Gemma 4 as a local model in Codex CLI”

27. Repurposing a decomed server or desktop with a GPU (3090 or RTX PRO 6000 Blackwell not DC class) with linux/podman and llama.cpp will help a team understand without much cost

    Recommends · Apr 2026 · in “Claude.ai Is Unavailable”

28. If we can make multi-GPU systems contribute compute instead of mostly acting as extra VRAM, a lot of users suddenly gain access to much faster local inference

    Recommends · Sep 2026 · in “Show HN: Llama.cpp fork with 2-4x multiGPU speed for MoE models bigger than VRAM”

29. Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention

    Recommends · Aug 2026 · in “Qwen 3.8 27B is out: open weights, best local dense model yet”

30. I'd be fine with storing both the LLM and the projection in a single GGUF if llama.cpp allowed dynamically loading and unloading the projection

    Mixed · May 2026 · in “What's in a GGUF, besides the weights – and what's still missing?”

31. llama.cpp probably the closest to deliver on "just use it, don't worry about speed" if your focus is about a pure LLM inference engine.

    Recommends · Aug 2026 · in “Run Qwen3.8 27B locally: real numbers from my Mac Studio”

32. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher.

    Mixed · Sep 2026 · in “Shapelearn Qwen 3.8 27B (13.1 GB VRAM)”

33. the somewhat baffling llama_chat_apply_template exposed in the libllama API, which hardcodes a handful of chat formats directly in C++

    Mixed · May 2026 · in “What's in a GGUF, besides the weights – and what's still missing?”

34. llama.cpp includes a tool specifically for benchmarking that will sweep the arguments for you so you don't have to restart the server

    Recommends · Jun 2026 · in “How to Setup a Local Coding Agent on macOS”

35. Until now llama.cpp was enough, since I was on an R9700 and it was only practical to run one agent with decent sized context anyway.

    Mixed · Oct 2026 · in “DeepInfer: High-performance inference framework for DeepSeek models”

36. if you only use it for a personal vibe coding assistant and less as a general home model for task execution llama.cpp may be better

    Recommends · Aug 2026 · in “Qwen3.8 27B scores 52 on Artificial Analysis”

37. llama.cpp serving gemma4-26b-a4b for image processing of things too sensitive to let openai and other similar companies to see it

    Recommends · Oct 2026 · in “Ask HN: What do you run on a $5 VPS that's worth keeping online 24/7?”

38. When the text is especially difficult for tesseract, I can use Gemma3-4B via llama.cpp's llama-mtmd-cli, but that takes a minute.

    Mixed · Nov 2025 · in “I hate screenshots of text”

39. that stores conversations to disk, so if you change conversations, or have multiple windows open the conversation goes to disk

    Recommends · Aug 2026 · in “Local+privacy+focus on cache AI CLI and llama.cpp branch + Qwen3.x”

40. This takes about 3 seconds to load from an SSD, and is smart enough to solve exactly these "remind me of the syntax" scenarios

    Recommends · Sep 2026 · in “Ember-1”