oMLX

“SSD token caching (like oMLX does) goes a long way for prefill.”

“I returned back to Ollama because of stability issues.”

“It's a lot smoother setup than Ollama and has a built in model sizer and all that in a nice ui.”

“omlx already supports all of this and has a 1000 people trying to optimize it constantly”

“Qwen3.8 27B seems to have weird memory and caching issues on oMLX”

“This helps with the usability a lot, when an agentic session is warm it runs pretty smoothly.”

“Use oMLX or something similar to serve the model with internal MTP enabled.”

“oMLX serves local models much, much faster than LM Studio”

“Just use qwen3.8-27b on a Mac with oMLX, ANE enabled and skip cloud models.”

“I switched to oMLX I get ~75 tok/s on Qwen3.6-35B-A3B-4bit on a 48GB M5 Pro”

“The good cacheing layer in oMLX will help things feel faster for some classes of tasks in my experience.”

“the main advantage of oMLX is it's less likely to invalidate the KV cache”

“offer some amazing performance optimizations for Apple Silicon that utilize UMA and NVME caching efficiently”

“They expose „open ai compatible“ endpoints, so you can configure almost any harness to use them.”

“keeps driving me up a wall with odd papercuts, bugs, and silent failures”

“In particular they have community benchmarks so you can see what kind of performance you're likely to get”

“I've been able to run local, quantized LLMs like Qwen 3.6 using oMLX + Opencode pretty comfortably and productively.”

“Has extra KV cache on SSD, and lots more options to tweak.”

“Omlx lets you run Claude code locally though as long as you bootstrap it with the right model”

“MLX in the forms I have tried (LM Studio, oMLX) with the models I have tried (Gemma and Qwen) have both been apparently slower, yes.”

“Pull request #1837 that enables tool calls on supported diffusion models was merged as 7c1971e today”

“I have not, on a 64GB M1 Max, seen any advantage from oMLX specifically or MLX generally over GGUF with llama.cpp.”

“gives you intelligent local hardware based model download recommendations”

“It truly is the SOTA for local inference on mac. Even when there are regressions the dev(s) are insanely responsive.”

“I have used omlx.ai with great success to both download multiple mlx models”

“oMLX does the caching I need to fit models that are near gross memory”

“makes running the mlx inference server quite easy for those interested in UI-based hosting”

“the oMLX performance with the MLX community MTP models is much worse”

“All that has to happen now is models improving (keeps happening) and oMLX, LM Studio, llamma, Ollama, etc.. to get simpler with more hand-holding and sane defaults.”

“it horribly kernel panics when it is running for too long”

“oMLX makes prefill effectively instantaneous on a Mac.”

“much more performant and efficient than LM Studio. It has block-level KV context caching that makes long chats and agentic/tool calling scenarios MUCH faster”

“TBH not seeing competitors in a benchmark makes it less a benchmark and more a data sheet”

“oMLX's token generation is 3–7x slower than llama.cpp's Metal backend.”

“The prompt decode caching keeps things coherent and fast even when contexts get north of 100k tokens.”

“generation is supported but not all special tokens are handled”

“With oMLX doing prompt caching I can run two in parallel doing different tasks pretty reasonably”

“omlx seems to me like the front runner right now for running MLX models locally in agent workflows”

“What really has been a game changer for my workflow is using https://omlx.ai/ that has SSD KV cold caching.”

“In practice, TTFT drops from 30-90s to 1-3s on follow-up requests.”