MLX

“I still use ollama due to mlx which of my understanding is not offered on vLLM”

“the insanse community work on Apple Silicon across whisper.cpp/mflux/llama.cpp/MLX/Exo”

“MLX uses pretty simple quantization methods so at lower bits without QAT, it is just not as good quality as llama.cpp ones.”

“You can take advantage of larger amounts of memory, higher memory bandwidth, and clustering multiple systems.”

“MLX on the software layer will not magically place hardware matrix multiplication units in your GPU cores.”

“FWIW, Ollama, LM Studio and Lemonade (and oMLX) also wrap Apple's MLX framework.”

“I do find that it reliably crashes on the first try, but then it picks up from where it finishes and completes the conversion reliably.”

“has some of the fastest updates upon the release of new models”

“Converting a model to mlx is literally a git clone and mlx_lm.convert. 5 minute operation after clone.”

“the MLX folks and the community usually have MLX versions of new model releases up within a day or two”

“With the model using MLX the speed increase is night and day.”

“MLX in the forms I have tried (LM Studio, oMLX) with the models I have tried (Gemma and Qwen) have both been apparently slower, yes.”

“MLX was and will not be able to access the ANE, so it's a total non-starter for anything user-facing”

“then shipped MLX to stop Tensorflow and Pytorch from replacing Apple's role in the stack entirely”

“I've found them to consistently reduce performance, contrary to what I've read online”

“Or a single mlx cluster if one can find second hand machines somewhere”

“MLX models have a higher time to first token (sometimes by an order of magnitude)”

“supports inference via mlx, which has better performance on apple silicon than llama.cpp”

“MLX tends provide much better performance than GGUF/Llama.cpp on macOS.”

“I found MLX variants to perform consistently worse (in terms of expected output, not speed) than GGUF”

“it worked fairly ok but definitely produced uncomfortable levels of heat”

“for the best performance on a Mac you want to use an MLX model”

“MLX is faster because it has better integration with Apple hardware.”

“MLX is a bit faster (low double digit percentage), but uses a bit more RAM. Worthwhile tradeoff for many.”

“MLX definitely works faster when it comes to extracting entities and triplets in batches”

“make sure to use the MLX over the GGUF version! You'll get a bit more speed out of it.”

“Sometimes I’m getting much faster responses but it feels like the quality is worse”

“I would most likely bet on CoreML since it have support for ANE (apple NPU) or MLX.”

“bugs keep getting piled up like they don't care if MLX is actually used in practice”

“had Claude Code implement a working diffusion forward pass in Swift + MLX”

“Apple is really seeing huge success with MLX, with an adoptable software stack that the PC world is super struggling to deliver.”

“Migrated to Apple's MLX framework and brought it back to 7s while maintaining 100% clinical pass rates.”

“Many user benchmarks report up to 30% better memory usage and up to 50% higher token generation speed”

“MLX versions specifically optimized for Apple Silicon will also give you noticeably better tok/s”

“MLX GPU kernels are 1.3x-3.1x faster than identical NumPy CPU kernels on compute-heavy queries”

“MLX has problems with kv cache consistency and especially with branching”

“the MLX library uses loads of undocumented properties in metal which isn't cool”

“The MLX inference is so fast that you could accomplish something like the latter”

“try out the ollama/lmstudio/mlx providers to see how they really perform”

“MLX is a lot more performant than Ollama and llama.cpp on Apple Silicon”