Apple's MLX runs an 816B-parameter model on four Mac Studios
3-line summary
- Apple showed how a new version of its MLX framework can link several Mac Studios to run very large models.
- Four Mac Studios connected over Thunderbolt ran Muse Spark Open, an 816-billion-parameter open model, at more than 36 tokens per second.
- M5 Ultra delivers the first token up to 4.6 times faster than M3 Ultra, with peak power of about 300 W.
Apple's machine learning research team released a new version of the open-source MLX framework and showed how to run large language models directly on Mac Studio. One Mac Studio can have up to 512 GB of unified memory, and bigger models are split across several machines linked by Thunderbolt cables.
The showcase is Meta's open model Muse Spark Open, a mixture-of-experts model with 816 billion parameters, 42 billion of them active. Even in 4-bit NVFP4 it needs about 493 GB, too much for one machine. Split across four Mac Studios with M5 Ultra, it generated 36.73 tokens per second, which Apple notes is well above average human reading speed.
There are two ways to split a model. Pipeline parallelism divides the layers front to back and runs them in turn, which is simple to implement; tensor parallelism slices each layer across machines that compute at the same time, which is faster but needs frequent communication. Apple says the low-latency RDMA over Thunderbolt 5 added in macOS 26.2, plus new communication features in MLX, made tensor parallelism pay off.
Apple also compared chip generations. Thanks to the Neural Accelerators in its GPU, M5 Ultra cut time to first token by up to 4.6 times versus M3 Ultra, and its higher memory bandwidth (1.2 TB/s) made generation 1.6 times faster. Peak power was about 200 W for M3 Ultra and 300 W for M5 Ultra. The figures come from models and settings Apple chose.
Why it matters
Models with hundreds of billions of parameters that once needed a data center can now run on a few desktop machines. That should widen the options for teams that can't send code outside the company but want to use coding agents in-house.
Who it's for
- Developers
- Start a local server with mlx_lm.server and point coding agents such as OpenCode at it. 27B- and 128B-class models fit on a single 512 GB Mac Studio, so try one machine first.
Sources
- Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio — Apple Machine Learning Research
