A MacBook Air trained this miniature DeepSeek-V3 recreation

Independent developer Vaibhaav Tiwari released a 36.2 million-parameter language model that adapts DeepSeek-V3’s design and was trained on a MacBook Air. Tiwari published the training code and model weights on GitHub on November 14. A day later, the project merged a separate contributor’s setup for running the same training on an AMD Radeon graphics card, giving developers a Linux route to repeat the experiment with the published data.

The project is called deepseek-v3-jax. About 190 lines of JAX contain its core model logic. Separate files prepare the text and run the training. The release also supplies the resulting weights and training logs, so readers can inspect the finished run or start a fresh one. The repository makes the model code available under the MIT license.

Six layers trained on crime fiction

Tiwari used a MacBook Air M5 with 24 GB of memory and Apple’s Metal plug-in for JAX to train on the laptop’s GPU. He assembled a collection of 207 detective books from sources including Project Gutenberg, set aside text for validation and included Dashiell Hammett’s work three times in the training split. The books were broken into tokens, the chunks of text the model learns to predict.

The run took about 2.6 hours and covered 12,000 training steps. The source material in the training split amounted to about 21.7 million tokens. Multiple passes brought the total processed during training to about 98 million. The release includes files for both splits and the tokenizer, which turns text into the numbers the model reads.

DeepSeek reported 671 billion parameters for the original V3, with 37 billion activated for each token. Tiwari’s six-layer implementation uses about 13.6 million of its 36.2 million parameters per token. Its 256-token window covers the prompt and the continuation together. The author’s samples show short continuations of detective-fiction prompts.

A second run on a Linux desktop

A GitHub contributor using the name joebarbere ran the training on Fedora Linux with a 16 GB Radeon RX 7600 XT. The contributor added a container environment to connect JAX to the AMD GPU and kept the existing training code intact. The change landed in the main branch on November 15. The run notes document settings that let JAX recognize the card and work around a failure in one matrix-multiplication path.

The contributor reported about 50 minutes for 12,000 steps. The run logged a final validation loss of 3.527 on text held out from training. The Mac run logged 3.507 with the same published data files. The two runs used different versions of JAX, and the Radeon run used a different random initialization method. The timing and loss figures describe those two setups.

The contributor verified file hashes to confirm that both runs used identical training and validation files. The project also publishes the Mac run’s loss logs. Those records let readers compare the two runs at the same points during training as well as at the end.

One part of the design inherited from DeepSeek-V3 is a mixture of experts, in which the model routes each token through selected small networks. Tiwari’s implementation uses the output of four routed experts out of 16 and a shared expert. A balancing adjustment lowers the selection scores of heavily used experts during training. The Radeon run notes report that expert shares ranged from 5.4% to 6.8% with the adjustment, and from 0.07% to 18.0% in a separate run with it disabled.

The project also implements latent attention, which stores a compressed record of the tokens the model has already read. It reuses that record as it generates new text. A separate training objective asks the model to predict an extra token beyond the next one. The published sampler generates one token at a time.