Inkling 2 Small weights are out, with 500 GB of GPU memory needed for self-hosting

Thinking Machines Lab released downloadable weights for Inkling 2 Small on November 12. The lab had introduced the model on November 5 through Tinker, where developers could run and fine-tune it as a hosted service. The release puts the trained model files on Hugging Face under Apache 2.0, giving developers a way to serve the model in an environment they operate.

The lab published a main NVFP4 checkpoint and an alternative MXFP8 checkpoint. A checkpoint is a saved set of trained weights. The model card calls for at least 500 GB of combined GPU memory for the NVFP4 version and gives two Nvidia B300s or four H200s as examples. That figure describes the GPU memory needed to run the model. The amount of storage needed to download the weights is a separate figure.

How the weights fit on a server

The lab lists 594 billion parameters in all and about 17 billion active for a token. A token is a unit of input or output the model processes. Inkling 2 Small uses a mixture-of-experts design, which routes a token through eight of 384 specialized components. Different tokens can use different components. The active parameter count describes the work done for an individual token, while the full checkpoint includes the weights available to the model across those choices.

The Hugging Face file listings show 24 main weight files for the NVFP4 version and 48 for MXFP8. Added together, the listed sizes come to about 390 GB and 619 GB, respectively, in decimal gigabytes. The totals cover the numbered weight files and exclude auxiliary files. When the model is running, GPU memory also holds working data such as the cache used to reuse calculations from earlier tokens.

The model card puts the MXFP8 requirement at 800 GB of combined GPU memory, with four B300s or eight H200s as examples. Thinking Machines Lab recommends NVFP4. The lab reported that MXFP8 scored one to two percentage points higher in its evaluations. Those results describe the lab's tests. Throughput and response time depend on how the model is served.

The main checkpoint itself uses a mix of precisions. In its release-day write-up, the vLLM team said the experts in the first 40 of the model's 48 layers use NVFP4, while those in the final eight use MXFP8. The inference engine uses the precision specified for each layer when it runs those computations.

Running it or calling an API

SGLang and vLLM published support alongside the weight release. Their inference engines load the model on GPUs and handle incoming requests. Thinking Machines Lab also lists integrations with TokenSpeed and Unsloth. The vLLM deployment guide gives eight B200s as its default NVFP4 example and points to a dedicated container image pending support in a vLLM release. SGLang lists the MI350X and MI355X as its validated AMD GPUs for ROCm.

Cloudflare added Inkling 2 Small to Workers AI on November 12, giving developers a hosted API alongside the downloads. It requires a Workers Paid plan or prepaid AI Gateway credits. The company lists a default rate limit of 20 requests a minute under standard Workers AI billing and 50 with prepaid credits. Tinker remains available for hosted use and fine-tuning, according to the model card.

Inkling 2 Small accepts text and image inputs and returns text. The lab lists a context window of up to roughly one million tokens. Cloudflare specifies a 1,048,576-token context window for the model on Workers AI. The requests-per-minute limit applies separately.