Baseten publishes DeepSeek engine, reports 2,060 tokens per second on eight B300s

Baseten published the code for DeepSeek Ultrafast, an inference engine for the existing DeepSeek V4.1 Flash model, on Nov. 13 UTC. In a test that streamed the code for a Tetris game, the AI infrastructure company reported 2,060 output tokens per second on eight Nvidia B300 GPUs. Baseten measured the speed on that configuration using one coding request.

Teams with compatible GPU servers can now run the same DeepSeek weights on Baseten’s engine and time their own requests. DeepSeek’s earlier V4.1 Flash release accepts text and images and produces text. To run it on their own infrastructure, teams need both the weights, which hold the values learned during training, and software that uses those weights to compute a response.

DeepSeek documents a context window of up to one million tokens, the chunks of text the model processes. A coding agent can consume much of its input by sending repository files and revision history together. Even a short answer can take time to begin when the model has substantial material to process first.

Reading the prompt and generating the answer

Baseten timed input processing, known as prefill, separately from the generation of new tokens, known as decoding. Prefill for one million input tokens took 11.7 seconds on the eight B300s. On four GB300s, the corresponding prefill took 13.2 seconds and the engine generated code at 1,740 tokens per second. Baseten’s social post calls the four-GPU setup B300. Its published benchmark table lists GB300.

The coding test requested the HTML for a Tetris game with the model’s reasoning mode turned off. Baseten’s example reports the wait for the first response and then tracks how quickly output arrives. The two rows change both GPU count and the hardware named in the table, so the difference alone cannot isolate the effect of doubling the number of GPUs.

A user’s wait also depends on how long the request sits in a queue and how long it takes data to cross the network. Nvidia’s guide to inference benchmarks distinguishes the speed at which one user receives a stream from the amount of work that the system processes across requests. The 2,060 figure comes from one stream. The published benchmark does not report throughput or wait times under concurrent traffic.

Reusing earlier work when a request changes servers

Baseten lists four GB300s, or four or eight B300s or B200s, among the supported configurations. Running the engine requires CUDA 13 and the DeepSeek weights. Baseten wrote the engine in Rust and CUDA. Its server accepts requests in a format compatible with OpenAI clients, which lets tools that support that format send requests to an endpoint on the team’s own infrastructure.

Baseten also describes what happens when a conversation moves between servers. A server retains intermediate calculations from the earlier turns in memory as a key-value cache. If another server receives the next turn, it uses Redis to locate the cache and transfers it directly between the machines’ memory over RDMA. The repository includes a Kubernetes example with two inference server replicas and a Redis instance.

The engine is distributed under the Elastic License 2.0. Its terms permit uses such as modifying and distributing the code, with conditions. One restricts offering it as a hosted or managed service that exposes a substantial set of the software’s features to third parties. DeepSeek distributes its model weights separately, and the model card lists the MIT license for the model.

Baseten already sells hosted access to the DeepSeek V4.1 Flash Fast model through its API, which it launched on Oct. 8. The company lists a price of $0.60 for regular input, $0.014 for cached input and $2.40 for output, each per million tokens. A deployment using the published engine on a team’s own server has its own GPU and operating costs.