humans& reports 10% more throughput with higher-precision four-bit AI

humans& says it increased inference throughput by raising the compute precision of a four-bit AI pipeline. Its October 23 report records 10% higher throughput and a 7% reduction in time between output tokens at eight concurrent requests.

The test ran GLM-5.2 with EAGLE on eight B300 GPUs, using 1,024 input and 8,192 output tokens. Throughput included input processing. At high concurrency, the two paths performed similarly. These are the company’s measurements, not an AI Ping reproduction.

FlashInfer’s implementation stores weights in NVFP4, then expands them to BF16 for calculation. Activations, the intermediate values, also stay in BF16. This W4A16 path removes their intervening conversion into four-bit form.

While the preceding operation finishes producing activations, the next operation loads and decodes its weights. Independent preparation proceeds early; only the activation-loading work waits for the preceding stage. Decoded weights go straight into on-chip tensor memory, without creating a separately repacked weight file.

For distributed mixture-of-experts workloads, MegaMoE folds input delivery, both matrix operations, output return and combination into one persistent kernel. It retains the existing packed NVFP4 weight layout. Two worker groups handle weight decoding. This implementation merged into FlashInfer on September 28.

An October 14 optimization added 32-token-wide tiles and 16 autotuning candidates for small batches. Its single-layer GLM-5 measurements improved at some sizes and regressed at others. After three warmups, each point used 100 samples and the median of the slowest GPU’s activity span per iteration. Those timings exclude attention, shared experts and routing preparation, giving them a different scope from the full serving result reported above.

SGLang merged the serving integration on October 16, using FlashInfer 0.7.1. W4A16 remains an opt-in mode; W4A4 is the default. Support requires SM100 or SM103, BF16 model data and equal attention-data, tensor and expert parallelism sizes. The published integration therefore has explicit hardware, numerical and topology requirements.

The weight representation also connects to reinforcement learning. Miles merged fused quantize–dequantize kernels on September 11. They retain packed values and block scales in registers, writing the restored BF16 or FP16 result instead of sending quantized intermediates through memory. The training recipe uses restored weights in BF16 matrix operations alongside a W4A16 rollout engine.

Miles documents an integration run with a five-layer GLM-5.2 configuration, eight B200 GPUs and two GRPO rollouts. Some numerical-equality checks were disabled while other diagnostics remained logged. The pull request describes this as integration coverage and explicitly excludes a claim of numerical parity between the training and rollout engines.