Cloudflare says 155 GLM-5.3 replies exposed other customers’ data

Cloudflare disclosed on October 26 that 155 responses from its GLM-5.3 endpoint on Workers AI contained data from other customers’ requests. All affected replies identified by the company followed prompts longer than 681,280 tokens. The incident broke separation between requests sharing the AI serving environment.

The exposure window ran from August 27 to September 9. Cloudflare restored public access on September 9 after remediation, giving customers who sent long documents or code during that period a newly published account of the scope and recovery. The company says other Workers AI models were unaffected and that it found no evidence of deliberate exploitation.

The historical status page described increased errors when using GLM 5.3. It recorded investigation at 14:25, identification at 15:02 and resolution at 20:09 on September 9, all UTC. Those entries describe the service disruption; the October report supplies the subsequently disclosed cross-customer exposure details.

The context window exceeded the processing memory

The configured input limit was 1,048,576 tokens, while the GPUs processing incoming prompts had KV-cache capacity for 681,280. This cache stores intermediate information produced as the model processes input. A request could fit the advertised context window while exceeding the memory available to process it. Cloudflare says it knew about the mismatch before launch but interpreted successful HTTP responses in long-context testing as evidence it was handled.

A rejection-handling defect in the SGLang inference engine compounded that mismatch. A rejected request still appeared live to the admission queues. Its long prompt became a one-token placeholder, while the response-generating side could read memory left by earlier requests. The failure involved serving with prompt processing and response generation on separate machines.

The correction history shows how aborted work and rejected intake were retired at different stages. A change merged on September 6 cleans up a request when its final aborted prefill result returns. It clears pending transfers, releases memory and associated buffer ownership, and ignores results arriving after retirement.

That change’s validation covered final and intermediate chunks, delayed results and sender failures. Its author reported 39 selected tests passing in an H200 environment. The approach cleaned up ownership before the engine’s memory-accounting checks, while keeping those checks fatal for remaining errors. These are the change author’s stated tests.

The September 16 follow-up stops rejected requests before handoff, avoiding placeholder computation and transfer setup. In its comparison with both workers rejecting, a 300-second decode timeout became an immediate error response. The test used two H200 GPUs and Qwen3-0.6B with cache capacity below the input limit. Its author describes a waste-and-latency improvement on top of the earlier isolation correction. Fast peer notification when only one worker rejects remained outside its scope.

The developers also addressed memory reuse after an interrupted transfer. Writes already launched can arrive after cancellation and overwrite memory reassigned to another request. A related change holds that memory until transfer-drain acknowledgements arrive or a bounded timeout expires. This was a separate hardening path alongside the long-prompt rejection incident. The initial change was opt-in and disabled by default; the later disclosure identifies releases that enable protection by default.

SGLang’s October 26 disclosure distinguishes protection by deployment and release: default protection for Mooncake and NIXL configurations starts with 0.5.21, while MORI support and generation-tagged acknowledgements require 0.5.22. The team says it has not backported this fix line to earlier branches. Further corrections preventing failed transfers from being acknowledged as drained ship in 0.5.23.