SK hynix shared memory test cuts time to first AI response from 80 to 15 seconds

SK hynix showed how several AI servers could draw on a separate memory rack at Intel AI Summit Seoul on November 12. In a test with successive queries, the system produced the first piece of an answer in an average of 15.45 seconds when it retrieved previously processed context from remote memory. Recomputing that context took an average of 80.76 seconds.

The design presented by SK hynix vice president Youngpyo Joo keeps memory on each server and gives it access to another rack for data it needs to bring back later. That gives AI infrastructure teams a choice about where to keep earlier calculations when long chats or documents fill GPU memory. Joo said SK hynix was working with Intel and other partners at the system design stage.

What it costs to process the same context twice

When someone asks several questions about the same long report, the model has already performed some of the work needed to answer the next question. A key-value cache, or KV cache, holds intermediate results from processing that text. If the beginning of a new request matches earlier input, inference software can reuse the corresponding data and resume work from the new material.

Longer requests and more simultaneous requests take up more space in the memory near the GPU. A service answering several questions about one report may want to keep the calculations for the report available between turns. If the service frees that space for other requests, it must process that material again when the next question arrives or reload a copy it saved elsewhere.

In Seoul, SK hynix compared three approaches. Recomputing the earlier context took 80.76 seconds on average. Retrieving the data from a remote solid-state drive took 33.56 seconds, and retrieving it from remote DRAM connected through Compute Express Link, or CXL, took 15.45 seconds. The last result was about 81 percent below the recomputation baseline. The metric, time to first token, measures how long it takes the model to start producing an answer. The time to finish the full response is a separate measure.

The figures came from an evaluation by SK hynix and its partners. The presentation slide did not specify the model, input lengths or how many requests the system handled at once. The chart compared recomputation with two kinds of remote retrieval. The chart gives no timing for a cache that is still available in GPU memory. The inference software project vLLM also notes that reusing context helps when requests share a common beginning. The gains shrink when requests have little overlap or the model spends most of its time generating a long answer.

How multiple servers can draw from one memory rack

A diagram in the presentation placed network switches in both a GPU rack and a standalone memory rack. Joo described adding tens to hundreds of terabytes of capacity in a separate rack alongside GPU and CPU servers. Cache data could move across the network between racks, while CXL connects processors and additional memory devices within a rack. The GPUs still run the model. The separate rack can hold data that needs to remain available for longer. Several servers can draw on that capacity, reducing the need to provision each server for its own peak demand.

SK hynix demonstrated pooled CXL memory at the Open Compute Project Global Summit in October. It also showed a system that kept a full copy of the cache on SSDs and staged data in DRAM when it expected the next exchange to need it. That gave the drives a role in holding more data and DRAM a role in retrieving data quickly. The Seoul test put separate numbers on retrieving from remote SSD and CXL DRAM.

The CXL Consortium says memory allocated from a pool can behave differently from memory attached directly to the server. A remote cache saves time when it can be retrieved faster than the server can reconstruct the data. SK hynix presented its test results in Seoul. Its public materials gave no customer rollout dates or sales terms for the design.