KV cache: where a growing conversation takes up memory

Imagine loading a language model onto a graphics card, whose processor is usually called a GPU. It starts, answers your first question and leaves some memory available. Then several people send long documents at once. The model’s stored weights, the numerical settings learned during training, have not grown. The work in progress needs room of its own.

One important part of that working memory is the key-value cache, usually shortened to KV cache. For many transformer language models, it holds information calculated from earlier tokens so the model can reuse it while producing the next one. Keeping that information saves repeated calculation. Keeping more of it uses memory. [1]

That is why “the model fits” answers only part of a serving question. The same model can need different amounts of working memory as the length and number of requests change.

The next word has a history behind it

A model generates an answer in small units called tokens. A token may be a word, part of a word or punctuation. Before generating the answer, it processes the supplied text. As it continues writing, its attention mechanism uses information about earlier positions in that sequence.

Within that mechanism, keys and values are numerical representations calculated from tokens at the relevant layers of the model. A new token also produces a query, another numerical representation. Comparing that query with the earlier keys helps determine how strongly to combine the corresponding values. The values carry the information being combined. Those comparisons happen again for a new query; the earlier keys and values can be reused. [1]

Take a tiny invented sequence of three tokens, A, B and C. After processing those three positions, a layer has cached their three key-value pairs. When the model processes the next token, D, it calculates D’s query, key and value. D’s query can use the stored keys for A, B and C alongside the new key for D. The corresponding values contribute to the result. D’s key and value join the cache for the following step. Other attention layers keep their own state. [1]

Why can the old entries stay? In this causal model, earlier positions cannot look ahead at tokens that have not arrived. Adding D therefore does not require changing A’s stored state. The next query still brings new calculation, but the earlier pairs are ready to use. This is reuse within generation, not a saved answer to an earlier question. [1]

Think of a person keeping useful notes beside a document. The comparison helps explain reuse, but the cache is not a written summary. It contains numerical state used by the model, not human-readable conclusions about the conversation.

Count the space one conversation carries

For a conventional full-attention model, retained KV state grows with the sequence it must keep. Several active requests bring several sets of state. The exact cost also depends on the model’s architecture and the numerical format used to store the cache. [1]

We can count the stored numbers in an invented configuration: 32 full-attention layers, eight key-value heads per layer and 128 numbers per key or value in each head. A head is one of the parallel attention components; use the number of key-value heads, which need not equal the number of query heads. Grouped-query attention lets several query heads share key-value heads. [5] Suppose each stored number occupies two bytes.

For each retained token position, the key-value payload is:

2 (key and value) × 32 layers × 8 KV heads × 128 numbers × 2 bytes
= 131,072 bytes = 128 KiB

At 4,096 retained positions, that gives 512 MiB of KV payload for one sequence. At 8,192, it gives 1 GiB. Two independent 8,192-position sequences need 2 GiB under these assumptions. These are calculated payloads for a hypothetical configuration, not measurements of a named model. They exclude weights, other working memory, allocator overhead, reserved unused space and any sharing between requests.

The count follows the shape of the retained arrays, not the size of the model download. A deployment using fewer key-value heads or a different cache format changes the count. Spreading a model across devices changes where the bytes live. The calculation is useful for understanding the extra occupant of memory, not for promising a particular server’s capacity.

The distinction matters when someone says a server handled ten users in a trial. Were they asking short questions, reading long documents or generating long answers? A count of users hides the work each conversation brings along.

Some architectures change the picture. Sliding-window attention retains a limited recent span at the applicable layers, and models can combine different kinds of layers. Cache allocation strategies can also reserve space ahead of use. Read the model and runtime’s actual behavior before applying the simple sketch to a memory estimate. [2]

When the notes stop fitting

A serving engine has to decide what to do when it cannot keep all the needed state in available memory. vLLM documents preemption under KV-cache pressure: it can interrupt a request and later recompute state so the work can continue. That extra work can hurt latency, the time a person waits for an answer. [3]

Waiting requests can also build up before their processing begins. The distinction is visible in vLLM’s documented metrics: time waiting in the request queue, time to the first token and end-to-end request latency describe different parts of the experience. Check the metrics exposed by the installed version. [4]

Set a service’s request limits from the lengths and concurrency it must actually handle, rather than infer a user count from leftover memory after model loading. Include both long inputs and long outputs in the trial. A single successful answer establishes that one answer worked under those conditions.

If long documents are essential to the service, lowering the allowed input length would remove part of the product people need. That boundary changes the choice: the team may need more suitable capacity or a supported cache strategy instead. Hugging Face documents options including offloading, which moves cache state to CPU memory and adds transfers, and quantization, which stores cache numbers more compactly. Compatibility and response time need their own evaluation. [2]

Put the conversations into the capacity question

Return to the first successful answer. It showed that the model could load and generate a response. To understand what happens when more people arrive, add the work their conversations carry: input length, output length, simultaneous requests and the cache behavior of the chosen runtime.

In our invented configuration, letting a sequence grow from 4,096 to 8,192 positions added another 512 MiB of KV payload. The model’s weights stayed the same. Once you can see where that extra memory goes, a capacity statement can describe something a person will recognize: this mix of document sizes and answer lengths, at this level of demand, with these observed waiting times. The model still has to fit. So do the conversations around it.

Sources and scope

Documentation researched November 13, 2026. The sequence and configuration are hypothetical; the arithmetic counts an idealized KV payload, not a benchmark or complete GPU-sizing estimate. No model was run for this article.

1. Hugging Face: how caching works.
2. Hugging Face: cache strategies.
3. vLLM: optimization and tuning.
4. vLLM: production metrics.
5. Ainslie and colleagues: grouped-query attention.