The key-value cache is a buffer that stores precomputed key and value tensors from transformer attention layers during autoregressive text generation. When a model generates tokens sequentially, it reuses these cached values instead of recomputing attention over all previous tokens. The cache grows linearly with sequence length and is accessed by position index. It persists through software implementation in inference engines like vLLM, TGI, and Transformers. [formal: memoria chaverum | substrate: matter | horizon: hours | explicit: yes | epoch: 0.01]
Accepted ontology entry
kv-cache
The key-value cache is a buffer that stores precomputed key and value tensors from transformer attention layers during autoregressive text generation. When a model generates tokens sequentially, it reuses these cached values instead of rec…
Definition
Why it is in scope
The key-value cache in transformer models — precomputed key and value vectors from earlier tokens, stored to avoid recomputation during autoregressive generation
Names and aliases
- kv-cacheen · CANONICAL
Relations from this entry
- cmsn38oug02uu1q13kmcfzvv6DEPENDS_ON →
Removal test: remove transformer architecture and kv-cache has nothing to operate on — it stops functioning entirely. kv-cache is a mechanism that exists only within the transformer inference loop.
- cmreq01yl00ydg8vu72b2ovahSERVES →
kv-cache is built and maintained for the sake of inference: it caches key-value pairs during transformer inference to avoid redundant computation, directly serving the goal of efficient inference. Its entire design purpose is to accelerate the forward pass — the servant (kv-cache) points at the master (inference).
- cmsc62wgo02ha3vv321fexvluINSTANCE_OF →
kv-cache IS a specific kind of data structure: it stores key-value pairs for efficient lookup during autoregressive generation. A competent speaker would call a kv-cache 'a data structure.' The nearest kind is data structure, which is already ACCEPTED.
Relations to this entry
No accepted relations in this direction.
Record identity
- Created
- Aug 10, 2026, 4:15 PM UTC
- Content hash
- 6d1c4641a463bff2e195cc5cce3c0d7a9a5a81d9e30302bb14c23d5862cd7089