Overview
Direct Answer
A memory optimisation technique in transformer-based models that caches previously computed key and value tensors during autoregressive generation, eliminating redundant recalculation as each new token is produced. This mechanism significantly reduces computational overhead during inference without altering model outputs.
How It Works
During token generation, the transformer computes queries for the current token whilst reusing cached key-value pairs from prior positions rather than recomputing them. The cache is sequentially extended as each new token is generated, allowing attention operations to access historical representations in constant rather than quadratic time relative to sequence length. Modern implementations store these tensors in GPU memory or system RAM, depending on batch size and model dimensions.
Why It Matters
Key-value caching reduces inference latency by 2–3× on typical sequence lengths, directly lowering operational costs for production language models and enabling real-time interactive applications. For resource-constrained environments and large-scale deployments, this optimisation determines practical feasibility of transformer inference at scale.
Common Applications
Used extensively in conversational AI systems, real-time code generation tools, and streaming text summarisation services. Dialogue systems relying on multi-turn interactions particularly benefit from avoiding reprocessing of prior conversation history.
Key Considerations
Cache memory consumption scales linearly with batch size and sequence length, creating practical limits on concurrency and maximum context window. Careful management is required to prevent memory exhaustion, and cache invalidation strategies vary across frameworks and hardware configurations.
Cross-References(2)
More in Deep Learning
Activation Function
Training & OptimisationA mathematical function applied to neural network outputs to introduce non-linearity, enabling the learning of complex patterns.
Knowledge Distillation
ArchitecturesA model compression technique where a smaller student model learns to mimic the behaviour of a larger teacher model.
Self-Attention
Training & OptimisationAn attention mechanism where each element in a sequence attends to all other elements to compute its representation.
Weight Decay
ArchitecturesA regularisation technique that penalises large model weights during training by adding a fraction of the weight magnitude to the loss function, preventing overfitting.
Weight Initialisation
ArchitecturesThe strategy for setting initial parameter values in a neural network before training begins.
Vanishing Gradient
ArchitecturesA problem in deep networks where gradients become extremely small during backpropagation, preventing earlier layers from learning.
Diffusion Model
Generative ModelsA generative model that learns to reverse a gradual noising process, generating high-quality samples from random noise.
Mixture of Experts
ArchitecturesAn architecture where different specialised sub-networks (experts) are selectively activated based on the input.