Multi-Head Latent Attention (MLA) vs Grouped-Query Attention (GQA)
Low-rank latent vector compression versus grouped key-value projection heads
Multi-Head Latent Attention (MLA), introduced in DeepSeek-V2 and DeepSeek-V3, compresses key and value states into a unified low-rank latent vector (d_c = 512) paired with decoupled rotary position embeddings (d_pe = 64). Grouped-Query Attention (GQA) shares KV heads across query head groups (e.g., 8 KV heads for 128 query heads). MLA achieves up to a 93% KV cache memory reduction relative to standard MHA, outperforming GQA by 4x to 7x in memory efficiency while retaining full multi-head expressive retrieval capacity.
Technical Tradeoff & Benchmark Matrix
| Attention Architecture | Multi-Head Latent Attention (MLA) | Grouped-Query Attention (GQA) |
|---|---|---|
| KV Cache Compression Mechanism | Low-rank projection to compressed latent vector (c_t^KV) + decoupled RoPE | Subsampled key-value projection heads shared across query head groups |
| KV Cache Size per Token (60 Layers) | Approx 69 KB per token (d_c=512, d_pe=64 in FP8) | Approx 245 KB per token (8 KV heads, d_h=128 in FP8) |
| Relative KV Compression vs MHA | 14x to 16x reduction in KV cache footprint | 4x to 8x reduction in KV cache footprint |
| Matrix Absorption During Inference | Key/Value projection matrices absorbed into query and output projections | Standard batched matrix multiplication over uncompressed projected vectors |
| Expressive Retrieval Capacity | Full rank multi-head expressive capacity preserved via up-projection | Information loss proportional to the query-to-KV head grouping ratio |
| Hardware Execution Efficiency | Higher arithmetic intensity; minimizes memory bandwidth bottle-necks | High kernel compatibility across standard FlashAttention implementations |
Interactive Calculation Engines
Calculate empirical values directly using our zero-latency edge computational models:
KV Cache Architecture & Latent Compression Sizer
Models KV cache memory savings across context windows up to 128k tokens for MHA, GQA, and MLA architectures.
Sparse Mixture of Experts (MoE) VRAM & Bandwidth Sizer
Calculates memory footprint and memory-bandwidth scaling for sparse Mixture of Experts (MoE) LLMs across H100/H200 SXM5 clusters.
Frequently Asked Questions & Technical Analysis
Q: How does DeepSeek MLA eliminate the traditional KV cache bottleneck during decoding?
In autoregressive generation, memory bandwidth to read the KV cache from HBM dominates generation latency. By compressing keys and values into a 512-dimensional latent vector and only storing that latent state in VRAM, MLA slashes required memory transfers by more than 90%, enabling massive batch sizes and long context windows on fewer GPUs.
Q: Can matrix absorption be used during the prefill phase in MLA?
No. Matrix absorption is mathematically valid only during single-token autoregressive decoding where the latent vector is multiplied directly by the combined projection weight tensor. During multi-token prefill, standard low-rank projections must be evaluated to allow full sequence attention computation.