KV Cache Management — Context Length Trade-Offs
The KV cache is the term people forget, and it is the reason a model that loaded fine OOMs twenty minutes into a long conversation.
The KV cache is the term people forget, and it is the reason a model that loaded fine OOMs twenty minutes into a long conversation. It grows linearly with context length and can rival the weights themselves (codingprotocols.com). Measured BF16 KV cache sizes (localaimaster.com):
| Context | Llama 3.1 8B | Llama 3.1 70B | Llama 3.1 405B |
|---|---|---|---|
| 4K | 0.5 GB | 1.3 GB | 3.0 GB |
| 32K | 4 GB | 10.7 GB | 24 GB |
| 128K | 16 GB | 43 GB | 96 GB |
| 200K | 25 GB | 67 GB | 150 GB |
For Llama 3.1 70B at 131K the KV cache alone is 43 GB per request — bigger than the model weights at FP8 (localaimaster.com). Per-token cost: 2 × 80 layers × 8 KV heads × 128 head_dim × 2 bytes = ~320 KB/token for Llama 3.1 70B with GQA; without GQA (full MHA) that is 8× larger, 86 GB at 32K — which is why GQA is effectively mandatory on modern models.
Practical consequence: raising OLLAMA_CONTEXT_LENGTH from 4096 to 32768 is not free. A 7B at Q4_K_M is only ~4.7 GB of weights, but a 32K window adds ~4 GB of KV (promptquorum.com).
Architectural KV reduction (localaimaster.com):
| Technique | Reduction | Quality loss | Examples |
|---|---|---|---|
| MHA (baseline) | 1× | 0% | GPT-2, original Llama |
| MQA | up to num_heads (e.g. 32×) | 1–2% | PaLM, Falcon |
| GQA | num_heads / num_kv_heads (~8×) | <0.5% | Llama 3, Qwen 2.5, Mistral |
| MLA (low-rank K/V) | 5–10× (~50 KB/token) | <0.5% | DeepSeek V2/V3 |
| CLA (cross-layer) | 2–3× | <0.5% | Hunyuan-Large |
DeepSeek V3's MLA compresses KV to ~50 KB/token (1.6 GB at 32K), which is what makes 671B at 128K context viable at all.
Options that exist in 2026: PagedAttention (vLLM/SGLang/TensorRT-LLM allocate KV in fixed-size pages instead of a contiguous block, killing fragmentation and allowing sharing), prefix/radix caching (reuse KV for shared system prompts), and quantized KV cache (FP8/INT8/INT4). llama.cpp now supports cache-type quantization such as asymmetric q8/q4 KV specifically to keep VRAM in check at long context (llama.cpp discussion #23470), and a paged KV cache + scheduler is in active design for the same engine (llama.cpp discussion #21961). Quality impact of KV quantization is small but measurable — test q8 before dropping to q4.
Sources
- https://codingprotocols.com/blog/local-llm-vram-requirements-quantization — The weights + KV + overhead formula, bytes/param table, worked VRAM examples, offloading behaviour.
- https://flaviocopes.com/llm-vram-requirements — Single-line VRAM formula with the 1.05 metadata factor and a 70B worked example.
- https://localaimaster.com/blog/kv-cache-paged-attention-guide — KV cache math, per-token cost, GQA/MLA/CLA reduction table, PagedAttention, prefix caching, quantized KV.
- https://presenc.ai/research/local-llm-tokens-per-second-benchmarks-2026 — Measured 2026 tok/s for RTX 5090, DGX Spark, M5 Max/Ultra by model, plus prefill vs decode split.
- https://modelfit.io/gpu — GPU speed ladder (Qwen3 8B Q4_K @16K), VRAM, street prices, multi-card VRAM pooling.
- https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp — Head-to-head GPTQ/AWQ/EXL2/Q4_K_M/NF4 perplexity, VRAM, prefill and tok/s on an RTX 3090.
- https://atomic.chat/blog/guides/exl3-vs-gguf — EXL3 vs GGUF Q4_K_M on size, KLD divergence and speed.
- https://willitrunai.com/blog/quantization-guide-gguf-explained — GGUF quantization guide: Q4_K_M saves ~72% VRAM, Q4 vs Q5 vs Q8 trade-offs.
- https://kaitchup.substack.com/p/choosing-a-gguf-model-k-quants-i — Choosing between K-quants, I-quants and legacy GGUF formats.
- https://arxiv.org/html/2601.14277v1 — Unified evaluation of llama.cpp quantization on Llama-3.1-8B-Instruct.
- https://turingpi.com/llm-inference-benchmarks-rk3588-gguf-quantization — GGUF quantization compared on an ARM RK3588 board (CPU/ARM inference reality).
- https://bmdpat.com/blog/gguf-quantization-q4-q5-q8-explained-2026 — Q4_K_M vs Q5_K_M vs Q8_0 selection criteria.
- https://huggingface.co/blog/4bit-transformers-bitsandbytes — bitsandbytes 4-bit / NF4 mechanics and use cases.
- https://ar5iv.labs.arxiv.org/html/2305.14314 — QLoRA paper: 4-bit NF4 + double quantization matching 16-bit fine-tuning.
- https://github.com/ggml-org/llama.cpp/discussions/23470 — Asymmetric q8/q4 KV cache quantization to control VRAM at long context.
- https://github.com/ggml-org/llama.cpp/discussions/21961 — Paged KV cache and scheduler design for llama.cpp.
- https://docs.vllm.ai/en/stable/serving/parallelism_scaling — Official vLLM tensor/pipeline/data/expert parallelism configuration.
- https://jarvislabs.ai/blog/scaling-llm-inference-dp-pp-tp — Practical walkthrough of DP vs PP vs TP trade-offs.
- https://www.glukhov.org/llm-hosting/comparisons/amd-rocm-vs-vulkan-llm-hosting — ROCm vs Vulkan decision matrix per engine; ROCm 10.0.0 and RDNA 4 support.
- https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1592861-amd-rocm-7-1-vs-radv-vulkan-for-llama-cpp-with-the-radeon-ai-pro-r9700 — ROCm 7.1 vs RADV Vulkan llama.cpp benchmarks on Radeon AI Pro R9700.
- https://www.compute-market.com/blog/best-cpu-for-local-llm-2026 — Memory bandwidth matters more than core count for CPU inference.
- https://dev.to/maximsaplin/ddr5-speed-and-llm-inference-3cdn — Measured effect of DDR5 speed on LLM inference throughput.
- https://www.corsair.com/us/en/explorer/diy-builder/how-tos/memory-for-local-llms-how-much-ram-do-you-need-and-when-speed-matters — How much RAM local LLMs need and when speed matters.
- https://www.promptquorum.com/local-llms/best-cpu-only-llm — CPU-only 2026 reality check: Phi-4-mini at ~12 tok/s.
- https://www.tomshardware.com/desktops/exploring-apple-silicons-local-ai-performance-with-the-mac-studio-and-m4-max-m4-max-beats-gb10-and-strix-halo-in-decode-throughput-but-memory-bandwidth-isnt-everything — M4 Max decode throughput vs GB10 and Strix Halo; bandwidth isn't everything.
- https://llmcheck.net/benchmarks — Apple Silicon LLM benchmarks across M1–M6 (figures to 258 tok/s).
- https://www.newegg.com/insider/best-gpus-for-ai-and-local-llms-in-2026-vram-is-king — 2026 GPU buying guide; VRAM as the binding constraint.
- https://www.compute-market.com/blog/best-local-llm-rtx-50-series-2026 — RTX 50-series VRAM, bandwidth, MSRP and best-paired model.
- https://modelfit.io/gpu/rtx-3090 — Used RTX 3090 24 GB value case: runs 32B at Q4, ~$900.
- https://intuitionlabs.ai/articles/local-llm-deployment-24gb-gpu-optimization — 24 GB GPU deployment and July 2026 used-market pricing.
- https://www.hostrunway.com/blog/best-gpu-for-running-local-llms-and-private-ai-in-2026-complete-buyers-guide-ollama-lm-studio-llama-cpp — Buyer's guide by tier; RTX 5060 Ti 16 GB pricing.
- https://fluence.ai/blog/best-gpu-for-llm — 2026 local LLM GPU shortlist and recommendation.
- https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html — vLLM MoE playbook: TP/DP/PP/expert parallelism in practice.
- https://docs.vllm.ai/en/v0.8.0/serving/distributed_serving.html — vLLM distributed/multi-GPU inference docs.
- https://www.spheron.network/blog/gpu-memory-requirements-llm — ~2 GB/B at FP16, ~0.6 GB/B at INT4 VRAM rules of thumb.
Date: 2026-10-09