VRAM Math: Sizing Your GPU Before You Buy
VRAM is the binding constraint, not FLOPS — a model either fits or it doesn't. The KV cache grows linearly with context and can exceed weight size.
Every local-inference sizing question reduces to three added terms (codingprotocols.com):
total VRAM = weights + KV cache + overhead
weights (GB) = parameters (B) × bytes-per-parameter
KV cache (GB) ≈ 2 × num_layers × num_kv_heads × head_dim × context_len × bytes_per_elem / 1e9
overhead ≈ 0.5–1.5 GB (CUDA context, activations, framework allocs)
An equivalent single-line form used in practice: weights = P × bits / 8 × 1.05 (the 1.05 covers GGUF metadata); a 70B at Q4 → 70 × 4 ÷ 8 × 1.05 ≈ 36.8 GB (flaviocopes.com).
| Precision | bits/param | bytes/param | 9B model needs |
|---|---|---|---|
| FP32 | 32 | 4.0 | ~36 GB |
| FP16 / BF16 | 16 | 2.0 | ~18 GB |
| Q8 / INT8 / FP8 | 8 | ~1.0 | ~9 GB |
| Q4_K_M (common default) | ~4.8 | ~0.60 | ~5.4 GB |
| Q3 | ~3.5 | ~0.44 | ~4 GB |
| Q2 | ~2.6 | ~0.33 | ~3 GB |
Source: codingprotocols.com; matching bytes/param table at dev.to/maxvyaznikov.
Use 0.6 GB/billion, not 0.5. K-quants keep some tensors at higher precision because degrading them hurts disproportionately, so the effective rate is always above nominal. Q4_K_M lands at 0.57–0.61 GB per billion in every real file: 7B → ~4.1 GB, 13B → ~7.9 GB, 70B → ~40 GB (codingprotocols.com). Corroborated: ~2 GB/B at FP16 and ~0.6 GB/B at INT4 (spheron.network).
Worked examples (codingprotocols.com):
| Setup | Weights | KV | Overhead | Total | Fits? |
|---|---|---|---|---|---|
| 8 GB card, 9B @ Q4, 4k ctx | 5.4 | ~0.5 | ~1.0 | 6.9 GB | ✅ tight |
| 8 GB card, 20B @ Q4 | 12.0 | — | — | >13 GB | ❌ |
| 24 GB card, 27B @ Q4, 16k ctx | 16.2 | ~2.0 | ~1.0 | 19.2 GB | ✅ sweet spot |
A shortcut when you don't want the arithmetic: the model's download size is a good proxy for weight memory; add a couple of GB.
Sources
- https://codingprotocols.com/blog/local-llm-vram-requirements-quantization — The weights + KV + overhead formula, bytes/param table, worked VRAM examples, offloading behaviour.
- https://flaviocopes.com/llm-vram-requirements — Single-line VRAM formula with the 1.05 metadata factor and a 70B worked example.
- https://localaimaster.com/blog/kv-cache-paged-attention-guide — KV cache math, per-token cost, GQA/MLA/CLA reduction table, PagedAttention, prefix caching, quantized KV.
- https://presenc.ai/research/local-llm-tokens-per-second-benchmarks-2026 — Measured 2026 tok/s for RTX 5090, DGX Spark, M5 Max/Ultra by model, plus prefill vs decode split.
- https://modelfit.io/gpu — GPU speed ladder (Qwen3 8B Q4_K @16K), VRAM, street prices, multi-card VRAM pooling.
- https://oobabooga.github.io/blog/posts/gptq-awq-exl2-llamacpp — Head-to-head GPTQ/AWQ/EXL2/Q4_K_M/NF4 perplexity, VRAM, prefill and tok/s on an RTX 3090.
- https://atomic.chat/blog/guides/exl3-vs-gguf — EXL3 vs GGUF Q4_K_M on size, KLD divergence and speed.
- https://willitrunai.com/blog/quantization-guide-gguf-explained — GGUF quantization guide: Q4_K_M saves ~72% VRAM, Q4 vs Q5 vs Q8 trade-offs.
- https://kaitchup.substack.com/p/choosing-a-gguf-model-k-quants-i — Choosing between K-quants, I-quants and legacy GGUF formats.
- https://arxiv.org/html/2601.14277v1 — Unified evaluation of llama.cpp quantization on Llama-3.1-8B-Instruct.
- https://turingpi.com/llm-inference-benchmarks-rk3588-gguf-quantization — GGUF quantization compared on an ARM RK3588 board (CPU/ARM inference reality).
- https://bmdpat.com/blog/gguf-quantization-q4-q5-q8-explained-2026 — Q4_K_M vs Q5_K_M vs Q8_0 selection criteria.
- https://huggingface.co/blog/4bit-transformers-bitsandbytes — bitsandbytes 4-bit / NF4 mechanics and use cases.
- https://ar5iv.labs.arxiv.org/html/2305.14314 — QLoRA paper: 4-bit NF4 + double quantization matching 16-bit fine-tuning.
- https://github.com/ggml-org/llama.cpp/discussions/23470 — Asymmetric q8/q4 KV cache quantization to control VRAM at long context.
- https://github.com/ggml-org/llama.cpp/discussions/21961 — Paged KV cache and scheduler design for llama.cpp.
- https://docs.vllm.ai/en/stable/serving/parallelism_scaling — Official vLLM tensor/pipeline/data/expert parallelism configuration.
- https://jarvislabs.ai/blog/scaling-llm-inference-dp-pp-tp — Practical walkthrough of DP vs PP vs TP trade-offs.
- https://www.glukhov.org/llm-hosting/comparisons/amd-rocm-vs-vulkan-llm-hosting — ROCm vs Vulkan decision matrix per engine; ROCm 10.0.0 and RDNA 4 support.
- https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1592861-amd-rocm-7-1-vs-radv-vulkan-for-llama-cpp-with-the-radeon-ai-pro-r9700 — ROCm 7.1 vs RADV Vulkan llama.cpp benchmarks on Radeon AI Pro R9700.
- https://www.compute-market.com/blog/best-cpu-for-local-llm-2026 — Memory bandwidth matters more than core count for CPU inference.
- https://dev.to/maximsaplin/ddr5-speed-and-llm-inference-3cdn — Measured effect of DDR5 speed on LLM inference throughput.
- https://www.corsair.com/us/en/explorer/diy-builder/how-tos/memory-for-local-llms-how-much-ram-do-you-need-and-when-speed-matters — How much RAM local LLMs need and when speed matters.
- https://www.promptquorum.com/local-llms/best-cpu-only-llm — CPU-only 2026 reality check: Phi-4-mini at ~12 tok/s.
- https://www.tomshardware.com/desktops/exploring-apple-silicons-local-ai-performance-with-the-mac-studio-and-m4-max-m4-max-beats-gb10-and-strix-halo-in-decode-throughput-but-memory-bandwidth-isnt-everything — M4 Max decode throughput vs GB10 and Strix Halo; bandwidth isn't everything.
- https://llmcheck.net/benchmarks — Apple Silicon LLM benchmarks across M1–M6 (figures to 258 tok/s).
- https://www.newegg.com/insider/best-gpus-for-ai-and-local-llms-in-2026-vram-is-king — 2026 GPU buying guide; VRAM as the binding constraint.
- https://www.compute-market.com/blog/best-local-llm-rtx-50-series-2026 — RTX 50-series VRAM, bandwidth, MSRP and best-paired model.
- https://modelfit.io/gpu/rtx-3090 — Used RTX 3090 24 GB value case: runs 32B at Q4, ~$900.
- https://intuitionlabs.ai/articles/local-llm-deployment-24gb-gpu-optimization — 24 GB GPU deployment and July 2026 used-market pricing.
- https://www.hostrunway.com/blog/best-gpu-for-running-local-llms-and-private-ai-in-2026-complete-buyers-guide-ollama-lm-studio-llama-cpp — Buyer's guide by tier; RTX 5060 Ti 16 GB pricing.
- https://fluence.ai/blog/best-gpu-for-llm — 2026 local LLM GPU shortlist and recommendation.
- https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html — vLLM MoE playbook: TP/DP/PP/expert parallelism in practice.
- https://docs.vllm.ai/en/v0.8.0/serving/distributed_serving.html — vLLM distributed/multi-GPU inference docs.
- https://www.spheron.network/blog/gpu-memory-requirements-llm — ~2 GB/B at FP16, ~0.6 GB/B at INT4 VRAM rules of thumb.
Date: 2026-10-09