Hardware

VRAM maths, buying guide, KV cache, quantisation, offloading, multi-GPU, Apple Silicon, AMD.

Page	Summary	Status
VRAM Math	total VRAM = weights + KV cache + overhead. Use 0.6 GB/B for Q4_K_M, not 0.5. KV cache can exceed weight size at long context.	verified
Buying Guide	VRAM is king. Used RTX 3090 24GB (~$900) is best value for 32T. RTX 5090 (32GB) for speed but can't hold 70B at Q4. Mac Studio for dense 70B single-user.	verified
KV Cache & Offloading	The KV cache grows linearly with context length and can rival model weights. GQA/MLA/CLA reduce it. At 128K context, Llama 3.1 70B needs ~43 GB of KV per request.	verified
Quantization Formats	Q4_K_M for portability (pay ~40% throughput), EXL3 for speed on NVIDIA, AWQ/GPTQ for vLLM stacks, FP8 for big models in limited VRAM, NF4 for training only.	verified
CPU & ARM Offloading	CPU is bandwidth-bound, not core-bound. Fast DDR5 beats more cores. Offloading splits the model across GPU+CPU but crosses PCIe per token — dramatically slower.	verified
Prefill vs Decode	Prompt processing (prefill) is compute-bound; token generation (decode) is memory-bandwidth-bound. RTX 5090: ~14,000 tok/s prefill vs 290-300 tok/s decode. For RAG/agent workloads prefill is the bottleneck.	verified
Apple Silicon	MLX is the native Apple Silicon path. M5 Neural Accelerators give up to 3.97x TTFT. Unified-memory sizing: budget ~2x the model requirement. M4 Max beats GB10/Strix Halo on dense models because it has more memory, not more bandwidth.	verified
Multi-GPU	TP splits layers across GPUs (best latency, needs fast NVLink). PP splits by layer (low comms, bubbles). DP replicates the full model (throughput only). Consumer: 2x RTX 5090 runs 70B at ~27 tok/s.	verified
AMD ROCm	ROCm 7.2.x is the first release with official RDNA 4 support and claims CUDA parity for inference. Vulkan remains the pragmatic default for llama.cpp.	verified