Practice & Ops
Drivers, deployment, monitoring, security, cost, regulation and how to follow the field.
| Page | Summary | Status |
|---|---|---|
| Learning Resources | Practical starting stack: Ollama for zero-to-running, llama.cpp for efficiency, vLLM for production. r/LocalLLaMA, Simon Willison's blog, Hugging Face courses, Latent Space podcast. | verified |
| Licensing | Open-source vs open-weight: Apache-2.0/MIT are genuinely OSI-open. Llama 4 has a 700M MAU cap. Gemma 3 carries Google-enforceable restrictions. Downloading locally does not exempt you from usage terms. | partial |
| EU AI Act | Digital Omnibus (Aug 2026) pushed high-risk rules to Dec 2027/Aug 2028. Article 50 transparency applies Aug 2026. The open-source exemption does NOT cover systemic-risk models (>10^25 FLOP). Fine-tuning + EU market placement can make you a provider. | verified |
| Security | Local is private by default, not automatically secure. Three leak vectors: tool telemetry, malicious model files, network exposure. SafeTensors eliminates the pickle RCE vector. trust_remote_code=True is the second vector. | verified |
| Driver Setup | Breaking change as of CUDA 13.4 Linux/13.1 Windows: the NVIDIA driver is no longer bundled with the CUDA Toolkit. ROCm 7.2.x is first with official RDNA 4 support. Vulkan is the cross-vendor llama.cpp fallback. | verified |
| Mac Deployment | MLX is Apple's native framework. M5 Neural Accelerators (macOS 26.2+) give up to 3.97x TTFT via Metal 4 TensorOps. Requires macOS 26.2+ for M5 NPU access. | verified |
| Model Management | Quantized 7-8B GGUF is ~4-5 GB. Maintain a manifest with SHA-256 digests per model version. Ollama and LM Studio each maintain separate stores — plan for duplication. | verified |
| Monitoring | OpenTelemetry GenAI semantic conventions are the 2026 standard. vLLM exposes Prometheus metrics at /metrics. Aspire Dashboard via Docker for zero-infrastructure local backend. gen_ai content attributes off by default (sensitive data). | verified |
| Cost vs Cloud | Breakeven depends entirely on what you compare. Local wins for frontier-class models at moderate-high utilisation. Cloud wins for sporadic use (<10% utilisation, 2-4 year payback). Local removes GDPR Article 28 obligations. | verified |
| Following the Field | r/LocalLLaMA, Hugging Face Daily Papers, Simon Willison's blog, Latent Space podcast, The Batch newsletter, AA-AgentPerf-Local benchmark, llama.cpp release feed. | verified |