Inference Engines
llama.cpp, Ollama, vLLM, SGLang, TensorRT-LLM and more engine profiles: pick by workload.
| Page | Summary | Status |
|---|---|---|
| Overview | The 2026 local inference stack has bifurcated into GGUF/C++ single-user engines, batched GPU serving engines, and agent-oriented engines; engine choice is now gated by model architecture. | verified |
| Engine Choice | When you need one user, use Ollama; concurrency, use vLLM; Apple Silicon agents, use Spllash or vllm-metal. A decision table derived from real H100 benchmarks. | verified |
| llama.cpp | What llama.cpp is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| Ollama | What Ollama is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| vLLM | What vLLM is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| SGLang | What SGLang is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| TensorRT-LLM | What TensorRT-LLM is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| ExLlamaV3 | What ExLlamaV3 is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| MLC-LLM | What MLC-LLM is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| KoboldCpp | What KoboldCpp is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| LM Studio | What LM Studio is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| LocalAI | What LocalAI is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| Jan | What Jan is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| GPT4All | What GPT4All is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| llamafile | What llamafile is, its backends, quantisation, API, multi-GPU, and what it's best for. | ready |
| Latency, Throughput, Memory | The headline 44x engine gaps are a concurrency artefact. At batch 1, engines are near parity. Concurrency, not single-user speed, decides the engine. | verified |
| Emergent Engines | vllm.cpp, Magnitude, Splash, vllm-metal, MLX — the new agent-oriented engines built for concurrent coding-agent sessions on personal machines. | unverified |
| Serving (OpenAI-compatible) | The 2026 standard: every local engine speaks the OpenAI API shape. How to wire apps, caveats per runtime, and Anthropic Messages API compatibility. | ready |