Application Stack
Agent frameworks, coding agents, RAG, fine-tuning, evaluation, speech and image generation.
| Page | Summary | Status |
|---|---|---|
| Overview | The 2026 local stack is coherent because OpenAI-compatible serving is universal: vLLM for serving, LlamaIndex/CrewAI for agents, Qwen3-Embedding+Qdrant for RAG, Unsloth for fine-tuning, whisper.cpp+Kokoro for speech, ComfyUI for images. | verified |
| Agent Frameworks | MCP is the standard 2026 tool bus (stateless spec since July). CrewAI most active for multi-agent. AutoGen slowing — verify before adopting. Local tool calls are a code-execution surface. | partial |
| Coding Agents | OpenCode is most active (212k stars). For local coding agents, a strong 30B+ model on vLLM is the minimum for usable function calling. Claude Code works via gateway config. | verified |
| RAG Pipeline | Recommended local pattern: Qwen3-Embedding -> Qdrant/FAISS -> Qwen3-Reranker -> LLM on vLLM, orchestrated by LlamaIndex. Qwen3-Embedding-8B ranks #1 on MTEB multilingual. Milvus PyPI package is stale — use Qdrant. | verified |
| Fine-Tuning | Unsloth is fastest for single-GPU LoRA/QLoRA (7-14B on 24GB comfortably, up to 70B via NF4). Axolotl for multi-GPU/production. MLX-LM on Mac. Export to GGUF afterwards. | verified |
| Evaluation | lm-evaluation-harness is the standard for reproducible open-model benchmarking. Evaluates local models directly via transformers or vLLM backend for fully-offline loops. | verified |
| Speech Stack | whisper.cpp for portable/CPU ASR; faster-whisper if you want GPU batched ASR. Kokoro (via kokoro-onnx) for quality neural TTS; Piper for lightweight embedded. | verified |
| Image Generation | ComfyUI is the de-facto local diffusion stack — node-graph GUI and API. FLUX-dev/schnell class models need 12GB+ consumer GPU. Don't confuse the comfyui PyPI package (v0.0.1, 2024) with the real product. | verified |
| Multi-Agent Orchestration | Topology: Qwen3 on vLLM -> LiteLLM proxy -> CrewAI crews -> MCP servers -> Qdrant + Qwen3-Embedding. Speech and image layers bolt on via whisper.cpp and ComfyUI on the same box. | verified |