The Local AI Stack: Guides
Eight guides, organized in learning-path order. Start at the top if you are new; jump to the section that matches your question if you are not.
The data these guides are built on: 50 community submissions from 11 contributors, across 3 hardware vendors and 8 backends โ last updated 2026-10-26.
Fundamentals
The building blocks: what the numbers mean, what is inside the file, and how the quants are made.
Quantization Basics
What are W4A16, FP8, Q4_K_M? The precision ladder, the VRAM formula, and which level to pick for your workload.
File Formats
GGUF vs safetensors vs MLX. What is inside a model file, which one to download for your backend, and how they convert.
Quantization Methods
RTN, GPTQ, AWQ, AutoRound, SmoothQuant, NVFP4/MXFP4, and the GGUF K-quant family. How quants are made and why one 4-bit beats another.
The Model & Its Memory
How the model is built, what eats your VRAM, and how to stretch the context window.
Model Architectures
Dense vs Mixture-of-Experts. What "A3B" means. Why a 125B model runs at 3B speed, and how MTP + speculative decoding make it faster.
KV Cache
The silent memory eater. Why context length costs VRAM, the formula that predicts it, and when to quantize the cache.
Context Extension
RoPE, YaRN, NTK. Pushing models beyond their native window โ and the honest 2ร/4ร/8ร quality rule.
Running It
The engines and the hardware strategies that make it all work.
Backends Deep Dive
llama.cpp, llama-swap, Ollama, LM Studio, Unsloth Desktop, vLLM, TensorRT-LLM, Strata, Radiance, MLX, and Colibri. Every engine compared, with the exact command to start.
Multi-GPU & Hybrid
Tensor parallel, pipeline parallel, layer offload, and the Strata hybrid (VRAM+RAM+SSD). Making big models fit on consumer hardware.
Not sure where to start?
- "I just want a model running." โ Quantization Basics to understand what you are downloading, then Backends to pick your engine.
- "My model is too big for my GPU." โ Multi-GPU & Hybrid for the split strategies, KV Cache for the memory math.
- "Why is my 125B model slow?" โ Model Architectures to understand MoE and active parameters.
- "What format do I download?" โ File Formats โ the backend picks the format, not the other way around.
- "I want to run longer contexts." โ Context Extension for RoPE scaling, KV Cache for the VRAM cost.
- "Which quant should I use?" โ Quantization Basics for the decision guide, Quantization Methods for why one 4-bit beats another.