The Local AI Stack: Guides

Eight guides, organized in learning-path order. Start at the top if you are new; jump to the section that matches your question if you are not.

The data these guides are built on: 50 community submissions from 11 contributors, across 3 hardware vendors and 8 backends โ€” last updated 2026-10-26.

Fundamentals

The building blocks: what the numbers mean, what is inside the file, and how the quants are made.

Quantization Basics

What are W4A16, FP8, Q4_K_M? The precision ladder, the VRAM formula, and which level to pick for your workload.

Beginner

File Formats

GGUF vs safetensors vs MLX. What is inside a model file, which one to download for your backend, and how they convert.

Beginner

Quantization Methods

RTN, GPTQ, AWQ, AutoRound, SmoothQuant, NVFP4/MXFP4, and the GGUF K-quant family. How quants are made and why one 4-bit beats another.

Intermediate

The Model & Its Memory

How the model is built, what eats your VRAM, and how to stretch the context window.

Model Architectures

Dense vs Mixture-of-Experts. What "A3B" means. Why a 125B model runs at 3B speed, and how MTP + speculative decoding make it faster.

Intermediate

KV Cache

The silent memory eater. Why context length costs VRAM, the formula that predicts it, and when to quantize the cache.

Intermediate

Context Extension

RoPE, YaRN, NTK. Pushing models beyond their native window โ€” and the honest 2ร—/4ร—/8ร— quality rule.

Intermediate

Running It

The engines and the hardware strategies that make it all work.

Backends Deep Dive

llama.cpp, llama-swap, Ollama, LM Studio, Unsloth Desktop, vLLM, TensorRT-LLM, Strata, Radiance, MLX, and Colibri. Every engine compared, with the exact command to start.

Advanced

Multi-GPU & Hybrid

Tensor parallel, pipeline parallel, layer offload, and the Strata hybrid (VRAM+RAM+SSD). Making big models fit on consumer hardware.

Advanced

Not sure where to start?