Quantization Basics: What a Quant Actually Does

The 30-second version: Quantization stores a model's weights with fewer bits โ€” 16 bits per weight becomes 8, 4, or even 2.5. A 27B model drops from ~52 GB in FP16 to ~15 GB at Q4_K_M โ€” a 3.5ร— reduction with a small, usually acceptable quality loss. The working rules: Q4_K_M is the community workhorse, FP8 is the sweet spot when your hardware supports it natively, and anything below Q3 is a fit-only choice you should verify on your own task. And the model you pick matters more than the quant you pick.

The one-idea version

A language model is, at the hardware level, a very large pile of numbers โ€” its weights โ€” that the GPU reads over and over for every token it generates. Those numbers are stored at some precision: 16 bits, 8 bits, 4 bits. Quantization is simply the act of storing them with fewer bits. It is the same trick JPEG uses on a photograph: throw away the detail the eye is least likely to notice, keep the shape.

The catch is that a model's "shape" is its behavior. Weights that carry important structure deserve more precision; weights that mostly add flavor can be compressed harder. Every quantization scheme is a different answer to the question which bits matter? โ€” and that answer is why two 4-bit files of the same model can behave differently. The deep dive into those schemes lives in Quantization Methods; here we cover what the numbers mean and how they hit your VRAM.

Why we quantize

Two reasons, in order of importance:

  1. Fit. VRAM is the bottleneck of local inference. A 70B model in FP16 needs ~130 GB โ€” no consumer card has that. At 4-bit it needs ~40 GB, and at 8-bit ~65 GB. Quantization is the difference between "impossible" and "a couple of cards."
  2. Speed. Generating a token is mostly reading weights from memory, so fewer bytes per weight means more tokens per second. The Matrix data shows the effect is not theoretical: s1ugh34d's Qwen3.8-27B at UD-Q2_K_XL decodes at 13.9 t/s on a 16GB Radeon VII, while MadGoat's FP8 build of the same model family hits 120 t/s on 2ร— R9700 โ€” precision changes bandwidth and compute.
FP16
2 bytes/weight
โ†’
FP8 / INT8
1 byte/weight
โ†’
Q4 / INT4
0.5 byte/weight
โ†’
Q2 / I-Nano
~0.3 byte/weight

Each step halves (or better) the memory a model needs. The quality cost is not even โ€” see the ladder below.

The naming code

The same idea wears three different costumes depending on the ecosystem. Learning to read the label saves you from a lot of mis-downloads:

Label How to read it Where you'll see it
W4A16 Weights at 4-bit, Activations at 16-bit GPTQ, AWQ, AutoRound (the vLLM/PyTorch world)
W8A8 8-bit weights and 8-bit activations SmoothQuant
FP8 / BF16 8-bit / 16-bit floating point Native training and serving checkpoints
Q4_K_M GGUF "k-quant", 4-bit, medium precision mix llama.cpp, Ollama, LM Studio, Strata
IQ3_S, I-Nano imatrix 3-bit / ~2.5 bits per weight GGUF
2.5bpw 2.5 bits per weight โ€” the raw number that matters Anywhere

There are two big families: floating point (FP16, FP8) and integer (INT8, INT4, and GGUF's Q/I variants). The W4A16-style names tell you what was quantized (weights vs activations); the GGUF names tell you how the low-bit numbers were packed. Both describe the same underlying trade: fewer bits per weight, a little quality left on the table.

The precision ladder

This is the memory cost of each rung, shown as a share of the FP16 footprint:

FP16 / BF16
FP8 / Q8_0
Q5_K_M
Q4_K_M
IQ3_S
Q2_K
I-Nano

And the honest quality read for each:

Level Bytes/weight 27B model needs Quality (honest read)
FP16 / BF16 2.0 ~52 GB The reference. Rarely needed for inference; the training standard.
FP8 1.0 ~26 GB Near-lossless on modern models. The 2026 sweet spot where hardware is native.
Q8_0 ~1.0 ~26 GB Indistinguishable for most tasks. The "safe" GGUF pick.
Q5_K_M ~0.69 ~18 GB Small degradation. The quality pick for 24โ€“48 GB cards.
Q4_K_M ~0.57 ~15 GB The workhorse. Measurable loss on hard reasoning; best size/quality for most people.
IQ3_S ~0.42 ~11 GB Noticeable on reasoning-heavy tasks; fine for chat and summarization.
Q2_K ~0.35 ~9.6 GB Fit-only. Verify on your own task before trusting it.
I-Nano ~0.31 (2.5bpw) ~8.7 GB The extreme. This is where MoE models shine โ€” see below.

The quality-vs-size curve is not linear. From FP16 down to Q8 it is nearly flat; Q8โ†’Q4 is a small dip; Q4โ†’Q3 is steeper; and below Q3 it depends heavily on the model and the task. Draw the curve and you get a long, flat shoulder and then a cliff โ€” which is exactly why Q4_K_M sits where it does: just past the edge of the shoulder, at a third of the memory.

The fine print: group sizes, symmetry, and QAT

The VRAM formula

The whole discipline reduces to one line:

VRAM (GiB) โ‰ˆ params ร— bytes_per_weight รท 1024ยณ

Add the KV cache and engine overhead on top (the KV Cache guide covers that number), and you can estimate any model before downloading it. Here is the formula checked against real Matrix submissions:

Model (from the Matrix) Quant Formula (weights) What the Matrix actually observed
Qwen3.5-4B Q4_K_M ~2.3 GiB 5.3 GB at 65k context (s1ugh34d)
DeepSeek-R1-0528-Qwen3-8B Q4_K_M ~4.5 GiB 7.4 GB (s1ugh34d)
Qwen3.8-27B UD-Q2_K_XL ~9.7 GiB 13.8 GB on a 16GB Radeon VII (s1ugh34d)
Qwen3.8-27B FP8 ~26 GiB Runs on 2ร— R9700, 48 GB total (MadGoat)
Qwen3.6-MoE-35B-A3B I-Nano (2.5bpw) ~10.2 GiB 12.7 GB incl. KV, 54.4 t/s (s1ugh34d)
Qwen3.8-Flash-Next (125B MoE) IQ3_S ~49 GiB total 13.4 GB in VRAM, rest in RAM/SSD (MadGoat, Strata)

The gap between formula and observation is KV cache plus overhead โ€” budget 10โ€“30% extra on top of the weight math, and more if you run long context. Notice the last row: the formula says ~49 GB, and the submission says 13.4 GB in VRAM. That is not a math error โ€” it is a hybrid engine keeping only the hot experts in VRAM. The formula tells you the model's total memory need; the engine decides where that memory lives.

VRAM calculator (weights only)

Model size FP16 FP8 Q4_K_M Q2_K / I-Nano
4B 7.5 GB 3.7 GB 2.3 GB 1.3 GB
8B 14.9 GB 7.5 GB 4.5 GB 2.6 GB
27B 52 GB 26 GB 15.3 GB 9.6 GB
35B (MoE) 65 GB 33 GB 19.9 GB 10.2 GB (I-Nano)
70B 130 GB 65 GB 39.7 GB 24 GB
125B (MoE) 233 GB 116 GB 65 GB ~49 GB (IQ3_S)

Read the MoE rows carefully: total parameter counts are huge, so FP16 is fantasy โ€” but at 2.5โ€“3 bits per weight a 125B MoE becomes a ~50 GB object, and a hybrid engine shrinks the VRAM slice of that to a single 24GB card. That is the entire reason the Matrix's headline 125B submission exists.

Which quant should I pick?

Walk the ladder from your workload, not from the file size:

Agentic coding / reasoning โ†’ Q4_K_M floor; Q5_K_M or FP8 if VRAM allows. Keep KV unquantized.
โ†“
Chat / writing / Q&A โ†’ Q4_K_M; IQ3_S is acceptable when VRAM is tight.
โ†“
Long-context exploration โ†’ drop one level (IQ3_S) to buy KV headroom; quantize the KV (Q8_0 / int8).
โ†“
16 GB or less, maximum fit โ†’ prefer MoE at I-Nano/Q2 over dense: the 35B-A3B at 54 t/s beats a dense 27B at 14 t/s on the same card.

Four concrete recommendations, tied to what the community is actually running:

For agentic coding and reasoning

Q4_K_M is the floor; go higher if you can. These workloads stress exactly the capabilities low-bit quantization degrades first โ€” precise recall, multi-step logic. If your card holds Q5_K_M (a 27B at ~18 GB fits a 24GB card with room for context), take it. If you have native-FP8 hardware, FP8 is near-lossless and the best quality per byte you can get. And keep the KV cache in FP16 here โ€” quantizing it saves memory you do not have to spend on a coding agent. (Details in the KV Cache guide.)

For chat, writing, and document Q&A

Q4_K_M. It is the reason the format won the consumer world: ~90%+ of FP16 behavior on general tasks at a third of the memory. The Matrix's 4B and 8B Q4_K_M submissions (5.3 GB and 7.4 GB) are the archetype โ€” small, fast, always-loaded.

For long-context work

Trade one quant level for KV headroom. A 3-bit model with 64kโ€“128k of context usually serves you better than a 4-bit model that runs out of memory at 16k. The s1ugh34d submissions show the pattern: quantized KV keeps VRAM nearly flat from 4k to 65k tokens, which is what makes the big contexts affordable.

For small cards (16GB and under)

Pick MoE, then go aggressive on bits. A 35B-A3B at I-Nano (2.5bpw) fit in 12.7 GB including KV on a 16GB card and decoded at 54.4 t/s โ€” because only ~3B of its parameters are active per token, the low-bit damage matters less. Dense models on 16GB are a speed cliff (see the 27B Q2_K_XL at 13.9 t/s in the same data). If you must run dense, use Q3 and verify it on your actual task.

Common misconceptions

  1. "Q4 is 75% of the model." No. The degradation curve has a long flat shoulder: Q4_K_M retains most FP16 behavior on general tasks; what you lose is the last 5โ€“10% of capability, concentrated in hard reasoning.
  2. "More quantization is strictly worse." A Q4 27B beats a FP16 7B on most tasks. Size beats precision up to a point โ€” the community's favorite trade.
  3. "The quant is the model." Base-model quality dominates; the quant is a delivery format. A well-quantized new model beats a poorly-quantized old one, every time.
  4. "FP8 is a datacenter thing." Not in 2026. Consumer Blackwell and RDNA4 hardware do FP8 natively, and the Matrix shows 27B FP8 serving at 120 t/s on two R9700s.
  5. "The KV cache is part of the weights." It is not. Weights are loaded once; the KV cache grows with every token of context. Quantizing weights and quantizing KV are separate decisions with separate tradeoffs.

How this fits the Matrix

Every row in the Matrix carries a quant, and now you can read it: the format family (GGUF vs FP8 vs โ€ฆ), the bit level, and the memory the formula predicts. Use the calculator above to sanity-check a submission's numbers before you trust its speed claims โ€” and when your own setup is ready, name both the format and the level in your submission.