Model Architectures: Dense, MoE, and Why 125B Runs at 3B Speed

The 30-second version: A model's *architecture* decides how much of it the GPU touches for each token. **Dense** models use every parameter every step โ€” straightforward, but expensive. **Mixture-of-Experts (MoE)** models are huge in total (125B) but light per token (only a few B active), which is why a 125B model can decode at the speed of a 3B one. The "A3B" in a name means *3B active* out of the total. The catch: MoE needs more memory (all experts must be loaded) but less compute โ€” exactly why hybrid offload engines like Strata shine on it. And MTP + speculative decoding is how some of these get even faster without losing quality.

Dense: every parameter, every token

A dense model is the classic design: one continuous stack of layers, and all of its parameters participate in computing every single token. A 27B dense model reads all 27B of its weights to produce each character. There is nothing to route around, nothing to skip โ€” the compute cost is simply proportional to the total parameter count.

The Matrix is full of dense models, and they are the "predictable" end of the spectrum:

The rule for dense is easy to remember and hard to beat on small cards: speed scales with total size. A 27B dense model will always be slower than a 4B dense model on the same hardware, because it does 6.75ร— the work per token. That is the cost of dense intelligence โ€” and the reason the next section exists.

MoE: a model that only uses what it needs

A Mixture-of-Experts model replaces most of those dense feed-forward layers with a bank of smaller sub-networks called experts, plus a lightweight router. For each token, the router picks a small subset of experts (often 2 out of dozens or thousands) and only those run. The rest sit idle for that token โ€” and a different subset runs for the next one.

Token
"the cat"
โ†’
Router
picks 2 of 6,892
โ†’
Expert 1
active
Expert 2
active
Experts 3โ€“N
idle
โ†’
Combine
one output

Per token, the router lights up a handful of experts and leaves the rest dark. The model is enormous in total, but each character costs the compute of a small model.

The genius is in the asymmetry. Because experts are specialized, a model can have far more total parameters than a dense model of the same "active" size โ€” storing much more knowledge โ€” while paying the per-token compute of the small active set. You get the recall of a big model at the speed of a small one.

"A3B" decoded

You will see names like Qwen3.6-MoE-35B-A3B or gemma4:26b-a4b in the Matrix. The convention is [Total]B-A[Active]B:

The total number is what must be loaded into memory (every expert has to exist somewhere, even the idle ones). The active number is what your GPU actually computes for each token โ€” and it is the number that determines your decode speed. A 35B-A3B model feels like a 3B model at the keyboard, while drawing on 35B of stored knowledge.

One honesty note: active-parameter counts in community submissions are estimates that sometimes differ from the name. The Matrix's 35B-A3B submission is named "A3B" but its notes describe the compute as "โ‰ˆ 9B" (a rougher figure that likely bundles the shared expert in with the routed ones). The name is the convention to read; treat a submitter's prose number as a ballpark, not a spec.

Why MoE is revolutionary for local

This is the single most important idea for people running models on consumer hardware. Look at the ratio:

Total params
Active / token

A 125B-total MoE activates ~3B per token โ€” about 2.4% of the model does the work for each generated character. The other 97.6% is loaded but idle, waiting for the tokens that need it.

That ratio is what makes the Matrix's headline submission possible. Qwen3.8-Flash-Next is a 125B MoE model with 6,892 experts. On a single RTX 4090 (24GB) running the Strata hybrid engine, it keeps 13.4GB of the hot experts in VRAM and parks the rest in RAM and on NVMe โ€” and it still decodes at 99โ€“128 t/s (median 108). A dense 125B model would need ~230GB of VRAM and would not fit on any consumer box; a 125B MoE fits on one 24GB card because only a sliver is active at a time. That is "125B of intelligence at 3B of speed," and it is the reason MoE changed what is feasible at home.

The tradeoff: more memory, less compute

The flip side is the constraint that shapes every MoE deployment: all experts must be resident in memory, even the ones that are idle. So an MoE model needs more total memory than a dense model of the same active size, while needing less compute per token.

This is exactly why hybrid offload works so well for MoE (the Multi-GPU & Hybrid guide covers the mechanics): because the idle experts are not being read every token, they can live in slow RAM or on an NVMe SSD with little speed penalty. Only the active experts need fast VRAM. A dense model can't do this โ€” every one of its parameters is hot on every token, so offloading a dense model to RAM collapses its speed (see the 27B UD-Q2_K_XL at 13.9 t/s when it nearly fills a 16GB card). MoE turns "doesn't fit" into "fits, and stays fast."

MTP + speculative decoding

Some models in the Matrix get a second speed boost from MTP (Multi-Token Prediction) and the speculative decoding it enables.

MTP is a special head, trained alongside the model, that predicts several future tokens at once. On its own that is just a faster draft. The power comes from pairing it with speculative decoding:

Draft
MTP head proposes k tokens
โ†’
Verify
target checks all k in one pass
โ†’
Accept
keep the valid prefix
โ†บ

The small/draft side proposes a handful of tokens; the target model verifies all of them in a single parallel forward pass and keeps the longest correct prefix. Repeat. Several tokens per step, at no quality loss.

The key insight is that verification is parallel: the target model checks k candidate tokens in the same cost as it would one, because a transformer processes a whole sequence in a single pass. So you get k tokens for roughly the price of 1, and the quality is guaranteed by the target model being the one that accepts. No approximation, no degradation โ€” just more tokens per second.

The Matrix shows this in the wild: - MadGoat's Qwen3.8-27B FP8 on vLLM runs with adaptive MTP=8 โ€” a dense 27B model that speculatively drafts up to 8 tokens and verifies them, landing at 120 t/s. - PawingUnplug's list includes a 9B MTP model (Qwen3.5-9B-โ€ฆ-MTP-Q8_0). - Burhan's Ollama setup runs gemma4:26b-a4b-it-mtp โ€” a MoE and MTP model in one.

MTP is most valuable when your model has one (it is a training-time feature, not something you add later), and it compounds beautifully with MoE: a big-but-light model that also drafts ahead.

Which architecture should I pick?

Walk it from your constraint, not from the parameter count:

Maximum speed on a small card โ†’ a small dense (4B/8B) or a MoE at low bits. The 4B dense at 96 t/s and the 35B-A3B at 54 t/s are the archetypes.
โ†“
Maximum quality in limited VRAM โ†’ a large MoE. This is the 125B-at-3B-speed trick: big knowledge, small per-token cost, hybrid-offload friendly.
โ†“
Long context โ†’ pick a model with a large *native* window first (see [Context Extension](context-extension/)); architecture matters less than the trained limit here.
โ†“
Agentic / coding throughput โ†’ favor MTP models; speculative decoding is a free speed multiplier for high-volume work.

The honest summary: MoE wins on quality-per-VRAM โ€” it is how you get flagship-level intelligence onto consumer hardware. Dense wins on predictability โ€” its speed is a clean function of its size, with no routing to reason about. If your card is small and your ambition is big, MoE is the lever. If you want simple, always-fast, low-memory behavior, a small dense model is the calm choice.

How this fits the Matrix

The data splits into the two architectures in plain sight, and now you can read each row:

Dense โ€” Qwen3.8-27B (two runs: 120 t/s FP8 on 2ร— R9700, 13.9 t/s UD-Q2_K_XL on 16GB), Qwen3.5-4B (96.4 t/s), DeepSeek-R1-8B (84.4 t/s), and the 9B coder/MTP models. Speed tracks total size, exactly as the dense rule predicts.

MoE โ€” Qwen3.6-MoE-35B-A3B (I-Nano, 12.7GB, 54.4 t/s) and Qwen3.8-Flash-Next, the 125B model (13.4GB VRAM hybrid at 99โ€“128 t/s; also a Q4_K_M run on 2ร— w7900 at ~35 t/s). Note the inversion that is the whole point: the biggest model in the data (125B) decodes faster than the mid-size dense 27B, because only a few billion of its parameters are active per token.

MTP โ€” appears on the 27B (vLLM MTP=8), the 9B model, and gemma4:26b-a4b-it-mtp, the speculative-decoding boost layered on top of both architectures.

When you submit your own setup, name the architecture the way the data does โ€” total and active params for MoE (35B-A3B, 125B), plus an MTP note where present โ€” and the next reader can see at a glance why your speeds are the way they are. The memory side of the story (why context length eats VRAM regardless of architecture) is the KV Cache guide, and the engines that make all of this run are in Backends.