Mixture of Experts: how a 30B model runs like a 3B one

A name like Qwen3-30B-A3B quietly tells you something clever is going on. 30 billion parameters in total, but only 3 billion “active” per token. That gap is Mixture of Experts, and it is a large part of why genuinely large models run on a single consumer card at all.

The idea: do not run the whole model

A normal (“dense”) model runs every one of its weights for every token. A Mixture-of-Experts model splits the bulk of itself into many parallel experts and, for each token, a small router picks just a few of them. The rest stay switched off.

Mixture of Experts: pick a few, skip the rest

For every token, a small router picks the top few experts out of many (Qwen3-30B-A3B has 128 experts and fires 8 per token). The model is huge in total but only a sliver runs per token, which is why it decodes like a much smaller one.

So a 30B model with 3B active does roughly the per-token compute of a 3B model, while keeping the knowledge capacity of a 30B one. Large capacity, light on its feet.

What the engine actually does

Inside imp, a token’s trip through an MoE layer is four steps:

Route. A tiny gating network scores all the experts and the top few win. Qwen3-30B-A3B fires 8 of its 128 experts per token; gpt-oss, 4 of 32; the exact split varies by model.
Sort. Tokens are bucketed by which experts they chose, so each expert receives a contiguous batch of work instead of scattered tokens.
Grouped GEMM. A single CUTLASS kernel runs every expert’s batch in one pass, straight on the 4-bit weights with no dequantization detour.
Combine. Each token’s chosen-expert outputs are weighted-summed back together, weighted by how strongly the router picked each one.

Why it is a boon on one GPU

Fewer active parameters means less compute per token, and since decode is memory-bound anyway, MoE models decode fast for their size. The measured decode speeds on a single 5090 make the point:

Model	Total	Active	Decode (tok/s)
Qwen3-Coder-30B-A3B	30B	3B	338
gpt-oss-20b	21B	3.6B	325
Qwen3-30B-A3B	30B	3B	305
Gemma-4-26B-A4B	26B	4B	266
Qwen3.6-35B-A3B	35B	3B	257

A 30B dense model simply wouldn’t fit in 32 GB. A 30B MoE, quantized to 4-bit, fits in about 16 GB and decodes north of 300 tokens a second.

Why it is also a headache

MoE is not free, it just moves the cost from compute to engineering.

It fights the decode fast path. imp normally records the per-token work once and replays it (a “CUDA graph”) to avoid overhead. But the router picks different experts for every token, so the path changes each step. imp keeps the graph for Gemma-4, whose routing it can do entirely on the GPU, and gives it up for the others, where the choice has to come back to the host first.

It is lumpy. If many tokens happen to pick the same expert, that expert gets a huge batch while others sit idle, so the GPU’s load is uneven in a way a dense model’s never is.

It is fiddly to make reproducible. The fast parallel bucketing can reorder tokens from run to run, which changes the order of floating-point sums and breaks bit-for-bit repeatability. imp carries a separate deterministic path for when that matters.

So 30B (3B active) is a genuine bargain, you get most of a large model’s quality at a small model’s speed, as long as you are willing to do the bookkeeping that makes it behave. The kernels post has the grouped-GEMM side of that story.