Raphael Friedmann
← The log

Quantisation lives in the scales

Three imp fixes that changed no weight bit: a block scale that rounded to zero, an activation scale the checkpoint already shipped, and one FP8 scale shared by K and V. Qwen3-30B-A3B perplexity 11.3565 to 10.9566.

A 4-bit number on its own can only be 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6. Everything else a quantised model does with it comes from the scale it is multiplied by. NVFP4 stacks two: an FP8 E4M3 micro-scale per 16 values, and one FP32 scale per tensor.

Weights are quantised once, by whoever made the checkpoint. Activations and the KV cache are quantised by the engine, at runtime, for every token. That is where an engine can get it wrong without a single weight bit changing. Between 2 and 3 November 2026, imp fixed three such cases. This post goes through them, and through how imp now checks itself against vLLM.

The format’s edges#

All three bugs sit at the low end of E4M3, the 8-bit float the scales are stored in.

E4M3 value
largest 448
smallest normal 2^-6 = 0.015625
smallest subnormal 2^-9 ≈ 0.00195
subnormal step 2^-9, so 3 bits of precision at best, fewer near the bottom

Below 2^-6 a value loses precision. Below 2^-9 it is gone. An NVFP4 micro-scale is the block’s largest magnitude divided by 6, divided by the tensor scale. So a 16-value block whose largest entry is under 6 × 2^-6 = 0.09375 gets a subnormal scale, and a block under 6 × 2^-9 has nothing left to represent it with.

Fix 1: a block that became zero#

#3019. imp has several NVFP4 activation quantizers: one for prompt tiles, one fused into the MoE kernels, one for small batches, one in the decode epilogue. For a block scale under 2^-9, some clamped it to 2^-9. Others wrote byte 0, and a scale of 0 reads the whole block as zero.

An illustration with a tensor scale of 1: a block whose largest entry is 0.003 needs a micro-scale of 0.003 / 6 = 0.0005, under 2^-9 ≈ 0.00195.

entry byte 0 clamped to 2^-9
0.003 0 1.5 × 2^-9 ≈ 0.00293
0.001 0 0.5 × 2^-9 ≈ 0.00098
0.0002 0 0

The clamped block is coarse but keeps its shape. The zeroed one is gone.

The symptom was strange: perplexity depended on the prefill chunk size. How many rows are processed together decides which quantizer a row goes through, so the same text through the same weights scored differently.

Qwen3-30B-A3B, deterministic PPL chunk 0 chunk 32 spread
before 11.3565 11.0110 0.3455
every quantizer clamps to 2^-9 11.0160 11.0190 0.0030

Normal-range bytes did not change: same expression, same result. The new unit test feeds activations small enough to hit the floor. On the old code, 9216 of 9216 scale bytes came out wrong.

The fix also moved imp’s own reference number. The pinned perplexity of Qwen3-8B, which every pull request is gated against, went from 11.2587 to 11.2352, on purpose, in the same commit. The 11.2587 itself came from fix 2, which merged first.

Fix 2: the scale the checkpoint already had#

#3015. imp quantised activations with a tensor scale of 1.0, so each micro-scale was simply block amax / 6. On Gemma-4, the inputs to the down projection are small. 27.3 % of the shared-MLP blocks and 28.0 % of the expert blocks had an amax under 0.09375, so their scales were subnormal, and in the worst layer 96.6 % were.

The checkpoints carry the fix. When ModelOpt or compressed-tensors quantise a model, they run calibration text through it and store a static input scale for every linear layer: the observed amax divided by 6 × 448 = 2688, so the largest expected activation maps to the largest value NVFP4 can hold. imp read none of it: 11725 linear layers on Gemma-4, ignored at load. Now every quantizer divides by that scale and every GEMM multiplies it back into its output.

Model PPL before PPL after
Qwen3-30B-A3B (ModelOpt) 11.3565 10.9678
Gemma-4-26B-A4B, chat-turn corpus 18.2854 17.8489
Qwen3-8B (compressed-tensors) 11.2949 11.2587
Qwen3-14B (ModelOpt) 9.8343 9.8152

Decode and prefill speed stayed within the pair spread (Gemma-4 tg128 283.59-284.55 before, 282.42-284.97 after). A scale of 1.0 reproduces the old result to the last digit, so the switch gemm.nvfp4_static_act_scale can be turned off for a clean comparison.

The static scale has a second benefit: it is per layer, not per batch. A row’s quantised value does not depend on which other rows happen to share its step, which is what imp’s determinism rules need.

With both fixes, Qwen3-30B-A3B lands at 10.9566 / 10.9436 (chunk 0 / chunk 32).

Fix 3: K and V sharing one FP8 scale#

The FP8 KV cache stores keys and values in E4M3 with a scale per layer. The old rule, still the default and called legacy, took the first real prefill’s absmax / 448 with a floor of 1.0. Across Qwen3 0.6B, 1.7B and 8B, 91 of 92 layers sit on that floor: keys and values go into E4M3 unscaled.

That works for Qwen3. It does not for Gemma-4, whose keys are small: amax about 1.6, against about 16 for the values. With a scale of 1, 17.98 % of Gemma-4’s K entries were subnormal or flushed to zero.

#3016 adds static scales, from the checkpoint or from one calibration prefill at startup: pow2ceil(amax × headroom / 448). Powers of two matter here: dividing by a power of two only shifts the exponent, so a value in the normal range rounds exactly as it did with scale 1, and the inverse is exact in FP16. #3023 then gives K and V separate scales, because one shared max(K, V) scale is set by the values and still leaves the keys near the bottom.

Gemma-4-26B-A4B: K entries subnormal or flushed in FP8
legacy (scale 1)17.98 %calibrated, shared K/V2.35 %calibrated, split K/V0.16 %
Track = 20 %. Headroom 2 for both calibrated arms. Source: the edge counts calibrate logs from an FP16 exponent histogram of the calibration prefill (#3023).
Gemma-4-26B-A4B: perplexity above the FP16 KV cache
legacy (scale 1)+1.92 %calibrated, shared K/V+2.98 %calibrated, split K/V+0.69 %
Track = 3.5 %. Chat-turn corpus, FP16 KV 17.5635, legacy 17.9006, shared 18.0870, split 17.6854.

The two charts do not tell the same story, and that is the honest part. The shared scale removed most of the subnormal keys and still scored worse than legacy. Fewer subnormals alone is not the goal. Per position the differences are small: mean absolute NLL change against FP16 of 0.4692 (legacy), 0.4739 (shared) and 0.4659 (split), and split with headroom 1 or 4 reads 0.4757 and 0.4789. The gain of split sits inside the spread of the headroom choice. That is why both new modes ship off by default, and why Qwen3, where no arm wins, keeps legacy.

Decode speed is not the issue: Gemma-4 FP8 KV tg128 299.13 legacy, 299.25 split.

How to know you got it right#

A perplexity going down is not proof. It could be lower than the model really is. So #3021 added a parity harness: the same token ids, the same rows, scored by imp and by vLLM 0.31.0.

Model imp vLLM (pooled over starts) Δ vLLM start-to-start spread
Qwen3-8B NVFP4 11.2178 11.1951 +0.203 % 0.571 %
Qwen3-8B FP8 (16-bit KV) 10.7604 10.7587 +0.016 % 0.000 %
Qwen3-30B-A3B NVFP4 (FP8 KV) 10.9265 10.9528 -0.240 % 0.328 %
Gemma-4-26B-A4B NVFP4 17.4793 17.7908 -1.751 % 1.917 %

Two things stand out. vLLM’s NVFP4 perplexity moves by 0.33 to 1.92 % between process starts. imp is byte-reproducible: repeated runs are equal. And on Gemma-4 imp now scores lower than vLLM. That gap appeared with the clamp of fix 1; before it the difference was +0.059 %. It is inside vLLM’s own start-to-start spread and not filed as a defect, but lower than the reference is still a deviation, and it is written down as one.

Honest limits#

  • Not bit-equal across chunks on MoE. With the static scale, some MoE blocks still fall under 2^-9. Gemma-4 gets 1.5 % worse at chunk 32 and 1.8 % better at chunk 0 after the clamp.
  • The checkpoint has to have it. Fix 2 reads the calibration of whoever exported the model. Every local ModelOpt FP8-KV export ships K and V scales of exactly 1.0, so checkpoint mode equals legacy there.
  • Calibration costs a feature. calibrate skips restoring the persisted prefix cache.
  • Perplexity is one metric. The degeneration suite (37 to 52 checks per model) passed with 0 failures on #3016, #3019 and #3023, but none of it is a task benchmark.

None of the three fixes touched a weight. They changed the number that sits next to sixteen of them.