# Quantisation lives in the scales
> Three imp fixes that changed no weight bit: a block scale that rounded to zero, an activation scale the checkpoint already shipped, and one FP8 scale shared by K and V. Qwen3-30B-A3B perplexity 11.3565 to 10.9566.
Source: https://rfriedmann.de/blog/quantization-lives-in-the-scales/
Published: 2026-11-04 · Track: log · Level: Expert
A 4-bit number on its own can only be `0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6`.
Everything else a quantised model does with it comes from the scale it is
multiplied by. [NVFP4](/blog/nvfp4-at-the-bit-level/) stacks two: an FP8 E4M3
micro-scale per 16 values, and one FP32 scale per tensor.
Weights are quantised once, by whoever made the checkpoint. Activations and the
KV cache are quantised by the engine, at runtime, for every token. That is where
an engine can get it wrong without a single weight bit changing. Between 2 and
3 November 2026, imp fixed three such cases. This post goes through them, and
through how imp now checks itself against vLLM.
## The format's edges
All three bugs sit at the low end of E4M3, the 8-bit float the scales are stored in.
| E4M3 | value |
|---|---|
| largest | 448 |
| smallest normal | 2^-6 = 0.015625 |
| smallest subnormal | 2^-9 ≈ 0.00195 |
| subnormal step | 2^-9, so 3 bits of precision at best, fewer near the bottom |
Below 2^-6 a value loses precision. Below 2^-9 it is gone. An NVFP4 micro-scale
is the block's largest magnitude divided by 6, divided by the tensor scale. So a
16-value block whose largest entry is under 6 × 2^-6 = 0.09375 gets a subnormal
scale, and a block under 6 × 2^-9 has nothing left to represent it with.
## Fix 1: a block that became zero
[#3019](https://github.com/kekzl/imp/pull/3019). imp has several NVFP4 activation
quantizers: one for prompt tiles, one fused into the MoE kernels, one for small
batches, one in the decode epilogue. For a block scale under 2^-9, some clamped it
to 2^-9. Others wrote byte 0, and a scale of 0 reads the whole block as zero.
An illustration with a tensor scale of 1: a block whose largest entry is 0.003
needs a micro-scale of 0.003 / 6 = 0.0005, under 2^-9 ≈ 0.00195.
| entry | byte 0 | clamped to 2^-9 |
|---:|---:|---:|
| 0.003 | 0 | 1.5 × 2^-9 ≈ 0.00293 |
| 0.001 | 0 | 0.5 × 2^-9 ≈ 0.00098 |
| 0.0002 | 0 | 0 |
The clamped block is coarse but keeps its shape. The zeroed one is gone.
The symptom was strange: perplexity depended on the prefill chunk size. How many
rows are processed together decides which quantizer a row goes through, so the
same text through the same weights scored differently.
| Qwen3-30B-A3B, deterministic PPL | chunk 0 | chunk 32 | spread |
|---|---:|---:|---:|
| before | 11.3565 | 11.0110 | 0.3455 |
| every quantizer clamps to 2^-9 | 11.0160 | 11.0190 | **0.0030** |
Normal-range bytes did not change: same expression, same result. The new unit test
feeds activations small enough to hit the floor. On the old code, 9216 of 9216
scale bytes came out wrong.
The fix also moved imp's own reference number. The pinned perplexity of
Qwen3-8B, which every pull request is gated against, went from 11.2587 to
11.2352, on purpose, in the same commit. The 11.2587 itself came from fix 2,
which merged first.
## Fix 2: the scale the checkpoint already had
[#3015](https://github.com/kekzl/imp/pull/3015). imp quantised activations with a
tensor scale of 1.0, so each micro-scale was simply block amax / 6. On Gemma-4,
the inputs to the down projection are small. 27.3 % of the shared-MLP blocks and
28.0 % of the expert blocks had an amax under 0.09375, so their scales were
subnormal, and in the worst layer 96.6 % were.
The checkpoints carry the fix. When ModelOpt or compressed-tensors quantise a
model, they run calibration text through it and store a static input scale for
every linear layer: the observed amax divided by 6 × 448 = 2688, so the largest
expected activation maps to the largest value NVFP4 can hold. imp read none of it:
11725 linear layers on Gemma-4, ignored at load. Now every quantizer divides by
that scale and every GEMM multiplies it back into its output.
| Model | PPL before | PPL after |
|---|---:|---:|
| Qwen3-30B-A3B (ModelOpt) | 11.3565 | **10.9678** |
| Gemma-4-26B-A4B, chat-turn corpus | 18.2854 | **17.8489** |
| Qwen3-8B (compressed-tensors) | 11.2949 | 11.2587 |
| Qwen3-14B (ModelOpt) | 9.8343 | 9.8152 |
Decode and prefill speed stayed within the pair spread (Gemma-4 tg128
283.59-284.55 before, 282.42-284.97 after). A scale of 1.0 reproduces the old
result to the last digit, so the switch `gemm.nvfp4_static_act_scale` can be
turned off for a clean comparison.
The static scale has a second benefit: it is per layer, not per batch. A row's
quantised value does not depend on which other rows happen to share its step,
which is what imp's [determinism](https://github.com/kekzl/imp/blob/main/docs/DETERMINISM.md)
rules need.
With both fixes, Qwen3-30B-A3B lands at 10.9566 / 10.9436 (chunk 0 / chunk 32).
## Fix 3: K and V sharing one FP8 scale
The FP8 KV cache stores keys and values in E4M3 with a scale per layer. The old
rule, still the default and called `legacy`, took the first real prefill's
absmax / 448 with a floor of 1.0. Across Qwen3 0.6B, 1.7B and 8B, 91 of 92
layers sit on that floor: keys and values go into E4M3 unscaled.
That works for Qwen3. It does not for Gemma-4, whose keys are small: amax about 1.6,
against about 16 for the values. With a scale of 1, 17.98 % of Gemma-4's K entries
were subnormal or flushed to zero.
[#3016](https://github.com/kekzl/imp/pull/3016) adds static scales, from the
checkpoint or from one calibration prefill at startup:
`pow2ceil(amax × headroom / 448)`. Powers of two matter here: dividing by a power
of two only shifts the exponent, so a value in the normal range rounds exactly as it did with
scale 1, and the inverse is exact in FP16. [#3023](https://github.com/kekzl/imp/pull/3023)
then gives K and V separate scales, because one shared `max(K, V)` scale is set by
the values and still leaves the keys near the bottom.
Gemma-4-26B-A4B: K entries subnormal or flushed in FP8
[diagram omitted — see the page for the chart]
Track = 20 %. Headroom 2 for both calibrated arms. Source: the edge counts calibrate logs from an FP16 exponent histogram of the calibration prefill (#3023).
Gemma-4-26B-A4B: perplexity above the FP16 KV cache
[diagram omitted — see the page for the chart]
Track = 3.5 %. Chat-turn corpus, FP16 KV 17.5635, legacy 17.9006, shared 18.0870, split 17.6854.
The two charts do not tell the same story, and that is the honest part. The
shared scale removed most of the subnormal keys and still scored *worse* than
legacy. Fewer subnormals alone is not the goal. Per position the differences are
small: mean absolute NLL change against FP16 of 0.4692 (legacy), 0.4739 (shared)
and 0.4659 (split), and split with headroom 1 or 4 reads 0.4757 and 0.4789. The
gain of split sits inside the spread of the headroom choice. That is why both new
modes ship off by default, and why Qwen3, where no arm wins, keeps `legacy`.
Decode speed is not the issue: Gemma-4 FP8 KV tg128 299.13 legacy, 299.25 split.
## How to know you got it right
A perplexity going down is not proof. It could be lower than the model really
is. So [#3021](https://github.com/kekzl/imp/pull/3021) added a parity harness:
the same token ids, the same rows, scored by imp and by vLLM 0.31.0.
| Model | imp | vLLM (pooled over starts) | Δ | vLLM start-to-start spread |
|---|---:|---:|---:|---:|
| Qwen3-8B NVFP4 | 11.2178 | 11.1951 | +0.203 % | 0.571 % |
| Qwen3-8B FP8 (16-bit KV) | 10.7604 | 10.7587 | +0.016 % | 0.000 % |
| Qwen3-30B-A3B NVFP4 (FP8 KV) | 10.9265 | 10.9528 | -0.240 % | 0.328 % |
| Gemma-4-26B-A4B NVFP4 | 17.4793 | 17.7908 | **-1.751 %** | 1.917 % |
Two things stand out. vLLM's NVFP4 perplexity moves by 0.33 to 1.92 % between
process starts. imp is byte-reproducible: repeated runs are equal. And on Gemma-4
imp now scores *lower* than vLLM. That gap appeared with the clamp of fix 1; before
it the difference was +0.059 %. It is inside vLLM's own start-to-start spread and
not filed as a defect, but lower than the reference is still a deviation, and it
is written down as one.
## Honest limits
- **Not bit-equal across chunks on MoE.** With the static scale, some MoE blocks
still fall under 2^-9. Gemma-4 gets 1.5 % worse at chunk 32 and 1.8 % better at
chunk 0 after the clamp.
- **The checkpoint has to have it.** Fix 2 reads the calibration of whoever exported
the model. Every local ModelOpt FP8-KV export ships K and V scales of exactly 1.0,
so `checkpoint` mode equals `legacy` there.
- **Calibration costs a feature.** `calibrate` skips restoring the persisted
prefix cache.
- **Perplexity is one metric.** The degeneration suite (37 to 52 checks per
model) passed with 0 failures on #3016, #3019 and #3023, but none of it is a task
benchmark.
None of the three fixes touched a weight. They changed the number that sits next
to sixteen of them.