A 4-bit number on its own can only be 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6.
Everything else a quantised model does with it comes from the scale it is
multiplied by. NVFP4 stacks two: an FP8 E4M3
micro-scale per 16 values, and one FP32 scale per tensor.
Weights are quantised once, by whoever made the checkpoint. Activations and the KV cache are quantised by the engine, at runtime, for every token. That is where an engine can get it wrong without a single weight bit changing. Between 2 and 3 November 2026, imp fixed three such cases. This post goes through them, and through how imp now checks itself against vLLM.
The format’s edges#
All three bugs sit at the low end of E4M3, the 8-bit float the scales are stored in.
| E4M3 | value |
|---|---|
| largest | 448 |
| smallest normal | 2^-6 = 0.015625 |
| smallest subnormal | 2^-9 ≈ 0.00195 |
| subnormal step | 2^-9, so 3 bits of precision at best, fewer near the bottom |
Below 2^-6 a value loses precision. Below 2^-9 it is gone. An NVFP4 micro-scale is the block’s largest magnitude divided by 6, divided by the tensor scale. So a 16-value block whose largest entry is under 6 × 2^-6 = 0.09375 gets a subnormal scale, and a block under 6 × 2^-9 has nothing left to represent it with.
Fix 1: a block that became zero#
#3019. imp has several NVFP4 activation quantizers: one for prompt tiles, one fused into the MoE kernels, one for small batches, one in the decode epilogue. For a block scale under 2^-9, some clamped it to 2^-9. Others wrote byte 0, and a scale of 0 reads the whole block as zero.
An illustration with a tensor scale of 1: a block whose largest entry is 0.003 needs a micro-scale of 0.003 / 6 = 0.0005, under 2^-9 ≈ 0.00195.
| entry | byte 0 | clamped to 2^-9 |
|---|---|---|
| 0.003 | 0 | 1.5 × 2^-9 ≈ 0.00293 |
| 0.001 | 0 | 0.5 × 2^-9 ≈ 0.00098 |
| 0.0002 | 0 | 0 |
The clamped block is coarse but keeps its shape. The zeroed one is gone.
The symptom was strange: perplexity depended on the prefill chunk size. How many rows are processed together decides which quantizer a row goes through, so the same text through the same weights scored differently.
| Qwen3-30B-A3B, deterministic PPL | chunk 0 | chunk 32 | spread |
|---|---|---|---|
| before | 11.3565 | 11.0110 | 0.3455 |
| every quantizer clamps to 2^-9 | 11.0160 | 11.0190 | 0.0030 |
Normal-range bytes did not change: same expression, same result. The new unit test feeds activations small enough to hit the floor. On the old code, 9216 of 9216 scale bytes came out wrong.
The fix also moved imp’s own reference number. The pinned perplexity of Qwen3-8B, which every pull request is gated against, went from 11.2587 to 11.2352, on purpose, in the same commit. The 11.2587 itself came from fix 2, which merged first.
Fix 2: the scale the checkpoint already had#
#3015. imp quantised activations with a tensor scale of 1.0, so each micro-scale was simply block amax / 6. On Gemma-4, the inputs to the down projection are small. 27.3 % of the shared-MLP blocks and 28.0 % of the expert blocks had an amax under 0.09375, so their scales were subnormal, and in the worst layer 96.6 % were.
The checkpoints carry the fix. When ModelOpt or compressed-tensors quantise a model, they run calibration text through it and store a static input scale for every linear layer: the observed amax divided by 6 × 448 = 2688, so the largest expected activation maps to the largest value NVFP4 can hold. imp read none of it: 11725 linear layers on Gemma-4, ignored at load. Now every quantizer divides by that scale and every GEMM multiplies it back into its output.
| Model | PPL before | PPL after |
|---|---|---|
| Qwen3-30B-A3B (ModelOpt) | 11.3565 | 10.9678 |
| Gemma-4-26B-A4B, chat-turn corpus | 18.2854 | 17.8489 |
| Qwen3-8B (compressed-tensors) | 11.2949 | 11.2587 |
| Qwen3-14B (ModelOpt) | 9.8343 | 9.8152 |
Decode and prefill speed stayed within the pair spread (Gemma-4 tg128
283.59-284.55 before, 282.42-284.97 after). A scale of 1.0 reproduces the old
result to the last digit, so the switch gemm.nvfp4_static_act_scale can be
turned off for a clean comparison.
The static scale has a second benefit: it is per layer, not per batch. A row’s quantised value does not depend on which other rows happen to share its step, which is what imp’s determinism rules need.
With both fixes, Qwen3-30B-A3B lands at 10.9566 / 10.9436 (chunk 0 / chunk 32).
Fix 3: K and V sharing one FP8 scale#
The FP8 KV cache stores keys and values in E4M3 with a scale per layer. The old
rule, still the default and called legacy, took the first real prefill’s
absmax / 448 with a floor of 1.0. Across Qwen3 0.6B, 1.7B and 8B, 91 of 92
layers sit on that floor: keys and values go into E4M3 unscaled.
That works for Qwen3. It does not for Gemma-4, whose keys are small: amax about 1.6, against about 16 for the values. With a scale of 1, 17.98 % of Gemma-4’s K entries were subnormal or flushed to zero.
#3016 adds static scales, from the
checkpoint or from one calibration prefill at startup:
pow2ceil(amax × headroom / 448). Powers of two matter here: dividing by a power
of two only shifts the exponent, so a value in the normal range rounds exactly as it did with
scale 1, and the inverse is exact in FP16. #3023
then gives K and V separate scales, because one shared max(K, V) scale is set by
the values and still leaves the keys near the bottom.
calibrate logs from an FP16 exponent histogram of the calibration prefill (#3023).The two charts do not tell the same story, and that is the honest part. The
shared scale removed most of the subnormal keys and still scored worse than
legacy. Fewer subnormals alone is not the goal. Per position the differences are
small: mean absolute NLL change against FP16 of 0.4692 (legacy), 0.4739 (shared)
and 0.4659 (split), and split with headroom 1 or 4 reads 0.4757 and 0.4789. The
gain of split sits inside the spread of the headroom choice. That is why both new
modes ship off by default, and why Qwen3, where no arm wins, keeps legacy.
Decode speed is not the issue: Gemma-4 FP8 KV tg128 299.13 legacy, 299.25 split.
How to know you got it right#
A perplexity going down is not proof. It could be lower than the model really is. So #3021 added a parity harness: the same token ids, the same rows, scored by imp and by vLLM 0.31.0.
| Model | imp | vLLM (pooled over starts) | Δ | vLLM start-to-start spread |
|---|---|---|---|---|
| Qwen3-8B NVFP4 | 11.2178 | 11.1951 | +0.203 % | 0.571 % |
| Qwen3-8B FP8 (16-bit KV) | 10.7604 | 10.7587 | +0.016 % | 0.000 % |
| Qwen3-30B-A3B NVFP4 (FP8 KV) | 10.9265 | 10.9528 | -0.240 % | 0.328 % |
| Gemma-4-26B-A4B NVFP4 | 17.4793 | 17.7908 | -1.751 % | 1.917 % |
Two things stand out. vLLM’s NVFP4 perplexity moves by 0.33 to 1.92 % between process starts. imp is byte-reproducible: repeated runs are equal. And on Gemma-4 imp now scores lower than vLLM. That gap appeared with the clamp of fix 1; before it the difference was +0.059 %. It is inside vLLM’s own start-to-start spread and not filed as a defect, but lower than the reference is still a deviation, and it is written down as one.
Honest limits#
- Not bit-equal across chunks on MoE. With the static scale, some MoE blocks still fall under 2^-9. Gemma-4 gets 1.5 % worse at chunk 32 and 1.8 % better at chunk 0 after the clamp.
- The checkpoint has to have it. Fix 2 reads the calibration of whoever exported
the model. Every local ModelOpt FP8-KV export ships K and V scales of exactly 1.0,
so
checkpointmode equalslegacythere. - Calibration costs a feature.
calibrateskips restoring the persisted prefix cache. - Perplexity is one metric. The degeneration suite (37 to 52 checks per model) passed with 0 failures on #3016, #3019 and #3023, but none of it is a task benchmark.
None of the three fixes touched a weight. They changed the number that sits next to sixteen of them.