Bumping a dependency you can't read

CUTLASS is NVIDIA’s library of GEMM building blocks, and it sits under the NVFP4 4-bit matrix multiplies that make imp fast. The other day I bumped it from v4.5.1 to v4.5.2. One character in one version string. In most projects that’s a shrug and a green checkmark.

In numerical CUDA it isn’t, and the reason is the whole point of this post. A bad dependency bump here doesn’t throw an exception or segfault. It compiles cleanly, runs happily, and returns numbers that are slightly wrong: a GEMM that accumulates a hair differently, a scale decoded one bit off. Silent. The kind of bug that ships and then shows up three weeks later as a model that’s mysteriously a little dumber.

You can’t read your way out of that. I can’t read every line of the 97,000 lines of CUDA in imp, and I certainly can’t read CUTLASS’s release diff and convince myself the maths is unchanged. So the bump can’t be reviewable. It has to be verifiable. Here is the whole procedure, done end to end with Claude Code.

Gate zero: does the tag even exist?

The cheapest check there is, and easy to skip: before changing anything, confirm the version you’re bumping to is real.

git ls-remote --tags https://github.com/NVIDIA/cutlass.git v4.5.2 v4.5.1
# db1c2889...  refs/tags/v4.5.2   <- it exists

A bump to a tag that doesn’t exist isn’t a subtle bug, it’s a guaranteed build break fifteen minutes into a recompile, after you’ve stopped paying attention. One line up front saves that.

One file, on purpose

The next failure mode is structural. Version pins love to hide in two places, the build config and the Dockerfile, and “bump both” is exactly the step a tired human or an eager agent forgets, leaving a build that pulls one version and a container that pulls another. So imp pins dependencies in exactly one authoritative file:

# cmake/imp-deps.cmake - authoritative pin for third-party deps (the make path)
set(IMP_DEP_CUTLASS_TAG  v4.5.2  CACHE STRING "NVIDIA/cutlass git tag")

This matters more with an agent than without. The instruction in the repo’s CLAUDE.md is now a single sentence: bump only this one file. Fewer places to be wrong is fewer ways to be wrong, for me and for the agent. (The old note still said “pins live in two places, bump both”; part of this bump was having the agent fix that stale instruction, since it stopped being true.)

Gate one: does it compile?

Now the first real gate. A clean Docker build that re-clones CUTLASS v4.5.2 and recompiles every translation unit that touches it, the NVFP4 GEMM, the grouped GEMM for mixture-of-experts, the MXFP4 path, the attention kernel, for exactly one architecture: sm_120a, the RTX 5090. About fifteen minutes on my build host (your mileage depends on CPU, cores, and toolkit).

A surprising amount of dependency pain dies right here. A header that moved, a template signature that changed, a deprecation that graduated to a hard error. None of that is subtle; it’s a wall of compiler output. Exit code zero, or the bump is already dead and you’ve lost nothing but a coffee break.

Gate two: do the numbers still agree to tolerance?

This is the gate that matters, and the one most “it compiles, ship it” stories never build. A GEMM library compiling is no evidence at all that it still computes the same answers. You cannot check that by reading the source. You check it by outcome, against a reference that was written without looking at CUTLASS at all.

imp’s oracle for the grouped NVFP4 GEMM is a per-expert matrix multiply done in fp64 on the CPU. The trick that makes it sharp: it decodes the exact same bits the GPU kernel consumed, the packed E2M1 4-bit values and their UE4M3 micro-scales, read back from the card and de-swizzled from CUTLASS’s internal layout, decoded straight from the published format definition and deliberately not from imp’s own dequantiser. That makes the oracle independent of imp’s dequantiser, but it is not free of CUTLASS entirely: to read the scale factors back it has to invert CUTLASS’s internal scale-factor swizzle, so it still encodes a dependency on CUTLASS’s memory layout.

Because the reference and the GPU eat identical quantised inputs, the quantisation error cancels. What’s left to compare is only the multiply-accumulate itself and the final 16-bit output store. Three different numbers show up here, and they are not directly comparable. The element resolution of a single 4-bit operation is coarse, on the order of 1e-1. The GEMM output, by contrast, lands at a relative error around 1e-3, smaller because the 16-bit output store and the accumulation order, not the 4-bit element grid, set it (NVFP4 uses 16-element blocks, which is why a 1e-3 figure is meaningful at all here rather than being swamped by the per-element grid). The test assertion then uses a threshold of 1e-2: loose enough to sit above the measured ~1e-3, tight enough that a real regression has nowhere to hide.

The test that can’t fail is the dangerous one

There’s a sibling test that compares the grouped GEMM against running each expert as a separate GEMM. It looks like verification. It isn’t, quite, and the difference is the whole lesson:

This is the same trap as an agent writing its own tests: a test that compares the engine to itself proves nothing, because both halves are wrong in the same way at the same time. Only a check against an outside source of truth can come back red. So the oracle also hits the corners where staging bugs live: an empty expert with zero tokens, a one-token expert, a large one that crosses the kernel’s tile boundary, a K dimension that crosses the scale-block boundary.

The result for v4.5.2: 187 of 187 numerical tests green, including the grouped fp64 oracle and the alpha-scaling reproducers. The 4-bit maths still agrees with the fp64 reference to within the ~1e-2 relative tolerance gate, and in practice measures ~1e-3. That isn’t bit-for-bit identical (the tolerance gate permits bit-level differences by design), it’s the right kind of “unchanged” for floating-point work. Now I trust the bump, and not before.

What I did not verify, and won’t pretend I did

One honest gap. imp also has a performance gate, a decode-speed regression check against a committed baseline. It didn’t meaningfully fire for this bump, because the baseline benchmark model doesn’t even take the NVFP4 path that CUTLASS drives, so there was nothing to compare before and after. For a dependency bump, correctness is the gate, not speed. If v4.5.2 had touched a hot path I actually exercise, I’d owe a single-session before/after measurement. It didn’t, so I’m not going to dress up a number I never took.

And the gates that matter here run on my own machine, the one with the 5090, as a local docker run --gpus all. The cloud CI is CPU-only; it can’t run the oracles at all. So the thing standing between a silent regression and main isn’t a green checkmark on a pull request. It’s a fp64 reference and a graphics card on my desk.

The principle

Bump it like you can’t read it, because you can’t, and neither can the agent. The agent did the entire mechanical job: checked the tag, edited the one pin file, fixed the stale docs, ran the build, ran the tests, opened the PR. None of that is what made the bump safe. What made it safe was an oracle, written long before this bump was on the table, that doesn’t care who or what changed the code. And to be precise, “0 lines read” is about the bump itself, not zero CUTLASS knowledge: building that oracle meant inverting CUTLASS’s scale-factor swizzle, which took understanding its scale layout. The knowledge was spent once, up front, so each future bump doesn’t have to re-pay it. It only asks one question, the same question every time: did the numbers change? Build that, once, and a scary dependency update becomes a fifteen-minute errand you can hand to a machine. The gates are all sitting in the public repo if you want to read the homework.