The oracle you build yourself

Dependabot opened five pull requests against this site. Three were major bumps: node 22 → 26 in the build stage, nginx 1.27 → 1.31 in the serve stage, and Astro 6.4.7 → 7, the renderer that turns the markdown you’re reading into HTML.

This repository has no tests. It has no CI. And a merge to main is a deploy: the workflow SSHes into the box, resets to origin/main, and rebuilds the container that serves rfriedmann.de. So the question was the same one I asked when bumping a dependency I couldn’t read, except this time none of the machinery that answered it existed.

The failure mode isn’t a crash

A renderer that breaks loudly is the easy case. The build exits non-zero, the image never gets built, the old container keeps serving, and you find out in ninety seconds.

What I was actually afraid of is the quiet version. A major version of a renderer changes how markdown becomes HTML and how that HTML gets minified. The build stays green, forty-five pages come out the other end, and some of them are slightly different: a heading that lost its anchor id, a paragraph where two words ran together, an entity escaped twice so a reader sees < in the middle of a sentence. Nothing errors. It just quietly ships.

Reading the upgrade guide doesn’t settle that. An upgrade guide tells you what the authors thought was worth mentioning, and it’s written against every site, not mine. It can’t tell me whether this markdown, with these forty-five pages, still comes out the same. That’s not a documentation problem, it’s a scope problem: only my content can answer a question about my content.

A static site is a pure function

Here’s what makes this tractable, and it’s the whole trick. A static site generator is a pure function. Feed it a git tree, get a directory of files. Same input, same output, every time.

So I don’t need a test suite. I need the same input run through both compilers:

# the new build, from the working tree with all five bumps applied
docker build -t rfriedmann-new .

# the old build, from an untouched checkout of main
git worktree add "$SCRATCH/old" main
docker build -t rfriedmann-old "$SCRATCH/old"

# pull dist out of both images
for v in old new; do
  cid=$(docker create "rfriedmann-$v")
  docker cp "$cid":/usr/share/nginx/html "$SCRATCH/dist-$v"
  docker rm "$cid"
done

The first comparison is free and nearly worthless. The file lists are identical apart from one content hash in a stylesheet name. Good. Then I compared the files themselves, and all forty-five HTML pages differed. That’s not a signal, that’s noise with a number attached.

The oracle was wrong before the code was

So I wrote something sharper: strip the tags, compare the visible text. Every page still differed, and every page differed in the same place — right at the top, in the favicon.

alt: "...Raphael FriedmannRF\"> Skip to content Raphael Friedmann The log..."
neu: "...Raphael FriedmannSkip to contentRaphael FriedmannThe log..."

That RF"> is not on the page. It’s an artefact of my own stripper. The favicon is an inline SVG in a data: URI, and Astro 6 emitted its angle brackets raw inside the attribute, so my <[^>]*> regex hit a > that belonged to the value and thought the tag had ended. Astro 7 escapes them as <, so the same regex swallows the whole tag cleanly. Two different renderings, identical to a browser — an HTML parser decodes the entity right back — and my sieve reported it as a difference on every single page.

The first thing a homemade oracle finds is usually a bug in the homemade oracle. That is not a reason to skip building one; it’s a reason to look at what it says instead of trusting the count. A checker that reports “45 of 45 pages changed” has told you nothing, and the temptation at that moment is to wave it off as whitespace noise and merge. That’s precisely when it’s worth another twenty minutes.

Asking a question worth answering

The insight that made the difference: whitespace between block elements is invisible — the layout decides that spacing, not the markup — while whitespace inside a sentence is the whole ballgame. So compare only the text inside prose elements, and classify each difference:

// same block, both builds. If removing every space makes them equal,
// only whitespace moved. If not, actual characters changed.
const strip = s => s.replace(/\s/g, '');

if (a === b) continue;                       // identical
if (strip(a) === strip(b)) { wsOnly++; }     // whitespace only
else { real++; report(a, b); }               // this is the interesting pile

Forty lines of JavaScript, most of it walking directories. And now the answer means something:

The two that mattered

The first real difference was a non-event, and I want to name it anyway, because “the oracle fired twice” sounds worse than it was:

alt: "...divide the accumulator by the normaliser, O <- O / l, to get..."
neu: "...divide the accumulator by the normaliser, O <- O / l, to get..."

< and < are two spellings of <. Different bytes, same character, same page.

The second one is why I’m writing this post. A heading in the roofline piece is written in markdown as ## "Compute is free" — straight quotes, which the renderer turns into typographic ones. Here’s what each version produced:

alt: id="compute-is-free">”Compute is free”      U+201D … U+201D
neu: id="compute-is-free">“Compute is free”      U+201C … U+201D

Astro 6 opened that heading with a closing quotation mark. Astro 7 opens it with an opening one. That’s a typographic bug that has been live on this site since June, on a page I have re-read a dozen times, and I never once saw it. The upgrade I was scrutinising for regressions quietly fixed something instead.

The forty-one that didn’t matter, and why

Astro 7 compresses HTML with JSX whitespace rules, which is more aggressive than what came before. Mostly that lands between flex items, where it’s invisible by construction. One case needed a real look:

alt: <p class="post-meta"> 27 March 2026 <span class="level-badge">…
neu: <p class="post-meta">27 March 2026<span class="level-badge">…

That space is gone, and .post-meta is a plain block, not a flex container — so nothing supplies the gap automatically. Whether this is a regression or a rounding error lives entirely in the stylesheet, and the stylesheet says the badge is inline-flex with margin-left: .6rem. So the date and the badge are about four pixels closer together than they were, and nothing collides.

What I did not verify

I argued that last paragraph from the CSS, not from a screenshot. The browser I’d have used for a visual diff had no Chromium installed, and I wasn’t going to install a browser on this host or pull a two-gigabyte image to confirm four pixels.

So the honest version is: the layout is verified by construction — every container whose whitespace changed is either flex with a gap or has an explicit margin — and not by eye. That’s a weaker claim than the text comparison, which is exhaustive and mechanical. If you’re doing this on something where four pixels matter, take the screenshots.

And then the deploy failed anyway

Merged, pushed, and the deploy went red inside twenty seconds. Not the bump:

ERROR: failed to resolve source metadata for docker.io/library/nginx:1.31-alpine:
unexpected status from HEAD request: 429 Too Many Requests

Docker Hub rate-limited the server. Two new base images in one build — node 26 and nginx 1.31, neither cached locally — and the anonymous pull limit said no. A re-run ten minutes later went green.

The failure was harmless in a way worth noticing: docker compose up --build failed at the build step, so the running container was never stopped. The site served the old, working build the entire time, and the daily news rebuild would have retried on its own regardless. But it’s a clean illustration that the oracle answers exactly one question — did the output change? — and infrastructure flakiness is a different question, needing a different gate. Mine now retries the pull three times.

The principle

The CUTLASS oracle was expensive. It meant re-deriving 4-bit decode from the format spec and inverting a vendor library’s internal swizzle, and it only pays off because it’s reused on every future bump.

This one took twenty minutes and forty lines of JavaScript, because the output is text and text diffs itself. That asymmetry is the point. Most projects that render something deterministic — a site, a report, a config bundle, a schema — have this oracle sitting there for the taking, and almost nobody builds it, because there’s no test file to put it in and it feels like it doesn’t count.

The shape is identical in both cases: compare against something outside the change. For CUTLASS that was fp64 arithmetic derived from a published spec. Here it’s the previous build’s own output. Neither one asks whether the new code is good — that’s not the question a dependency bump raises. Both ask whether it does something different, and then make you look at every place where it does.

The last step was to stop doing it by hand. The repo now builds every pull request into a real image, starts the container, and checks that the routes it should serve come back 200 with content on them. Patch and minor updates merge themselves once that’s green; majors wait for a human, because a major is exactly the case where the diff deserves reading. The mechanical part is worth automating. The judgement isn’t.