How 97,000 lines of CUDA got written by an AI agent

Both of my other imp posts end on the same note: every line of the first-party engine code was written by Claude Code (vendored third-party code, such as third_party/stb, is excluded). That always draws the same question, usually with some scepticism: how does that actually work, and how do you trust ninety-seven thousand lines of CUDA that an AI wrote? That headline figure is the raw count, so let me be honest about it up front: cloc puts the engine at roughly 75k lines of actual code (excluding comments and blanks) and about 105k lines raw once comments and blank lines are included. Here is the honest answer. It is less remarkable and more disciplined than people expect.

Who does what

The division of labour is what matters most, and it is not 50/50.

I am the architect and the quality inspector. It is the implementer. The interesting question is not whether an agent can write a CUDA kernel, it plainly can, it is how you keep a thousand of those commits from quietly rotting the thing.

You cannot read your way out of this

The instinctive answer to “how do you trust it” is “read the code”. At this scale you cannot, and the arithmetic is unforgiving. Reviewing novel CUDA properly, the kind where a wrong line is a silent numerical bug and not a crash, goes, in my experience, maybe a couple hundred lines an hour if you are honest about actually understanding it. Ninety-seven thousand lines is then on the order of three months of full-time reading, and that is the engine alone; on top of it sit roughly 40k lines of authored test code, plus generated benchmark data (the roofline JSON), which is machine output rather than authored code and does not belong in that reading total. It also moves: a thousand-plus commits mean that by the time you reached the end, you would be reading a different codebase than the one you started.

So line-by-line review simply cannot be the discipline here, the way it is for a script you paste into a terminal. You verify what the code does, not what it says. Which is the entire reason the gates exist.

The answer is gates, not trust

You do not trust agent-written code because it reads well. You gate it. A change only lands if it clears three checkpoints, in order:

1. Build and tests green

   The full suite, including GPU kernel tests checked against independent CPU reference implementations, the kind of test that can catch a kernel that was wrong from day one.

2. Performance gate

   Decode speed must stay within 3% of a committed baseline, prefill within 5%, measured in a single session because cross-session GPU numbers drift. A "speedup" that trips this gate isn't one.

3. Coherence gate

   A suite called degen_suite runs against a live server, hunting the failures unit tests can't see: repetition loops, reasoning leaking into the answer, prompt-blindness, long-context forgetting, multi-turn garble.

Only after all three does it merge.

The war story that built the third gate

That coherence gate exists because of a day things went wrong.

At one point the throughput numbers climbed and the output quality quietly fell. A change to the attention kernel made 9 of 12 models effectively prompt-blind, ignoring the actual question, while every single correctness test stayed green.

The tests passed because they ran on synthetic inputs that could not expose the failure. An audit afterwards made the point plain: only a minority of the suite had independent ground truth capable of catching a kernel that was wrong from the start.

So degen_suite was created to test the model the way a user would, on real prompts against a running server, and a rule was established: a performance optimisation that regresses quality is not a performance optimisation. Throughput numbers say nothing about correctness. You need both gates, because the agent will readily make the first number go up while the second one falls, and so, honestly, will a human.

And the agent writes the tests, too

There is an obvious hole in all of this: the agent writes the gates as well. The tests, the coherence suite, the benchmark harness, all of it is agent-written. So what stops it from writing a test that passes for the same wrong reason the code is wrong? Nothing automatic. An agent left to check its own homework will readily produce a suite that is green and meaningless, and the prompt-blind failure above was exactly that: a wall of tests passing on inputs that could never have failed.

What saves it is two things, and both need the human. First, the tests that count are the ones with an independent source of truth the agent cannot fake by construction, a kernel checked against a from-scratch CPU implementation of the same maths, a coherence suite that talks to a real server the way a user would. A test that compares the engine to itself proves nothing; a test that compares it to an outside reference can actually fail. Second, someone has to distrust a green run on principle and ask what it is not testing, which is precisely the audit that found only a minority of the suite had independent ground truth earning its keep.

So “the AI writes its own tests” is not the catch it sounds like, as long as you remember that a test is only ever as good as the ground truth behind it, and that deciding what ground truth to demand is still a human’s job.

What it’s good at, and what it isn’t

After a thousand commits, the shape of the collaboration is clear.

The agent is excellent at producing a large volume of correct, conventional code quickly, following the patterns already in the codebase, and working through the unglamorous parts: the kernel, plus its reference implementation, plus the test, plus the three reverts when the clever version turns out slower. It does not get bored.

What it still needs a human for is judgement. Knowing which bottleneck is worth a week. Distrusting a green test suite. Calling a clever-but-slower result dead, the TMA experiment that was built, measured, and deliberately kept as a documented failure. Holding the whole architecture in mind across a thousand commits. The reverts, it turns out, matter as much as the successes.

So the real answer to “how do you trust it?” is: I do not. I gate it. And every one of those gates, the reference tests, the performance baseline, the coherence suite, is sitting in the public repo for anyone who wants to check the homework.