Raphael Friedmann · AI infrastructure · Karlsruhe
How fast can one consumer card serve modern AI?
Fast enough that the answer needed its own engine.
I started in C++ two decades ago and spent the years since keeping real systems running. That question is what I point the experience at now, and imp is the answer in code: a from-scratch inference engine for a single RTX 5090. I write it up two ways — a log from the frontier, and a ground-up guide for everyone who just wants to understand the thing.
- ~300 tok/s — 30B MoE, single-stream
- 37–72% faster than llama.cpp (dense GGUF)
- NVFP4 native — no other engine on this card
The card's native 4-bit, spelled out
Deep dive: NVFP4 at the bit level →Here is the whole format — four bits, two scales. Every cell on this site comes from it.
Latest writing
Browse all: the log · understanding AIThe newest from both tracks, freshest first. Each band is the post's bit signature.
The oracle you build yourself
Dependabot opened five PRs against this site, three of them major. There is no test suite here, so the check had to be built: render the site twice, once with the old renderer and once with the new one, and diff what came out.
The logNo oracle for a meadow
Everything else I've handed to an agent could be checked against a number. This one couldn't: one sentence to Claude Opus 5, 1,922 lines of WebGL, and a honeybee in backlight. What stands in for a test when the only judge is your eye.
The logContinuous batching: how one GPU serves a crowd
A single decode stream wastes most of the GPU. The fix is to serve many requests at once and let them share each pass over the weights, but only if you stop waiting for the whole batch to finish. Continuous batching, and why it's the throughput trick that matters.
Understanding AIPrompt injection: the security hole in every LLM app
The moment your AI reads anything an attacker can influence, a web page, an email, a document, that content can hijack it. There's no clean fix, only containment. The honest ops briefing on the vulnerability nobody demos.
Understanding AIHow a model sees a picture
You can hand a modern AI a photo and ask about it. But a language model only understands tokens, so what happens to the image? A plain-words look at how vision gets bolted onto a model that only ever knew words.
Understanding AIHow to actually ask: prompting without the magic words
There are no secret incantations. Good prompting is just clear instructions to a brilliant, literal-minded assistant with no memory. A practical guide that follows straight from how the model works.
Selected work
The question, answered in code: one engine, built from scratch, running the consumer card's native 4-bit tensor cores that the big engines leave idle here.
A from-scratch engine that runs large language models on a single NVIDIA RTX 5090. Single-stream latency is its axis — one person, one card, one answer coming back as fast as the silicon allows. vLLM still leads on batched throughput, and that is a different job. Written entirely with Claude Code.
- ~97k lines C++/CUDA
- NVFP4 4-bit native
- CUDA graphs · continuous batching
- MIT licensed
Every figure is dated and commit-anchored in BENCHMARKS.md, with the exact command to reproduce it.
Also tinkering: axo, a neuromorphic learner that trains without backprop →From the labs
Official feeds, rebuilt daily: all AI news →Other people's announcements, straight from the source and unranked. Everything above is mine.