AI Infrastructure
Raphael Friedmann
Twenty years in IT, aimed at the AI frontier.
I started in C++ two decades ago and spent the years since keeping real systems running. Now I point that experience at one question: how fast can a single consumer graphics card serve modern AI? The answer is imp, a from-scratch engine that decodes 30B-class models at around 300 tokens per second on one RTX 5090, single-stream, using 4-bit tensor-core math no other engine runs on this card. I write it up two ways, a log from the frontier and a ground-up guide to understanding and using AI.
- ~300 tok/s — 30B MoE, single-stream
- 37–72% faster than llama.cpp (dense GGUF)
- only native NVFP4 on this card
The card's native 4-bit, spelled out
Deep dive: NVFP4 at the bit level →imp is the only engine running this card's native 4-bit math. Here is the whole format — four bits, two scales.
Latest writing
Browse all: the log · understanding AIThe newest from both tracks, freshest first.
Continuous batching: how one GPU serves a crowd
A single decode stream wastes most of the GPU. The fix is to serve many requests at once and let them share each pass over the weights, but only if you stop waiting for the whole batch to finish. Continuous batching, and why it's the throughput trick that matters.
Understanding AIPrompt injection: the security hole in every LLM app
The moment your AI reads anything an attacker can influence, a web page, an email, a document, that content can hijack it. There's no clean fix, only containment. The honest ops briefing on the vulnerability nobody demos.
Understanding AIHow a model sees a picture
You can hand a modern AI a photo and ask about it. But a language model only understands tokens, so what happens to the image? A plain-words look at how vision gets bolted onto a model that only ever knew words.
Understanding AIHow to actually ask: prompting without the magic words
There are no secret incantations. Good prompting is just clear instructions to a brilliant, literal-minded assistant with no memory. A practical guide that follows straight from how the model works.
The logSpeculative decoding: let a small model do the guessing
Decode is bandwidth-bound, so the GPU's maths units sit half-idle one token at a time. Speculative decoding spends that idle compute to verify several guessed tokens at once, for the same output, faster.
Understanding AIWhat the model remembers: the context window
An LLM has no memory between messages. Everything it 'knows' about your conversation is re-read from scratch each turn, and it only fits so much. Meet the context window, the single most useful thing to understand about how chatbots behave.
From the labs
Official feeds, rebuilt daily: all AI news →The latest model releases and announcements, straight from the source.
Two ways in
A running log from the frontier, and a ground-up guide. Pick the door that fits.
From zero, and the honest calls
New to this, or running AI for a team? What an LLM is from the ground up, and when it helps versus when it bites.
Start reading → The engineering log · 15 postsFrontier CUDA, down to the bits
The build log for imp, and how fast inference actually works on a single consumer card. Newest first.
Open the log →Selected work
The question, answered in code: one engine, built from scratch, running the consumer card's native 4-bit tensor cores that the big engines leave idle here.
A from-scratch engine that runs large language models on a single NVIDIA RTX 5090: ~300 tok/s on 30B-class MoE single-stream, 37–72% faster than llama.cpp on dense GGUF, and the only engine using the card's native 4-bit math. Single-stream is its axis; vLLM still leads on batched throughput. Written entirely with Claude Code.
- ~97k lines C++/CUDA
- NVFP4 4-bit native
- 37–72% over llama.cpp · dense GGUF, batch 1
- MIT licensed
Every figure is dated and commit-anchored in BENCHMARKS.md, with the exact command to reproduce it.
Also tinkering: axo, a neuromorphic learner that trains without backprop →