# Raphael Friedmann — AI Infrastructure

> CUDA inference on consumer NVIDIA Blackwell. Building imp, a from-scratch engine that decodes 30B-class models at around 300 tokens per second on a single RTX 5090, plus a ground-up guide to understanding and using AI. Twenty years of infrastructure underneath.

Every post is also available as clean Markdown by appending `.md` to its URL (for example https://rfriedmann.de/blog/how-to-prompt.md). The whole corpus in one file is at https://rfriedmann.de/llms-full.txt.

## The engineering log

- [The oracle you build yourself](https://rfriedmann.de/blog/the-oracle-you-build-yourself/): Dependabot opened five PRs against this site, three of them major. There is no test suite here, so the check had to be built: render the site twice, once with the old renderer and once with the new one, and diff what came out.
- [No oracle for a meadow](https://rfriedmann.de/blog/no-oracle-for-a-meadow/): Everything else I've handed to an agent could be checked against a number. This one couldn't: one sentence to Claude Opus 5, 1,922 lines of WebGL, and a honeybee in backlight. What stands in for a test when the only judge is your eye.
- [Continuous batching: how one GPU serves a crowd](https://rfriedmann.de/blog/continuous-batching/): A single decode stream wastes most of the GPU. The fix is to serve many requests at once and let them share each pass over the weights, but only if you stop waiting for the whole batch to finish. Continuous batching, and why it's the throughput trick that matters.
- [Speculative decoding: let a small model do the guessing](https://rfriedmann.de/blog/speculative-decoding/): Decode is bandwidth-bound, so the GPU's maths units sit half-idle one token at a time. Speculative decoding spends that idle compute to verify several guessed tokens at once, for the same output, faster.
- [axo: learning without backprop](https://rfriedmann.de/blog/axo-learning-without-backprop/): A from-scratch spiking neural network in C++/CUDA on one RTX 5090 that learns with purely local rules, no backprop and no central optimiser, and grows into a little creature that learns to hunt over a single continuous life.
- [Bumping a dependency you can't read](https://rfriedmann.de/blog/bumping-a-dependency-you-cant-read/): Updating CUTLASS, the GEMM library under imp's 4-bit maths, from v4.5.1 to v4.5.2. In numerical CUDA a bad bump doesn't crash, it quietly returns wrong numbers. So you don't review the change, you make it verifiable.
- [Online softmax, and the register that can't move](https://rfriedmann.de/blog/online-softmax-and-flashattention/): The deep version of FlashAttention on consumer Blackwell: how online softmax avoids the giant score matrix, and why keeping the output in registers forces raw mma.sync over WMMA.
- [Decoding GGUF faster than llama.cpp on a 5090](https://rfriedmann.de/blog/gguf-decode-beats-llama-cpp/): imp decodes dense GGUF 37 to 72% faster than llama.cpp on a 5090. Not because llama.cpp is naive, it isn't, but because imp is built for one exact chip. And the one model where it loses.
- [NVFP4 at the bit level](https://rfriedmann.de/blog/nvfp4-at-the-bit-level/): Every other post says '4-bit' and moves on. Here's what NVFP4 actually is, down to the four bits and the two scales, and the two Blackwell instructions (cvt and mma) that make it fly.
- [When the model isn't a transformer: GDN and Mamba2](https://rfriedmann.de/blog/when-the-model-is-not-a-transformer/): Not every LLM is built on attention. Gated DeltaNet and Mamba2 replace the score matrix with a recurrence, which gives constant memory, and one stubborn precision floor.
- [Three kernels for a chip the ecosystem skipped](https://rfriedmann.de/blog/optimizing-kernels-consumer-blackwell/): FlashAttention-2 and NVFP4 GEMM, tuned from scratch for the RTX 5090, and what the profiler taught me when every textbook optimisation turned out to be a red herring.
- [How 97,000 lines of CUDA got written by an AI agent](https://rfriedmann.de/blog/writing-cuda-with-an-agent/): Both of my imp posts end with 'every line was written by Claude Code'. This is the question that always follows: how does that actually work, and how do you trust it?
- [Serving 30B models at 300 tok/s on a single RTX 5090](https://rfriedmann.de/blog/serving-30b-models-rtx-5090/): Why no existing inference engine fully exploits consumer Blackwell, what NVFP4 changes, and the numbers from building one that does.
- [What a consumer RTX 5090 is missing next to a datacenter GPU](https://rfriedmann.de/blog/what-the-5090-lacks-vs-datacenter/): The 5090 and the B200 are both Blackwell, but the consumer chip is missing whole capabilities. Here is what is gone, what it costs, and why you cannot just port data centre code.
- [The KV cache, and what really limits long context](https://rfriedmann.de/blog/kv-cache-and-long-context/): The beginner version calls it short-term memory. The engineering reality: the KV cache, not the weights, is what decides how long your context can get, and what runs you out of memory.
- [Prefill, decode, and the roofline that explains everything](https://rfriedmann.de/blog/prefill-vs-decode-roofline/): An inference engine doesn't have one speed, it has two, and they obey opposite laws. The roofline model is the single most useful lens for reasoning about LLM performance.
- [How imp turns a model file into words](https://rfriedmann.de/blog/how-imp-works-model-to-token/): A plain-language tour of what happens inside an AI engine: how a model gets loaded, and the steps every message runs through to come back as text. Jargon explained as we go.

## Understanding AI

- [What is an LLM, and how does it actually make words?](https://rfriedmann.de/blog/what-is-an-llm/): A jargon-free explanation of large language models: what they are, why they're basically a very good autocomplete, and how they write one word at a time.
- [Why run AI on your own machine?](https://rfriedmann.de/blog/why-run-ai-locally/): Cloud chatbots are easy and, honestly, hard to beat. The real and narrower case for running a model yourself, and the big things you give up to do it.
- [What a GPU is, and why AI needs one](https://rfriedmann.de/blog/what-is-a-gpu/): Why running AI means buying a graphics card, what makes a GPU different from a CPU, and why the amount of memory on the card is the number that really matters.
- [How a 30-billion-parameter model fits on one card](https://rfriedmann.de/blog/how-a-big-model-fits-on-one-card/): Quantisation, explained for normal people: how shrinking each number in a model lets a giant fit on a desktop graphics card, and what it costs.
- [How to read a model's name and specs](https://rfriedmann.de/blog/reading-model-specs/): Model names like Qwen3-30B-A3B-Q4_K_M look like a cat walked across the keyboard. Here's how to decode them, and the handful of specs that actually matter.
- [Mixture of Experts: how a 30B model runs like a 3B one](https://rfriedmann.de/blog/mixture-of-experts-explained/): The trick behind names like Qwen3-30B-A3B: split the model into many experts, run only a few per token. Why it suits a single GPU so well, and the headaches it brings.
- [What an LLM can and can't do in your infrastructure](https://rfriedmann.de/blog/llms-in-your-infrastructure/): After twenty years of keeping systems running and a couple of years deep in LLMs, an honest split: where AI genuinely helps an ops team, and where it's a liability waiting to happen.
- [LLMs in the terminal: a sysadmin's honest list](https://rfriedmann.de/blog/llms-in-the-terminal/): The concrete, everyday ways an LLM actually earns its place in a sysadmin's workflow, and the handful of rules that keep it from causing real damage.
- [Self-hosting an LLM for your team: usually don't](https://rfriedmann.de/blog/self-hosting-an-llm-for-your-team/): The tempting case for running your own model, and the honest reason most teams shouldn't: a small model's error rate burns more working time than the API fees it saves. Plus the cases where it still wins.
- [How a model learns: training, in plain words](https://rfriedmann.de/blog/how-a-model-learns/): Every LLM starts as random noise and is shaped by one repeated loop: guess, measure the error, nudge billions of dials. Here's how that training actually works, and why it costs a fortune.
- [Pretraining, fine-tuning, and RLHF](https://rfriedmann.de/blog/pretraining-finetuning-rlhf/): A raw trained model can continue text but won't answer you. The three stages that turn a text-continuer into a helpful assistant, and which one you actually need.
- [When not to use AI: an ops take](https://rfriedmann.de/blog/when-not-to-use-ai/): The unfashionable half of the conversation. The cases where reaching for an LLM is the wrong choice, and a simple heuristic for telling them apart from the good ones.
- [AI that isn't an LLM](https://rfriedmann.de/blog/ai-that-isnt-an-llm/): Language models get all the attention, but they're one corner of AI. A quick tour of the other big families, what each is for, and how to tell when an LLM is the wrong tool.
- [What an AI agent actually is](https://rfriedmann.de/blog/what-is-an-ai-agent/): Strip the buzzword and an agent is one simple thing: a language model put in a loop and handed tools, so it can do things instead of just talking about them.
- [How an agent uses tools and memory](https://rfriedmann.de/blog/how-an-agent-uses-tools-and-memory/): The loop, one level down: how the model actually requests a tool, why its whole memory is just the growing transcript, and why long multi-step tasks fall apart.
- [When to let an agent loose: an ops take](https://rfriedmann.de/blog/letting-an-agent-loose/): An agent that can act can also act wrongly. After twenty years of running systems, here's how I decide how much rope to give one, and the guardrails that earn their keep.
- [Why an LLM trips over the r's in 'strawberry'](https://rfriedmann.de/blog/why-models-cant-spell/): Ask a top model to count the letters in a word and it often gets it wrong. Not a bug, and not stupidity: it's because the model never sees letters at all. A plain-words tour of tokens.
- [Why you get a different answer every time](https://rfriedmann.de/blog/why-different-answers/): Ask a model the same thing twice and you can get two different replies. That's a deliberate dice-roll, not a glitch, and one knob controls how loaded the dice are. Meet temperature.
- [Why models make things up](https://rfriedmann.de/blog/why-models-make-things-up/): A model will hand you a wrong fact, a fake citation or an invented function with total confidence. It's not lying and it's not broken. Here's where 'hallucinations' actually come from, and how to work around them.
- [Pointing an LLM at your own documents (RAG, honestly)](https://rfriedmann.de/blog/your-own-documents/): Everyone wants 'a ChatGPT that knows our internal docs'. The mechanism is called RAG, it's simpler than the hype, and most of the work is the boring retrieval half nobody demos. An ops-eye view.
- [What the model remembers: the context window](https://rfriedmann.de/blog/the-context-window/): An LLM has no memory between messages. Everything it 'knows' about your conversation is re-read from scratch each turn, and it only fits so much. Meet the context window, the single most useful thing to understand about how chatbots behave.
- [How a model sees a picture](https://rfriedmann.de/blog/how-models-see-images/): You can hand a modern AI a photo and ask about it. But a language model only understands tokens, so what happens to the image? A plain-words look at how vision gets bolted onto a model that only ever knew words.
- [How to actually ask: prompting without the magic words](https://rfriedmann.de/blog/how-to-prompt/): There are no secret incantations. Good prompting is just clear instructions to a brilliant, literal-minded assistant with no memory. A practical guide that follows straight from how the model works.
- [Prompt injection: the security hole in every LLM app](https://rfriedmann.de/blog/prompt-injection/): The moment your AI reads anything an attacker can influence, a web page, an email, a document, that content can hijack it. There's no clean fix, only containment. The honest ops briefing on the vulnerability nobody demos.

## Optional

- [Full text of every post in one file](https://rfriedmann.de/llms-full.txt)
- [RSS feed](https://rfriedmann.de/rss.xml)
- [imp on GitHub](https://github.com/kekzl/imp)