# Raphael Friedmann — AI Infrastructure > CUDA inference on consumer NVIDIA Blackwell. imp is a from-scratch engine that decodes 30B-class models at around 350 tokens per second on a single RTX 5090, plus a ground-up guide to understanding and using AI. Based on twenty years of infrastructure experience. Every post is also available as clean Markdown by appending `.md` to its URL (for example https://rfriedmann.de/blog/how-to-prompt.md). The whole corpus in one file is at https://rfriedmann.de/llms-full.txt. ## The engineering log - [The kernel that ran zero times](https://rfriedmann.de/blog/the-kernel-that-ran-zero-times/): A rewrite made an imp kernel up to 33 % faster, and serving never launches that kernel. Four results from one day that changed no engine code, and the ledger that stops the next audit from paying for them twice. - [Speculative decoding when the chats overlap](https://rfriedmann.de/blog/mtp-when-chats-overlap/): Qwen3.8-27B ships its own draft head. On imp it sped up one chat by 35 %, and any second chat or UI side request switched it off. Nine changes later, four overlapping chats decode at about 140 tok/s instead of 84. - [Quantisation lives in the scales](https://rfriedmann.de/blog/quantization-lives-in-the-scales/): Three imp fixes that changed no weight bit: a block scale that rounded to zero, an activation scale the checkpoint already shipped, and one FP8 scale shared by K and V. Qwen3-30B-A3B perplexity 11.3565 to 10.9566. - [APA: sloppy attention, exact where it counts](https://rfriedmann.de/blog/apa-adaptive-precision-attention/): A prefill kernel for the RTX 5090 that runs every attention tile in FP4, recomputes the 6 to 10 % that carry the softmax mass in exact FP16, and merges both: 2.50x faster than all-exact at 122880 keys. - [How imp runs Qwen3.8-Flash-Next on a 32 GB card](https://rfriedmann.de/blog/flash-next-on-a-32gb-card/): 56 GiB of experts, 32 GB of VRAM: all experts on the host, a GPU-managed cache in VRAM, and decode from 5.8 to 75-80 tokens per second on real text. - [Making a model emit valid JSON, every time](https://rfriedmann.de/blog/constrained-decoding-valid-json/): Asking nicely gets you valid JSON almost always, and almost is a broken integration. The fix is in the sampler: delete every invalid token before you sample. - [The oracle you build yourself](https://rfriedmann.de/blog/the-oracle-you-build-yourself/): Dependabot opened five PRs against this site, three of them major. No test suite, so I built the check: render the site with both versions and diff it. - [No oracle for a meadow](https://rfriedmann.de/blog/no-oracle-for-a-meadow/): One sentence to Claude Opus 5, 1,922 lines of WebGL, a honeybee in backlight, no number to check. What stands in for a test when the only judge is your eye? - [Continuous batching: how one GPU serves a crowd](https://rfriedmann.de/blog/continuous-batching/): A single decode stream wastes most of the GPU. Serve many requests at once, share each pass over the weights, and stop waiting for the whole batch to finish. - [Speculative decoding: let a small model do the guessing](https://rfriedmann.de/blog/speculative-decoding/): Decode is bandwidth-bound, so the GPU's maths units sit mostly idle. Speculative decoding spends that compute verifying guessed tokens: same output, faster. - [axo: learning without backprop](https://rfriedmann.de/blog/axo-learning-without-backprop/): A from-scratch spiking neural network in C++/CUDA on one RTX 5090 that learns with local rules, no backprop, and grows into a creature that learns to hunt. - [Bumping a dependency you can't read](https://rfriedmann.de/blog/bumping-a-dependency-you-cant-read/): Updating CUTLASS under imp's 4-bit maths. In numerical CUDA a bad bump doesn't crash, it returns wrong numbers. You don't review it, you make it verifiable. - [Online softmax, and the register that can't move](https://rfriedmann.de/blog/online-softmax-and-flashattention/): FlashAttention on the 5090, in depth: how online softmax avoids the giant score matrix, and why register-resident output forces raw mma.sync over WMMA. - [What a consumer RTX 5090 is missing next to a datacenter GPU](https://rfriedmann.de/blog/what-the-5090-lacks-vs-datacenter/): The 5090 and the B200 are both Blackwell, but the consumer chip lacks whole capabilities. What is gone, what it costs, and why data centre code won't port. - [Decoding GGUF faster than llama.cpp on a 5090](https://rfriedmann.de/blog/gguf-decode-beats-llama-cpp/): imp decodes dense GGUF 37 to 72% faster than llama.cpp on a 5090. Not because llama.cpp is naive: imp is built for one chip. And where it loses. - [NVFP4 at the bit level](https://rfriedmann.de/blog/nvfp4-at-the-bit-level/): Every other post says '4-bit' and moves on. Here's NVFP4 down to the four bits and two scales, and the two Blackwell instructions (cvt, mma) that make it fly. - [When the model isn't a transformer: GDN and Mamba2](https://rfriedmann.de/blog/when-the-model-is-not-a-transformer/): Not every LLM is built on attention. Gated DeltaNet and Mamba2 swap the score matrix for a recurrence: constant memory, and one stubborn precision question. - [Three kernels for a chip the ecosystem skipped](https://rfriedmann.de/blog/optimizing-kernels-consumer-blackwell/): FlashAttention-2 and NVFP4 GEMM tuned from scratch for the RTX 5090, and what the profiler taught me when every textbook optimisation was a red herring. - [How 97,000 lines of CUDA got written by an AI agent](https://rfriedmann.de/blog/writing-cuda-with-an-agent/): Both of my imp posts end with 'every line was written by Claude Code'. The question that always follows: how does that actually work, and how do you trust it? - [Serving 30B models at 300 tok/s on a single RTX 5090](https://rfriedmann.de/blog/serving-30b-models-rtx-5090/): Why no existing inference engine fully exploits consumer Blackwell, what NVFP4 changes, and the numbers from building one that does. - [The KV cache, and what really limits long context](https://rfriedmann.de/blog/kv-cache-and-long-context/): The beginner version calls it short-term memory. In reality the KV cache, not the weights, decides how long your context gets and what runs you out of memory. - [Prefill, decode, and the roofline that explains everything](https://rfriedmann.de/blog/prefill-vs-decode-roofline/): An inference engine doesn't have one speed but two, and they obey opposite laws. The roofline is the most useful lens for reasoning about LLM performance. - [How imp turns a model file into words](https://rfriedmann.de/blog/how-imp-works-model-to-token/): A plain-language tour of what happens inside an AI engine: how a model gets loaded, and the steps every message runs through to come back as text. ## Understanding AI - [What an LLM is, and how it produces words](https://rfriedmann.de/blog/what-is-an-llm/): A jargon-free explanation of large language models: what they are, why they're basically a very good autocomplete, and how they write one word at a time. - [When running AI on your own machine makes sense](https://rfriedmann.de/blog/why-run-ai-locally/): Cloud chatbots are easy to use and hard to beat. The narrower case for running a model yourself, and what you give up to do it. - [What a GPU is, and why AI needs one](https://rfriedmann.de/blog/what-is-a-gpu/): Why running AI means buying a graphics card, what makes a GPU different from a CPU, and why the amount of memory on the card is the most important number. - [How a 30-billion-parameter model fits on one card](https://rfriedmann.de/blog/how-a-big-model-fits-on-one-card/): Quantisation in plain terms: how storing each number in a model with fewer bits lets a large model fit on a desktop graphics card, and what it costs. - [How to read a model's name and specs](https://rfriedmann.de/blog/reading-model-specs/): Model names like Qwen3-30B-A3B-Q4_K_M look cryptic at first. How to decode them, and the handful of specs that matter. - [Your first local model, step by step](https://rfriedmann.de/blog/your-first-local-model/): From nothing to a model answering on your own computer: which size to pick, four commands, how to see what it's using, and how to remove it again. - [Open weights, open source, and model licenses](https://rfriedmann.de/blog/open-weights-and-licenses/): A model you can download for free is not automatically free to use for anything. What "open" means for a model, the license families you'll meet, and five questions to ask before a model goes to work. - [Mixture of Experts: how a 30B model runs like a 3B one](https://rfriedmann.de/blog/mixture-of-experts-explained/): The design behind Qwen3-30B-A3B: split the model into many experts, run only a few per token. Why it suits one GPU, and the engineering costs it brings. - [How a model learns: training, in plain words](https://rfriedmann.de/blog/how-a-model-learns/): Every LLM starts as random noise, shaped by one loop: guess, measure the error, nudge billions of dials. How training works, and why it is expensive. - [Pretraining, fine-tuning, and RLHF](https://rfriedmann.de/blog/pretraining-finetuning-rlhf/): A raw trained model can continue text but won't answer you. The three stages that turn it into a helpful assistant, and which one you need. - [How a small model learns from a big one](https://rfriedmann.de/blog/how-small-models-learn-from-big-ones/): Many of the small open models you can run at home were taught by a bigger model, not just by the internet. What distillation is, why it works so well, and what it can't hand down. - [AI that isn't an LLM](https://rfriedmann.de/blog/ai-that-isnt-an-llm/): Language models are one family of AI among several. A tour of the other big families, what each is for, and when an LLM is the wrong tool. - [What an AI agent is](https://rfriedmann.de/blog/what-is-an-ai-agent/): An agent is a language model put in a loop and given tools, so it can carry out actions instead of only describing them. - [How an agent uses tools and memory](https://rfriedmann.de/blog/how-an-agent-uses-tools-and-memory/): The loop, one level down: how the model requests a tool, why its whole memory is just the growing transcript, and why long multi-step tasks fall apart. - [MCP: one plug for every tool an agent uses](https://rfriedmann.de/blog/what-is-mcp/): The Model Context Protocol is how tools, files and business systems plug into an AI assistant. What the pieces are, what a call looks like on the wire, and what an IT team should allow before anyone installs a server. - [Why an LLM miscounts the r's in 'strawberry'](https://rfriedmann.de/blog/why-models-cant-spell/): Ask a top model to count the letters in a word and it often gets it wrong. The cause is not a bug: the model never sees letters at all. A tour of tokens. - [Why you get a different answer every time](https://rfriedmann.de/blog/why-different-answers/): Ask a model the same thing twice and you can get two different replies. That is deliberate weighted sampling, not a fault; one setting controls how much the answers vary. - [Why models make things up](https://rfriedmann.de/blog/why-models-make-things-up/): A model hands you a wrong fact, a fake citation or an invented function with total confidence. Where 'hallucinations' come from, and how to work around them. - [Why a model doesn't know what happened yesterday](https://rfriedmann.de/blog/why-models-dont-know-yesterday/): A model can explain quantum physics and still get today's date wrong. What a knowledge cutoff is, why a model is out of date on the day it ships, and how to work around it. - [What the model remembers: the context window](https://rfriedmann.de/blog/the-context-window/): An LLM has no memory between messages: it re-reads your whole conversation every turn, and only so much fits. Meet the context window. - [Embeddings: how text becomes a list of numbers](https://rfriedmann.de/blog/embeddings-explained/): An embedding turns text into a list of numbers so that similar meaning lands close together. How that powers search and RAG, where it fails, and what the vectors cost to store. - [How a model sees a picture](https://rfriedmann.de/blog/how-models-see-images/): You can hand a modern AI a photo and ask about it. A language model only understands tokens; this is how vision is added to a model that only knew words. - [How an image generator creates a picture from noise](https://rfriedmann.de/blog/how-image-generators-work/): An image generator neither paints nor assembles a collage: the model starts from random noise and removes it step by step, steered by your words. How that works, and why it explains the distorted hands. - [How a model hears and speaks](https://rfriedmann.de/blog/how-models-hear-and-speak/): Dictation, meeting transcripts, voice assistants and cloned voices all rest on one idea: turn sound into a picture, then into tokens. How speech recognition and speech synthesis work, where they fail, and what to watch for at work. - [Why a model thinks before it answers](https://rfriedmann.de/blog/why-models-think-before-answering/): Modern models write pages of working-out before answering. That 'thinking' is computation bought with tokens: what it buys, what it costs, when it's wasted. - [How to write a prompt: clear instructions instead of special phrases](https://rfriedmann.de/blog/how-to-prompt/): Good prompting means clear instructions to a capable, literal-minded assistant with no memory, derived from how the model works. - [What an LLM can and can't do in your infrastructure](https://rfriedmann.de/blog/llms-in-your-infrastructure/): Twenty years of running systems, a couple working with LLMs: where AI helps an ops team, and where it is a liability. - [LLMs in the terminal: a sysadmin's list](https://rfriedmann.de/blog/llms-in-the-terminal/): The concrete, everyday ways an LLM earns its place in a sysadmin's workflow, and the handful of rules that keep it from causing damage. - [Self-hosting an LLM for your team: usually not worth it](https://rfriedmann.de/blog/self-hosting-an-llm-for-your-team/): Why most teams shouldn't run their own model: a small model's error rate costs more working time than the API fees it saves. Plus where it still makes sense. - [How much electricity does an AI answer use?](https://rfriedmann.de/blog/energy-per-ai-answer/): Published figures put one chatbot answer at 0.24 to 0.34 Wh. What they include, why they spread so far, and my own RTX 5090: 0.115 Wh per answer alone, 0.024 Wh shared by 32. - [When not to use AI: an operations view](https://rfriedmann.de/blog/when-not-to-use-ai/): When reaching for an LLM is the wrong choice, and a simple heuristic for telling those cases apart. - [When to let an agent act on its own: an operations view](https://rfriedmann.de/blog/letting-an-agent-loose/): An agent that can act can also act wrongly. After twenty years of running systems: how much autonomy I give one, and which guardrails are worth it. - [Pointing an LLM at your own documents (RAG)](https://rfriedmann.de/blog/your-own-documents/): A common request is 'a ChatGPT that knows our internal docs'. That's RAG: simpler than often presented, and most of the work is in the retrieval half. - [Fine-tuning on your own data, with LoRA](https://rfriedmann.de/blog/fine-tuning-with-lora/): Fine-tuning is the wrong tool for facts and the right one for style, format and narrow tasks. When it pays off, how LoRA makes it fit on one GPU, and what data and checks you need. - [AI and the law: what GDPR and the AI Act ask of your team](https://rfriedmann.de/blog/ai-and-the-law-gdpr-ai-act/): Before staff paste work into a chatbot, two EU laws apply: GDPR for the data in the prompt, the AI Act for what the system is used for. What each asks of an IT team, with the dates as they stand in November 2026. - [Copyright and AI output: who owns what the model writes](https://rfriedmann.de/blog/who-owns-ai-output/): A generated text, image or function: who owns it, whether it can infringe someone else's work, and the status of the data the model learned from. The court cases that matter as of November 2026, and practical rules for a team. - [Prompt injection: the security risk in LLM applications](https://rfriedmann.de/blog/prompt-injection/): The moment your AI reads anything an attacker can influence, that content can hijack it. There is no complete fix, only containment: an operations briefing. - [When a downloaded model is safe to run](https://rfriedmann.de/blog/is-a-downloaded-model-safe/): A model file looks like a set of numbers, and some formats are exactly that. Others run code the moment you load them. Which formats are which, what has already gone wrong, and the checks I'd make before a model touches a work machine. - [Testing a model on your own work](https://rfriedmann.de/blog/testing-a-model-on-your-own-work/): Leaderboards measure performance on public benchmarks. Forty real cases from your own backlog, graded consistently, show how a model performs on your work. ## Optional - [Full text of every post in one file](https://rfriedmann.de/llms-full.txt) - [RSS feed](https://rfriedmann.de/rss.xml) - [imp on GitHub](https://github.com/kekzl/imp)