Notes on CUDA, LLM inference and training on consumer GPUs, and the infrastructure underneath.https://rfriedmann.de/https://rfriedmann.de/blog/continuous-batching/https://rfriedmann.de/blog/continuous-batching/A single decode stream wastes most of the GPU. The fix is to serve many requests at once and let them share each pass over the weights, but only if you stop waiting for the whole batch to finish. Continuous batching, and why it's the throughput trick that matters.Wed, 24 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/prompt-injection/https://rfriedmann.de/blog/prompt-injection/The moment your AI reads anything an attacker can influence, a web page, an email, a document, that content can hijack it. There's no clean fix, only containment. The honest ops briefing on the vulnerability nobody demos.Wed, 24 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/how-models-see-images/https://rfriedmann.de/blog/how-models-see-images/You can hand a modern AI a photo and ask about it. But a language model only understands tokens, so what happens to the image? A plain-words look at how vision gets bolted onto a model that only ever knew words.Tue, 23 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/how-to-prompt/https://rfriedmann.de/blog/how-to-prompt/There are no secret incantations. Good prompting is just clear instructions to a brilliant, literal-minded assistant with no memory. A practical guide that follows straight from how the model works.Tue, 23 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/speculative-decoding/https://rfriedmann.de/blog/speculative-decoding/Decode is bandwidth-bound, so the GPU's maths units sit half-idle one token at a time. Speculative decoding spends that idle compute to verify several guessed tokens at once, for the same output, faster.Mon, 22 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/the-context-window/https://rfriedmann.de/blog/the-context-window/An LLM has no memory between messages. Everything it 'knows' about your conversation is re-read from scratch each turn, and it only fits so much. Meet the context window, the single most useful thing to understand about how chatbots behave.Mon, 22 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/your-own-documents/https://rfriedmann.de/blog/your-own-documents/Everyone wants 'a ChatGPT that knows our internal docs'. The mechanism is called RAG, it's simpler than the hype, and most of the work is the boring retrieval half nobody demos. An ops-eye view.Sun, 21 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/why-models-make-things-up/https://rfriedmann.de/blog/why-models-make-things-up/A model will hand you a wrong fact, a fake citation or an invented function with total confidence. It's not lying and it's not broken. Here's where 'hallucinations' actually come from, and how to work around them.Sat, 20 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/why-different-answers/https://rfriedmann.de/blog/why-different-answers/Ask a model the same thing twice and you can get two different replies. That's a deliberate dice-roll, not a glitch, and one knob controls how loaded the dice are. Meet temperature.Fri, 19 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/why-models-cant-spell/https://rfriedmann.de/blog/why-models-cant-spell/Ask a top model to count the letters in a word and it often gets it wrong. Not a bug, and not stupidity: it's because the model never sees letters at all. A plain-words tour of tokens.Thu, 18 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/axo-learning-without-backprop/https://rfriedmann.de/blog/axo-learning-without-backprop/A from-scratch spiking neural network in C++/CUDA on one RTX 5090 that learns with purely local rules, no backprop and no central optimiser, and grows into a little creature that learns to hunt over a single continuous life.Wed, 17 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/bumping-a-dependency-you-cant-read/https://rfriedmann.de/blog/bumping-a-dependency-you-cant-read/Updating CUTLASS, the GEMM library under imp's 4-bit maths, from v4.5.1 to v4.5.2. In numerical CUDA a bad bump doesn't crash, it quietly returns wrong numbers. So you don't review the change, you make it verifiable.Wed, 17 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/how-an-agent-uses-tools-and-memory/https://rfriedmann.de/blog/how-an-agent-uses-tools-and-memory/The loop, one level down: how the model actually requests a tool, why its whole memory is just the growing transcript, and why long multi-step tasks fall apart.Wed, 17 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/letting-an-agent-loose/https://rfriedmann.de/blog/letting-an-agent-loose/An agent that can act can also act wrongly. After twenty years of running systems, here's how I decide how much rope to give one, and the guardrails that earn their keep.Wed, 17 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/online-softmax-and-flashattention/https://rfriedmann.de/blog/online-softmax-and-flashattention/The deep version of FlashAttention on consumer Blackwell: how online softmax avoids the giant score matrix, and why keeping the output in registers forces raw mma.sync over WMMA.Wed, 17 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/gguf-decode-beats-llama-cpp/https://rfriedmann.de/blog/gguf-decode-beats-llama-cpp/imp decodes dense GGUF 37 to 72% faster than llama.cpp on a 5090. Not because llama.cpp is naive, it isn't, but because imp is built for one exact chip. And the one model where it loses.Tue, 16 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/what-is-an-ai-agent/https://rfriedmann.de/blog/what-is-an-ai-agent/Strip the buzzword and an agent is one simple thing: a language model put in a loop and handed tools, so it can do things instead of just talking about them.Tue, 16 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/ai-that-isnt-an-llm/https://rfriedmann.de/blog/ai-that-isnt-an-llm/Language models get all the attention, but they're one corner of AI. A quick tour of the other big families, what each is for, and how to tell when an LLM is the wrong tool.Mon, 15 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/nvfp4-at-the-bit-level/https://rfriedmann.de/blog/nvfp4-at-the-bit-level/Every other post says '4-bit' and moves on. Here's what NVFP4 actually is, down to the four bits and the two scales, and the two Blackwell instructions (cvt and mma) that make it fly.Mon, 15 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/pretraining-finetuning-rlhf/https://rfriedmann.de/blog/pretraining-finetuning-rlhf/A raw trained model can continue text but won't answer you. The three stages that turn a text-continuer into a helpful assistant, and which one you actually need.Sun, 14 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/when-not-to-use-ai/https://rfriedmann.de/blog/when-not-to-use-ai/The unfashionable half of the conversation. The cases where reaching for an LLM is the wrong choice, and a simple heuristic for telling them apart from the good ones.Sun, 14 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/how-a-model-learns/https://rfriedmann.de/blog/how-a-model-learns/Every LLM starts as random noise and is shaped by one repeated loop: guess, measure the error, nudge billions of dials. Here's how that training actually works, and why it costs a fortune.Sat, 13 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/when-the-model-is-not-a-transformer/https://rfriedmann.de/blog/when-the-model-is-not-a-transformer/Not every LLM is built on attention. Gated DeltaNet and Mamba2 replace the score matrix with a recurrence, which gives constant memory, and one stubborn precision floor.Sat, 13 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/self-hosting-an-llm-for-your-team/https://rfriedmann.de/blog/self-hosting-an-llm-for-your-team/The tempting case for running your own model, and the honest reason most teams shouldn't: a small model's error rate burns more working time than the API fees it saves. Plus the cases where it still wins.Fri, 12 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/optimizing-kernels-consumer-blackwell/https://rfriedmann.de/blog/optimizing-kernels-consumer-blackwell/FlashAttention-2 and NVFP4 GEMM, tuned from scratch for the RTX 5090, and what the profiler taught me when every textbook optimisation turned out to be a red herring.Thu, 11 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/llms-in-the-terminal/https://rfriedmann.de/blog/llms-in-the-terminal/The concrete, everyday ways an LLM actually earns its place in a sysadmin's workflow, and the handful of rules that keep it from causing real damage.Wed, 10 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/writing-cuda-with-an-agent/https://rfriedmann.de/blog/writing-cuda-with-an-agent/Both of my imp posts end with 'every line was written by Claude Code'. This is the question that always follows: how does that actually work, and how do you trust it?Tue, 09 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/llms-in-your-infrastructure/https://rfriedmann.de/blog/llms-in-your-infrastructure/After twenty years of keeping systems running and a couple of years deep in LLMs, an honest split: where AI genuinely helps an ops team, and where it's a liability waiting to happen.Mon, 08 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/serving-30b-models-rtx-5090/https://rfriedmann.de/blog/serving-30b-models-rtx-5090/Why no existing inference engine fully exploits consumer Blackwell, what NVFP4 changes, and the numbers from building one that does.Tue, 02 Jun 2026 00:00:00 GMThttps://rfriedmann.de/blog/what-the-5090-lacks-vs-datacenter/https://rfriedmann.de/blog/what-the-5090-lacks-vs-datacenter/The 5090 and the B200 are both Blackwell, but the consumer chip is missing whole capabilities. Here is what is gone, what it costs, and why you cannot just port data centre code.Tue, 26 May 2026 00:00:00 GMThttps://rfriedmann.de/blog/kv-cache-and-long-context/https://rfriedmann.de/blog/kv-cache-and-long-context/The beginner version calls it short-term memory. The engineering reality: the KV cache, not the weights, is what decides how long your context can get, and what runs you out of memory.Tue, 19 May 2026 00:00:00 GMThttps://rfriedmann.de/blog/mixture-of-experts-explained/https://rfriedmann.de/blog/mixture-of-experts-explained/The trick behind names like Qwen3-30B-A3B: split the model into many experts, run only a few per token. Why it suits a single GPU so well, and the headaches it brings.Tue, 12 May 2026 00:00:00 GMThttps://rfriedmann.de/blog/prefill-vs-decode-roofline/https://rfriedmann.de/blog/prefill-vs-decode-roofline/An inference engine doesn't have one speed, it has two, and they obey opposite laws. The roofline model is the single most useful lens for reasoning about LLM performance.Tue, 05 May 2026 00:00:00 GMThttps://rfriedmann.de/blog/how-imp-works-model-to-token/https://rfriedmann.de/blog/how-imp-works-model-to-token/A plain-language tour of what happens inside an AI engine: how a model gets loaded, and the steps every message runs through to come back as text. Jargon explained as we go.Tue, 28 Apr 2026 00:00:00 GMThttps://rfriedmann.de/blog/reading-model-specs/https://rfriedmann.de/blog/reading-model-specs/Model names like Qwen3-30B-A3B-Q4_K_M look like a cat walked across the keyboard. Here's how to decode them, and the handful of specs that actually matter.Fri, 17 Apr 2026 00:00:00 GMThttps://rfriedmann.de/blog/how-a-big-model-fits-on-one-card/https://rfriedmann.de/blog/how-a-big-model-fits-on-one-card/Quantisation, explained for normal people: how shrinking each number in a model lets a giant fit on a desktop graphics card, and what it costs.Fri, 10 Apr 2026 00:00:00 GMThttps://rfriedmann.de/blog/what-is-a-gpu/https://rfriedmann.de/blog/what-is-a-gpu/Why running AI means buying a graphics card, what makes a GPU different from a CPU, and why the amount of memory on the card is the number that really matters.Fri, 03 Apr 2026 00:00:00 GMThttps://rfriedmann.de/blog/why-run-ai-locally/https://rfriedmann.de/blog/why-run-ai-locally/Cloud chatbots are easy and, honestly, hard to beat. The real and narrower case for running a model yourself, and the big things you give up to do it.Fri, 27 Mar 2026 00:00:00 GMThttps://rfriedmann.de/blog/what-is-an-llm/https://rfriedmann.de/blog/what-is-an-llm/A jargon-free explanation of large language models: what they are, why they're basically a very good autocomplete, and how they write one word at a time.Thu, 19 Mar 2026 00:00:00 GMT