Raphael Friedmann

Raphael Friedmann, AI infrastructure, Karlsruhe

How fast can one consumer card serve modern AI?

Fast enough that the answer needed its own engine.

SEEMsignexpexpmantissaFP8 scaleper 16 valuesFP32 scaleper tensor
E2M1, the card's native 4-bit number: one sign, two exponent, one mantissa, sixteen values. The range comes from the two scales. Every cell on this site is one of these. NVFP4 at the bit level

I started in C++ twenty years ago, mostly import and export interfaces and small tools, and even then I wanted to push the hardware to its limits. Most of my career since went into keeping production systems running. imp is my answer to the question above: an inference engine built from scratch for one RTX 5090. I write about it two ways: an engineering log with the details and numbers, and a course for anyone who wants to understand how AI runs on a GPU.

  • 6 of 6 GGUF models decode faster single-stream than on llama.cpp, up to +141%
  • +16 to +76% more throughput than vLLM at 8 and 32 streams, Qwen3.8-27B
  • 126 432 tokens of context for a 27B model, weights and KV cache in NVFP4 on 32 GB
Start the courseThe engineering logimp on GitHub ↗

Latest writing

All posts: the log and understanding AI

The newest from both tracks, freshest first. Each grid is the post's name hashed into 4-bit numbers.

Understanding AIBeginner10 Oct 2026, 7 min read

How a small model learns from a big one

Many of the small open models you can run at home were taught by a bigger model, not just by the internet. What distillation is, why it works so well, and what it can't hand down.

How imp runs Qwen3.8-Flash-Next on a 32 GB card

56 GiB of experts, 32 GB of VRAM: all experts on the host, a GPU-managed cache in VRAM, and decode from 5.8 to 75-80 tokens per second on real text.The logExpert27 Sep 2026, 14 min

Making a model emit valid JSON, every time

Asking nicely gets you valid JSON almost always, and almost is a broken integration. The fix is in the sampler: delete every invalid token before you sample.The logExpert16 Sep 2026, 6 min

Testing a model on your own work

Leaderboards tell you a model is good at leaderboards. Forty real cases from your own backlog, graded honestly, answer the question you actually have.Understanding AIOps15 Sep 2026, 6 min

Why a model thinks before it answers

Modern models write pages of working-out before answering. That 'thinking' is computation bought with tokens: what it buys, what it costs, when it's wasted.Understanding AIBeginner14 Sep 2026, 6 min

The oracle you build yourself

Dependabot opened five PRs against this site, three of them major. No test suite, so I built the check: render the site with both versions and diff it.The logOps11 Aug 2026, 9 min

imp, measured on one RTX 5090

Every figure is dated and commit-anchored in BENCHMARKS.md ↗

The question, answered in code: one engine, built from scratch, on the consumer card's native 4-bit tensor cores that the big engines leave idle here. The numbers below are the engine's own, with the command to reproduce each.

imp↗

A from-scratch C++23/CUDA engine that runs large language models on a single NVIDIA RTX 5090. Single-stream latency is its axis: one person, one card, one answer coming back as fast as the silicon allows. It speaks the OpenAI and the Anthropic API natively, so an agent stack written against either runs without a shim. Written entirely with Claude Code.

  • ~154k lines C++23/CUDA
  • NVFP4 4-bit native
  • OpenAI and Anthropic API
  • CUDA graphs, continuous batching
  • v0.34.0, 2 Sep 2026
  • MIT licensed

Qwen3.8-27B on one card

The README's worked example: a 27B multimodal model, quantized to NVFP4 with imp's own quantizer so it fits the 5090 with room for a real context. The same checkpoint loads in vLLM, which is what makes the comparison below fair.

Single-stream decode102 tok/s128 tokens, temperature 0, imp-cli, 2026-08-31
Context on 32 GB126 432 tokensweights and KV cache in NVFP4; 8-bit KV stops at 86 848
Decode at a 77k prompt74 tok/s100 with sparse decode on, batch 8, 2026-08-30
32 streams1 834 tok/saggregate, 300-token generations, 2026-09-02
Prefill11 730 tok/spp4096, two runs 11 730 and 11 740, v0.34.0
On the card18.3 GiB19.2 GiB download, about 7 GiB left for the KV cache
Serving throughput, imp against vLLM, Qwen3.8-27B in NVFP4
imp v0.34.0vLLM, same checkpoint
8 streams36-tok prompts, vLLM 0.27.1imp v0.34.0: 573.0 tok/s573.0vLLM, same checkpoint: 495.8 tok/s495.8+16%32 streams38-tok prompts, vLLM 0.27.1imp v0.34.0: 1807.9 tok/s1807.9vLLM, same checkpoint: 1447.8 tok/s1447.8+25%32 streams38-tok prompts, vLLM 0.28.0imp v0.34.0: 1833.8 tok/s1833.8vLLM, same checkpoint: 1410.7 tok/s1410.7+30%32 streams1082-tok prompts, vLLM 0.27.1imp v0.34.0: 873.4 tok/s873.4vLLM, same checkpoint: 497.8 tok/s497.8+75%0500100015002000tok/s
Aggregate tokens per second, both engines reading the same 19.2 GiB compressed-tensors file, one client for both, fresh server per arm, three alternating trials, imp ahead in every pair. Measured 2026-09-02. vLLM has no native FP4 kernel on this chip and falls back to Marlin, which its own startup log says; this is a consumer-Blackwell result, not a statement about vLLM on the hardware it targets. Reproduce with tools/analysis/vllm_conc_ab.sh.
Single-stream decode, imp against llama.cpp, six GGUF models
imp v0.33.0llama.cpp, image pinned by digest
Qwen3-8BQ8_0imp v0.33.0: 385.4 tok/s385.4llama.cpp, image pinned by digest: 160.1 tok/s160.1+141%Qwen3-14BQ6_Kimp v0.33.0: 162.5 tok/s162.5llama.cpp, image pinned by digest: 114.8 tok/s114.8+42%Qwen3.6-35B-A3BUD-Q4_K_Mimp v0.33.0: 287.9 tok/s287.9llama.cpp, image pinned by digest: 235.8 tok/s235.8+22%gpt-oss-20bMXFP4imp v0.33.0: 382.7 tok/s382.7llama.cpp, image pinned by digest: 335.9 tok/s335.9+14%Gemma-4-26B-A4BUD-Q4_K_Mimp v0.33.0: 245.0 tok/s245.0llama.cpp, image pinned by digest: 214.4 tok/s214.4+14%Qwen3-30B-A3BQ4_K_Mimp v0.33.0: 305.5 tok/s305.5llama.cpp, image pinned by digest: 295.7 tok/s295.7+3%0100200300400tok/s
Tokens per second, single stream, the same GGUF file and flags on both sides, measured 2026-08-30. The bottom row is the honest end of the range: a MoE that is already bandwidth-bound at batch 1 leaves little to win. imp's default n-gram speculation is most of the Qwen3-8B figure; with it off that row reads 285, still +78%. Reproduce with make bench-competitive.
Also tinkering: axo, a neuromorphic learner that trains without backprop

From the labs

Official feeds, rebuilt daily: all AI news

Other people's announcements, straight from the source and unranked. Everything above is mine.