# Why run AI on your own machine?
> Cloud chatbots are easy and, honestly, hard to beat. The real and narrower case for running a model yourself, and the big things you give up to do it.
Source: https://rfriedmann.de/blog/why-run-ai-locally/
Published: 2026-03-27 · Track: learn · Level: Beginner
Every time you ask a cloud chatbot a question, your words travel across the
internet to a company's servers, get answered there, and travel back. It works
beautifully, and I will be blunt up front: for most people, most of the time, it is
very hard to beat. So the honest question isn't "why would anyone bother
running the model themselves," as if the answer were obviously yes. It's narrower:
*when is that trade actually worth it?*
Local AI: it all stays on your machine
[diagram omitted — see the page for the chart]
With a cloud chatbot, your text travels to a company's servers. Run the model yourself and nothing leaves the room you're sitting in. That's the upside. Now the honest downsides.
## What you give up, said plainly
I build [an engine for running models locally](https://github.com/kekzl/imp), so
take it as the opposite of a sales pitch when I say the case against is strong.
**The capability gap is real, and it's not small.** The models the big labs serve
are enormous, and on the hard stuff, deep reasoning, serious coding, long documents,
they are meaningfully better than anything you'll run at home. Matching that locally
is not a graphics card, it is a rack of datacentre GPUs costing tens of thousands of
euros, well past 100k for the top tier. On a single consumer card you run *small to
mid-sized* models. They are genuinely useful, but pretending the gap is a footnote
does no one any favours: for the hardest tasks, the frontier wins by a wide margin.
Put a number on it. SWE-bench Verified asks an AI to fix real, filed GitHub issues,
the kind of work people actually want help with:
SWE-bench Verified: share of real GitHub issues the AI resolves
[diagram omitted — see the page for the chart]
The bottom two bars are the same open model, Qwen2.5-Coder-32B, the sort that genuinely fits on a single card. Raw, it fixes a handful of issues; with heavy custom scaffolding it climbs to the middle, and still trails the hosted frontier. Exact numbers move every month and depend on the harness; the gap is the point.
The leap from that bottom bar to the middle one isn't a bigger model, it's
specialised scaffolding wrapped around the same weights, which is exactly the kind
of thing the hosted products ship for you. Which brings us to the second gap.
**It's not just the model, it's the whole toolkit.** This is the part people miss.
Anthropic, Google, OpenAI and the rest don't ship you a model, they ship a platform
wrapped around it: web search, [tool use](/blog/what-is-an-ai-agent/), code
execution, huge context windows, images and audio, retrieval over your files, and a
model they quietly improve every few weeks. Run a model at home and you get the raw
engine. The workshop around it, the part that makes the hosted product feel like
magic, you build and maintain yourself.
**"Pay once" isn't the whole bill.** A card you own looks free per question, but
the honest cost includes the hardware up front, the electricity, and your own time
to set it up and keep it running. For light or occasional use, the cloud's
per-question pricing is simply cheaper, and someone else runs the operations.
## So when is local actually the right call?
Given all that, local AI earns its place for *specific* reasons, not as a general
replacement:
- **Privacy you can't compromise.** Not "I'd rather not share this," but "this data
is legally not allowed to leave," health records, legal files, regulated or
secret material. When that's the rule, local stops being a preference and becomes
the only option.
- **Offline or air-gapped.** A plane, a ship, a secure facility, a flaky connection.
No server to reach means the model has to be in the room.
- **High, steady volume.** If you'd run the meter all day every day, a fixed-cost
card you already own can beat paying per question, once you're past the break-even.
- **Control and stability.** The model won't change tone, get more restrictive, or
get retired underneath you. It's yours, frozen until you decide otherwise.
- **Learning and building.** If you want to actually understand the stack, or build
on top of it, there's no substitute for running it yourself.
Cloud chatbot
The largest, most capable models
A full toolkit: search, tools, files, multimodal
Nothing to set up or maintain
Your text leaves your machine
Pay per use, forever; can change under you
Your own GPU
Private: data never leaves, works offline
Fixed cost, yours to control and customise
Smaller, less capable models than the frontier
You build the toolkit around it yourself
Real hardware, power and time costs
## The honest verdict
For most people doing everyday things, the hosted frontier models are the right
answer, by a wide margin. Reach for local when you have a concrete reason from the
list above, and the same sober reasoning applies twice over the moment a whole team
relies on it, which is its own post:
[self-hosting an LLM for your team, and why you usually shouldn't](/blog/self-hosting-an-llm-for-your-team/).
Where does that leave [imp](https://github.com/kekzl/imp)? Its point was never to
beat Claude or Gemini at being a chatbot. It's to make the cases where local is the
right call, privacy, scale, offline, control, genuinely viable on a card you can
actually afford, instead of a poor compromise you put up with. That is a narrower
claim than "run everything at home", and it is the accurate one. If you want to see
how far a single consumer card can be pushed, start with
[what an LLM actually is](/blog/what-is-an-llm/), then
[the speed numbers](/blog/serving-30b-models-rtx-5090/).