Why run AI on your own machine?
Cloud chatbots are easy and, honestly, hard to beat. The real and narrower case for running a model yourself, and the big things you give up to do it.
Every time you ask a cloud chatbot a question, your words travel across the internet to a company’s servers, get answered there, and travel back. It works beautifully, and I will be blunt up front: for most people, most of the time, it is very hard to beat. So the honest question isn’t “why would anyone bother running the model themselves,” as if the answer were obviously yes. It’s narrower: when is that trade actually worth it?
What you give up, said plainly
I build an engine for running models locally, so take it as the opposite of a sales pitch when I say the case against is strong.
The capability gap is real, and it’s not small. The models the big labs serve are enormous, and on the hard stuff, deep reasoning, serious coding, long documents, they are meaningfully better than anything you’ll run at home. Matching that locally is not a graphics card, it is a rack of datacentre GPUs costing tens of thousands of euros, well past 100k for the top tier. On a single consumer card you run small to mid-sized models. They are genuinely useful, but pretending the gap is a footnote does no one any favours: for the hardest tasks, the frontier wins by a wide margin.
Put a number on it. SWE-bench Verified asks an AI to fix real, filed GitHub issues, the kind of work people actually want help with:
The leap from that bottom bar to the middle one isn’t a bigger model, it’s specialised scaffolding wrapped around the same weights, which is exactly the kind of thing the hosted products ship for you. Which brings us to the second gap.
It’s not just the model, it’s the whole toolkit. This is the part people miss. Anthropic, Google, OpenAI and the rest don’t ship you a model, they ship a platform wrapped around it: web search, tool use, code execution, huge context windows, images and audio, retrieval over your files, and a model they quietly improve every few weeks. Run a model at home and you get the raw engine. The workshop around it, the part that makes the hosted product feel like magic, you build and maintain yourself.
“Pay once” isn’t the whole bill. A card you own looks free per question, but the honest cost includes the hardware up front, the electricity, and your own time to set it up and keep it running. For light or occasional use, the cloud’s per-question pricing is simply cheaper, and someone else runs the operations.
So when is local actually the right call?
Given all that, local AI earns its place for specific reasons, not as a general replacement:
- Privacy you can’t compromise. Not “I’d rather not share this,” but “this data is legally not allowed to leave,” health records, legal files, regulated or secret material. When that’s the rule, local stops being a preference and becomes the only option.
- Offline or air-gapped. A plane, a ship, a secure facility, a flaky connection. No server to reach means the model has to be in the room.
- High, steady volume. If you’d run the meter all day every day, a fixed-cost card you already own can beat paying per question, once you’re past the break-even.
- Control and stability. The model won’t change tone, get more restrictive, or get retired underneath you. It’s yours, frozen until you decide otherwise.
- Learning and building. If you want to actually understand the stack, or build on top of it, there’s no substitute for running it yourself.
Cloud chatbot
- The largest, most capable models
- A full toolkit: search, tools, files, multimodal
- Nothing to set up or maintain
- Your text leaves your machine
- Pay per use, forever; can change under you
Your own GPU
- Private: data never leaves, works offline
- Fixed cost, yours to control and customise
- Smaller, less capable models than the frontier
- You build the toolkit around it yourself
- Real hardware, power and time costs
The honest verdict
For most people doing everyday things, the hosted frontier models are the right answer, by a wide margin. Reach for local when you have a concrete reason from the list above, and the same sober reasoning applies twice over the moment a whole team relies on it, which is its own post: self-hosting an LLM for your team, and why you usually shouldn’t.
Where does that leave imp? Its point was never to beat Claude or Gemini at being a chatbot. It’s to make the cases where local is the right call, privacy, scale, offline, control, genuinely viable on a card you can actually afford, instead of a poor compromise you put up with. That is a narrower claim than “run everything at home”, and it is the accurate one. If you want to see how far a single consumer card can be pushed, start with what an LLM actually is, then the speed numbers.