Why run AI on your own machine?

Every time you ask a cloud chatbot a question, your words travel across the internet to a company’s servers, get answered there, and travel back. It works beautifully, and I will be blunt up front: for most people, most of the time, it is very hard to beat. So the honest question isn’t “why would anyone bother running the model themselves,” as if the answer were obviously yes. It’s narrower: when is that trade actually worth it?

What you give up, said plainly

I build an engine for running models locally, so take it as the opposite of a sales pitch when I say the case against is strong.

The capability gap is real, and it has narrowed. The models the big labs serve are enormous, and on the hard stuff, deep reasoning, serious coding, long documents, they are still better than anything you’ll run at home. Matching the top of the frontier locally is not a graphics card, it is a rack of datacentre GPUs costing tens of thousands of euros, well past 100k for the top tier. On a single consumer card you run small to mid-sized models. Over 2026 those became genuinely good, good enough that the best of them now sits level with the mid-tier hosted models. But the top of the frontier is still a clear step ahead, and pretending otherwise does no one any favours.

Put a number on it. SWE-bench Pro asks an AI to fix real, filed GitHub issues, the kind of work people actually want help with. (Its older sibling, SWE-bench Verified, is what this chart showed in March. The frontier now scores 95% and above on it, and none of the three big labs put it in their latest launch tables, so it no longer separates anything.)

Now the fine print on that grey bar, because it matters. Qwen’s 61.7% is measured with Claude Code driving the model, Anthropic’s own coding agent wrapped around Qwen’s weights. The weights alone don’t fix issues, the harness does the driving, and its quality moves the score a long way: one researcher took the previous generation, Qwen3.6-27B, from its official 77% on SWE-bench Verified to about 90% without changing a single weight, purely by engineering the agent stack around it. That stack is exactly the kind of thing the hosted products ship for you. Which brings us to the second gap.

It’s not just the model, it’s the whole toolkit. This is the part people miss. Anthropic, Google, OpenAI and the rest don’t ship you a model, they ship a platform wrapped around it: web search, tool use, code execution, huge context windows, images and audio, retrieval over your files, and a model they quietly improve every few weeks. Run a model at home and you get the raw engine. The workshop around it, the part that makes the hosted product feel like magic, you build and maintain yourself.

“Pay once” isn’t the whole bill. A card you own looks free per question, but the honest cost includes the hardware up front, the electricity, and your own time to set it up and keep it running. For light or occasional use, the cloud’s per-question pricing is simply cheaper, and someone else runs the operations.

So when is local actually the right call?

Given all that, local AI earns its place for specific reasons, not as a general replacement:

• Privacy you can’t compromise. Not “I’d rather not share this,” but “this data is legally not allowed to leave,” health records, legal files, regulated or secret material. When that’s the rule, local stops being a preference and becomes the only option.
• Offline or air-gapped. A plane, a ship, a secure facility, a flaky connection. No server to reach means the model has to be in the room.
• High, steady volume. If you’d run the meter all day every day, a fixed-cost card you already own can beat paying per question, once you’re past the break-even.
• Control and stability. The model won’t change tone, get more restrictive, or get retired underneath you. It’s yours, frozen until you decide otherwise.
• Learning and building. If you want to actually understand the stack, or build on top of it, there’s no substitute for running it yourself.

The honest verdict

For most people doing everyday things, the hosted frontier models are the right answer, by a wide margin. Reach for local when you have a concrete reason from the list above, and the same sober reasoning applies twice over the moment a whole team relies on it, which is its own post: self-hosting an LLM for your team, and why you usually shouldn’t.

Where does that leave imp? Its point was never to beat Claude or Gemini at being a chatbot. It’s to make the cases where local is the right call, privacy, scale, offline, control, genuinely viable on a card you can actually afford, instead of a poor compromise you put up with. That is a narrower claim than “run everything at home”, and it is the accurate one. If you want to see how far a single consumer card can be pushed, start with what an LLM actually is, then the speed numbers.