# Self-hosting an LLM for your team: usually don't

> The tempting case for running your own model, and the honest reason most teams shouldn't: a small model's error rate burns more working time than the API fees it saves. Plus the cases where it still wins.

Source: https://rfriedmann.de/blog/self-hosting-an-llm-for-your-team/
Published: 2026-06-12 · Track: learn · Level: Ops

Running your own model sounds great on paper. Your data stays in-house, no per-seat
bill, total control. It lands on the infrastructure person's desk as an obvious
win, and I'm the infrastructure person, so I've been asked plenty of times. Here's
the honest answer the pitches skip: for most teams, right now, it doesn't pay. And
the reason isn't the hardware or the ops work. It's the model.

## The catch that beats every other argument

[The model matters enormously](/blog/llms-in-your-infrastructure/), and the model
you can self-host on a sensible budget is not in the same league as a frontier model
like Claude Opus running in someone's data centre. That gap is what matters most,
because [you have to verify the output either way](/blog/when-not-to-use-ai/).
Every wrong answer is time spent catching it, redoing it, or debugging what it
quietly broke.

So the real cost of a weaker model isn't the licence you saved, it's the salary
you burn. A model that's wrong twice as often doesn't cost twice as much to run, it
costs a multiple of that in your team's time. Cheap hardware buys a small model,
which buys a high error rate, which buys expensive humans cleaning up after it. The API fee you
avoided is usually the smallest number in the equation, and the one everyone
fixates on.

That's why, today, a frontier API beats self-hosting for most teams on the only
thing that really matters: good answers with less wasted effort.

## When it still makes sense

It's not never. There are real cases where self-hosting wins, and they share a
shape: either the data legally can't leave, or the task is narrow enough that a
small model is genuinely good enough, or you're willing to spend serious money.

- **Hard privacy or regulatory walls.** If the data legally cannot leave your
  network, you pay the quality tax on purpose, because the alternative isn't a
  better model, it's a compliance problem. [Privacy was the strongest case for
  local in the first place](/blog/why-run-ai-locally/).
- **Narrow, well-defined, high-volume work.** Classification, extraction, tagging,
  autocomplete, bulk-processing similar items. A smaller model can be genuinely
  good enough here, you'd gate and verify it anyway, and the volume is where
  per-call API pricing starts to hurt. This is where local actually shines.
- **Offline or air-gapped.** No internet, no API. Then self-hosting isn't a
  preference, it's the only option.
- **You're ready to invest a lot.** Multiple high-end GPUs, the biggest open
  models, real ops headcount. Spend enough and the quality gap narrows. Most teams
  asking "should we self-host to save money" are not in this bucket, and that is
  itself the answer.

If none of those describe you, the honest call is: use a frontier API, and revisit
in six months, because open models are improving fast.

The money side follows the same logic. Cloud pricing scales with how much you use it,
while a card you bought is a fixed cost no matter how hard you push it, so the two
only cross once your volume is high and steady.

<figure class="chart"><div class="chart-title">When self-hosting pays off</div>[diagram omitted — see the page for the chart]<figcaption class="chart-caption">Only past a steady, high volume does the fixed cost of a local card beat the per-use pricing of a cloud API.</figcaption></figure>

## If you've decided it's worth it

Say one of those cases does fit. Then the practical reality, briefly.

**Sizing.** The number that decides everything is [VRAM](/blog/what-is-a-gpu/): the
weights plus the working memory have to fit on the card. A single 32 GB card runs a
[30B-class model in 4-bit](/blog/serving-30b-models-rtx-5090/) comfortably for one
user. The catch is concurrency, a dozen people hitting it at once share that
throughput, which is a [different performance regime](/blog/prefill-vs-decode-roofline/)
than serving one.

**Serving.** Run it behind an OpenAI-compatible API and every existing tool just
points at your box instead of a provider. No app changes.

**The ops burden.** It's a service you now operate, not an app you installed: a
single point of failure until you make it not one, a model to keep upgrading,
observability to add, and a clear owner so it doesn't become shadow IT. None of it
is a dealbreaker; it's just the cost of admission.

## The honest bottom line

For most teams, today, a frontier model in the cloud is so far ahead of anything
you can self-host on a reasonable budget that running your own rarely makes sense.
Self-host when the data can't leave, when the task is narrow enough that small is
fine, or when you're genuinely ready to spend real money. Otherwise the error rate
of a cheap model will quietly cost you more than the API ever would. That balance will
shift as open models catch up. It just hasn't yet. And when it does, self-hosting stops being a compromise and becomes the obvious choice, which is the whole reason I work on it.