# Pretraining, fine-tuning, and RLHF

> A raw trained model can continue text but won't answer you. The three stages that turn a text-continuer into a helpful assistant, and which one you actually need.

Source: https://rfriedmann.de/blog/pretraining-finetuning-rlhf/
Published: 2026-06-14 · Track: learn · Level: Advanced

The [training loop](/blog/how-a-model-learns/) produces a model that's superb at one
thing: continuing text. That is not the same as being helpful. Drop a real question
into a freshly trained model and you might get this:

<div class="buildup">
<div class="row">You: How do I reset my password?</div>
<div class="row">Base model: <span class="new">How do I change my email? How do I delete my account?</span></div>
</div>

It isn't broken. It learned to *continue* text, and on the open web a question is
often followed by more questions. Turning that into an assistant takes three stages,
each doing a different job.

## Stage 1: Pretraining

This is the [training loop](/blog/how-a-model-learns/) from the last post, run on a
vast pile of general text: guess the next word, over trillions of words, until
grammar, facts, and reasoning patterns settle into the weights.

<aside class="defbox"><span class="tag">In plain words</span><b>Base model:</b> the result of pretraining. It has absorbed how language works and a great deal of world knowledge, but it only knows how to continue text, not how to follow an instruction or hold a conversation.</aside>

Pretraining is where almost all the compute and money go: thousands of GPUs, weeks
to months. It's also the stage almost nobody outside a handful of labs ever does
from scratch. The next two stages are far cheaper, and they're where a base model
becomes the thing you actually talk to.

## Stage 2: Fine-tuning to follow instructions

Now you show the model a much smaller, carefully made set of examples, each one an
instruction paired with a good response:

<div class="buildup">
<div class="row">Instruction: Reset my password.</div>
<div class="row">Good response: <span class="new">Sure. Go to Settings, then Security, and click "Reset password"...</span></div>
</div>

Same training machinery as before, guess, measure, nudge, but now the model is
learning the *shape* of being helpful: when you see an instruction, produce an
answer, not more instructions.

<aside class="defbox"><span class="tag">In plain words</span><b>Fine-tuning (instruction tuning):</b> continuing to train a base model on a smaller, focused dataset to teach it a behaviour or domain. Cheap compared to pretraining, because the heavy lifting is already done; you're adjusting the dials, not setting them from scratch.</aside>

The result is an **instruct** model, which is why model names so often end in
`-Instruct` or `-Chat` ([the suffix that tells you what you're getting](/blog/reading-model-specs/)).
The base version exists too; it's just rarely what you want.

## Stage 3: RLHF, learning what people prefer

Following instructions isn't the same as being *good*. Of two technically correct
answers, one might be clearer, safer, or less likely to confidently make something
up. The final stage tunes for that.

People (and increasingly, other models trained to imitate people) are shown pairs of
responses and asked which is better. Those preferences train the model to lean towards
the kind of answer humans actually wanted.

<aside class="defbox"><span class="tag">In plain words</span><b>RLHF (reinforcement learning from human feedback):</b> instead of one "correct" answer to copy, the model gets a reward signal for producing responses people rate as better, and is nudged to earn more reward. It's how raw helpfulness gets shaped into a consistent, well-behaved tone.</aside>

This is the stage that sands off the rough edges: the refusals on genuinely harmful
requests, the steadier tone, the reduced (never eliminated) tendency to bluff.

## The three stages at a glance

The same three stages, with the bars showing how much data and cost each one demands.

<figure class="chart"><div class="chart-title">Three stages, shrinking cost</div>[diagram omitted — see the page for the chart]<figcaption class="chart-caption">Almost the entire cost lives in the first stage; the two that turn a base model into an assistant are comparatively cheap.</figcaption></figure>

<ol class="pipeline">
<li data-n="1"><h4>Pretraining</h4><p>Trillions of words of general text. Builds knowledge and language. Enormous cost, done rarely. Output: a base model.</p></li>
<li data-n="2"><h4>Fine-tuning</h4><p>Thousands to millions of instruction/response examples. Teaches it to follow instructions. Modest cost. Output: an instruct model.</p></li>
<li data-n="3"><h4>RLHF</h4><p>Human preference comparisons. Aligns tone, helpfulness, and safety. <span class="loop">The polish on top.</span></p></li>
</ol>

## Which stage do you actually need?

This matters the moment someone says "let's fine-tune a model on our data". They
almost never mean stage 1. Pretraining from scratch is a multi-million-dollar
project; you start from someone else's base or instruct model.

<div class="compare">
<div><h4>What teams usually need</h4><ul>
<li class="has">Stage 2 on a small in-house dataset</li>
<li class="has">To nudge tone or teach a niche format</li>
<li class="has">Often: no training at all, just better prompts</li>
</ul></div>
<div><h4>What they almost never need</h4><ul>
<li class="has">Stage 1 pretraining from scratch</li>
<li class="has">To "teach it facts" (that's what [retrieval](/blog/how-an-agent-uses-tools-and-memory/) is for)</li>
<li class="has">A bigger model than the job requires</li>
</ul></div>
</div>

Fine-tuning is the right tool for changing *how* a model responds, not for stuffing
in new facts; for facts, giving it a search tool or documents to read usually beats
retraining. Knowing which stage your problem actually calls for saves a lot of money
and a lot of disappointment, which is the same honest theme as
[self-hosting a model for your team](/blog/self-hosting-an-llm-for-your-team/).