Pretraining, fine-tuning, and RLHF
A raw trained model can continue text but won't answer you. The three stages that turn a text-continuer into a helpful assistant, and which one you actually need.
The training loop produces a model that’s superb at one thing: continuing text. That is not the same as being helpful. Drop a real question into a freshly trained model and you might get this:
It isn’t broken. It learned to continue text, and on the open web a question is often followed by more questions. Turning that into an assistant takes three stages, each doing a different job.
Stage 1: Pretraining
This is the training loop from the last post, run on a vast pile of general text: guess the next word, over trillions of words, until grammar, facts, and reasoning patterns settle into the weights.
Pretraining is where almost all the compute and money go: thousands of GPUs, weeks to months. It’s also the stage almost nobody outside a handful of labs ever does from scratch. The next two stages are far cheaper, and they’re where a base model becomes the thing you actually talk to.
Stage 2: Fine-tuning to follow instructions
Now you show the model a much smaller, carefully made set of examples, each one an instruction paired with a good response:
Same training machinery as before, guess, measure, nudge, but now the model is learning the shape of being helpful: when you see an instruction, produce an answer, not more instructions.
The result is an instruct model, which is why model names so often end in
-Instruct or -Chat (the suffix that tells you what you’re getting).
The base version exists too; it’s just rarely what you want.
Stage 3: RLHF, learning what people prefer
Following instructions isn’t the same as being good. Of two technically correct answers, one might be clearer, safer, or less likely to confidently make something up. The final stage tunes for that.
People (and increasingly, other models trained to imitate people) are shown pairs of responses and asked which is better. Those preferences train the model to lean towards the kind of answer humans actually wanted.
This is the stage that sands off the rough edges: the refusals on genuinely harmful requests, the steadier tone, the reduced (never eliminated) tendency to bluff.
The three stages at a glance
The same three stages, with the bars showing how much data and cost each one demands.
Pretraining
Trillions of words of general text. Builds knowledge and language. Enormous cost, done rarely. Output: a base model.
Fine-tuning
Thousands to millions of instruction/response examples. Teaches it to follow instructions. Modest cost. Output: an instruct model.
RLHF
Human preference comparisons. Aligns tone, helpfulness, and safety. The polish on top.
Which stage do you actually need?
This matters the moment someone says “let’s fine-tune a model on our data”. They almost never mean stage 1. Pretraining from scratch is a multi-million-dollar project; you start from someone else’s base or instruct model.
What teams usually need
- Stage 2 on a small in-house dataset
- To nudge tone or teach a niche format
- Often: no training at all, just better prompts
What they almost never need
- Stage 1 pretraining from scratch
- To "teach it facts" (that's what [retrieval](/blog/how-an-agent-uses-tools-and-memory/) is for)
- A bigger model than the job requires
Fine-tuning is the right tool for changing how a model responds, not for stuffing in new facts; for facts, giving it a search tool or documents to read usually beats retraining. Knowing which stage your problem actually calls for saves a lot of money and a lot of disappointment, which is the same honest theme as self-hosting a model for your team.