# Fine-tuning on your own data, with LoRA
> Fine-tuning is the wrong tool for facts and the right one for style, format and narrow tasks. When it pays off, how LoRA makes it fit on one GPU, and what data and checks you need.
Source: https://rfriedmann.de/blog/fine-tuning-with-lora/
Published: 2026-11-08 · Track: learn · Level: Ops
The [lesson on your own documents](/blog/your-own-documents/) ended with a firm "no":
don't fine-tune a model to teach it your facts, retrieve them instead. That advice
stands. But "never fine-tune" is not the lesson either. There is a class of problems
where training really is the right move, and it no longer needs a cluster: the QLoRA
paper trained its second-best chat model in under 12 hours on a single consumer GPU.
This lesson is about recognising that class, and measuring whether the training worked.
## What fine-tuning changes, and what it doesn't
A model learns almost everything it knows in [pretraining](/blog/pretraining-finetuning-rlhf/).
The LIMA paper (Zhou et al., 2023) made the point sharply: a 65B model, tuned on only
1,000 carefully chosen examples, learned to answer like a polished assistant. The
authors' reading: knowledge comes from pretraining, and tuning mostly teaches *which
format and style to use*.
That is exactly the line to draw at work. Fine-tuning is good at *how*, weak at *what*.
Fine-tune for
A house style: tone, length, the way your team writes tickets
A fixed output shape the model keeps getting slightly wrong
Domain jargon and abbreviations it misreads
A small model doing one narrow task as well as a big one
The last row on the left deserves a word. A small model tuned on the answers of a
big one, for one job only, is the cheapest serious use of fine-tuning, and it is
[distillation](/blog/how-small-models-learn-from-big-ones/) in practice: classify
tickets, extract fields from invoices, rewrite log lines into a summary. A 4B model
that does one thing well runs on hardware a 70B model never will. If the big model is a
commercial API, check its terms first: some forbid training competing models on its
output.
## The ladder: climb only when the step below fails
1. **Prompt.** Write the instructions and three good examples into the prompt
([how to prompt](/blog/how-to-prompt/)). Costs minutes. Often sufficient on its own.
2. **Retrieve.** If the model lacks *information*, give it the text at question time.
3. **Constrain.** If the problem is a strict format like JSON, the server can force
valid output at generation time ([constrained decoding](/blog/constrained-decoding-valid-json/)),
no training needed.
4. **Fine-tune.** Only when the model has the information and the format rules, and
still doesn't behave the way you need, across hundreds of cases.
Each rung up costs more to build and more to maintain. A tuned model is a new
artefact: it needs versioning, re-testing when the base model updates, and someone
who remembers how it was made.
## LoRA: train a small add-on, not the model
Training all weights of a model is expensive mostly because of memory. With the
usual Adam optimizer in mixed precision, each parameter costs about 16 bytes during
training: weights, gradients and optimizer state (the ZeRO paper's accounting). For
an 8B model, that is an estimate of 8 billion x 16 bytes = 128 GB, before activations.
Not a one-GPU job.
One weight matrix with a LoRA adapter (rank 16)
[diagram omitted — see the page for the chart]
Every input runs through the frozen matrix as before, plus a thin detour through A and B. Training only touches the detour. Afterwards A x B can be added into W once, so the tuned model runs exactly as fast as the original.
The numbers from the original paper (Hu et al., 2021): on GPT-3 175B, LoRA cut the
trainable parameters by 10,000 times and the training memory by up to 3 times, with
quality on par with full fine-tuning in their tests. Because the correction can be
merged into the weights, it adds no latency at inference.
**QLoRA** (Dettmers et al., 2023) went one step further: store the frozen base model
in 4 bits and train the adapters on top. The paper fine-tuned a 65B model on a single
48 GB GPU, where full 16-bit fine-tuning needed more than 780 GB.
Here is what that means for a typical 8B model (32 layers, width 4,096, MLP width
14,336), LoRA rank 16 on all seven weight matrices per layer. These are estimates,
derived, not measured:
| Item | Arithmetic | Estimate |
|---|---|---|
| Adapter size | 1,310,720 per layer x 32 layers | 42 M parameters, 0.5 % of 8B |
| Adapter file | 42 M x 2 bytes | 84 MB |
| Frozen base in 4 bits | 8 B x 0.5 bytes | 4 GB |
| Adapter training state | 42 M x 16 bytes | 0.7 GB |
| Full fine-tune, for comparison | 8 B x 16 bytes | 128 GB |
Activations come on top and grow with batch size and sequence length, so plan with
headroom. Still: a job that needs a rack in full fine-tuning fits on a single
consumer card with QLoRA.
## The data is the project
As with RAG, the model part is the easy part. The work is the examples.
- **Quality over quantity.** Hundreds to a few thousand examples are a common range
for a narrow task. LIMA got there with 1,000. Every bad example teaches the bad
behaviour too, so one careful person reviewing 500 examples beats 50,000 scraped ones.
- **Show the real input.** Use the messy tickets, the real invoices, the actual log
lines, not tidy samples written for the occasion.
- **Hold back a test set.** Put 10 to 20 % aside before training and never train on it.
- **Mind what's in it.** Training data with names, customer data or HR content ends
up baked into the adapter. Treat the adapter like the data it learned from, and read
[the lesson on data protection](/blog/ai-and-the-law-gdpr-ai-act/) first.
## Measure before and after
A fine-tune that "feels better" is not a result. Run the base model and the tuned
model on the same held-back test set with the same scoring, as in [testing a model
on your own work](/blog/testing-a-model-on-your-own-work/). Check two things: did the
target task improve, and did anything else get worse? Narrow training can make a
model noticeably worse at the things you didn't train, so keep a handful of general
questions in the test set as a canary.
## Licence and serving
The adapter is only half a model. It runs on one specific base model and inherits its
licence: whatever the base licence says about commercial use, naming or attribution
still applies to your tuned version. Check it before you start, as in [open weights
and licenses](/blog/open-weights-and-licenses/).
For serving there are two options. Merge the adapter into the weights and ship one
ordinary model. Or keep base and adapters separate: servers such as vLLM and llama.cpp
can load LoRA adapters on top of a base model, so one GPU can host several tuned
variants, one per task, for the memory cost of one base plus a few small files.
## Recommendation
Fine-tune last, not first. Climb the ladder: prompt, retrieve, constrain, and only
then train. When you do train, use LoRA or QLoRA on an open model, spend the effort
on a few hundred excellent examples, and measure against the base model on a test
set you never trained on. Done that way, fine-tuning is a modest, repeatable
engineering task. Done as "let's train it on our wiki", it is expensive and
produces a model that states wrong facts confidently, in your house style.