# Fine-tuning on your own data, with LoRA > Fine-tuning is the wrong tool for facts and the right one for style, format and narrow tasks. When it pays off, how LoRA makes it fit on one GPU, and what data and checks you need. Source: https://rfriedmann.de/blog/fine-tuning-with-lora/ Published: 2026-11-08 · Track: learn · Level: Ops The [lesson on your own documents](/blog/your-own-documents/) ended with a firm "no": don't fine-tune a model to teach it your facts, retrieve them instead. That advice stands. But "never fine-tune" is not the lesson either. There is a class of problems where training really is the right move, and it no longer needs a cluster: the QLoRA paper trained its second-best chat model in under 12 hours on a single consumer GPU. This lesson is about recognising that class, and measuring whether the training worked. ## What fine-tuning changes, and what it doesn't A model learns almost everything it knows in [pretraining](/blog/pretraining-finetuning-rlhf/). The LIMA paper (Zhou et al., 2023) made the point sharply: a 65B model, tuned on only 1,000 carefully chosen examples, learned to answer like a polished assistant. The authors' reading: knowledge comes from pretraining, and tuning mostly teaches *which format and style to use*. That is exactly the line to draw at work. Fine-tuning is good at *how*, weak at *what*.

Fine-tune for

Don't fine-tune for

The last row on the left deserves a word. A small model tuned on the answers of a big one, for one job only, is the cheapest serious use of fine-tuning, and it is [distillation](/blog/how-small-models-learn-from-big-ones/) in practice: classify tickets, extract fields from invoices, rewrite log lines into a summary. A 4B model that does one thing well runs on hardware a 70B model never will. If the big model is a commercial API, check its terms first: some forbid training competing models on its output. ## The ladder: climb only when the step below fails 1. **Prompt.** Write the instructions and three good examples into the prompt ([how to prompt](/blog/how-to-prompt/)). Costs minutes. Often sufficient on its own. 2. **Retrieve.** If the model lacks *information*, give it the text at question time. 3. **Constrain.** If the problem is a strict format like JSON, the server can force valid output at generation time ([constrained decoding](/blog/constrained-decoding-valid-json/)), no training needed. 4. **Fine-tune.** Only when the model has the information and the format rules, and still doesn't behave the way you need, across hundreds of cases. Each rung up costs more to build and more to maintain. A tuned model is a new artefact: it needs versioning, re-testing when the base model updates, and someone who remembers how it was made. ## LoRA: train a small add-on, not the model Training all weights of a model is expensive mostly because of memory. With the usual Adam optimizer in mixed precision, each parameter costs about 16 bytes during training: weights, gradients and optimizer state (the ZeRO paper's accounting). For an 8B model, that is an estimate of 8 billion x 16 bytes = 128 GB, before activations. Not a one-GPU job.
One weight matrix with a LoRA adapter (rank 16)
[diagram omitted — see the page for the chart]
Every input runs through the frozen matrix as before, plus a thin detour through A and B. Training only touches the detour. Afterwards A x B can be added into W once, so the tuned model runs exactly as fast as the original.
The numbers from the original paper (Hu et al., 2021): on GPT-3 175B, LoRA cut the trainable parameters by 10,000 times and the training memory by up to 3 times, with quality on par with full fine-tuning in their tests. Because the correction can be merged into the weights, it adds no latency at inference. **QLoRA** (Dettmers et al., 2023) went one step further: store the frozen base model in 4 bits and train the adapters on top. The paper fine-tuned a 65B model on a single 48 GB GPU, where full 16-bit fine-tuning needed more than 780 GB. Here is what that means for a typical 8B model (32 layers, width 4,096, MLP width 14,336), LoRA rank 16 on all seven weight matrices per layer. These are estimates, derived, not measured: | Item | Arithmetic | Estimate | |---|---|---| | Adapter size | 1,310,720 per layer x 32 layers | 42 M parameters, 0.5 % of 8B | | Adapter file | 42 M x 2 bytes | 84 MB | | Frozen base in 4 bits | 8 B x 0.5 bytes | 4 GB | | Adapter training state | 42 M x 16 bytes | 0.7 GB | | Full fine-tune, for comparison | 8 B x 16 bytes | 128 GB | Activations come on top and grow with batch size and sequence length, so plan with headroom. Still: a job that needs a rack in full fine-tuning fits on a single consumer card with QLoRA. ## The data is the project As with RAG, the model part is the easy part. The work is the examples. - **Quality over quantity.** Hundreds to a few thousand examples are a common range for a narrow task. LIMA got there with 1,000. Every bad example teaches the bad behaviour too, so one careful person reviewing 500 examples beats 50,000 scraped ones. - **Show the real input.** Use the messy tickets, the real invoices, the actual log lines, not tidy samples written for the occasion. - **Hold back a test set.** Put 10 to 20 % aside before training and never train on it. - **Mind what's in it.** Training data with names, customer data or HR content ends up baked into the adapter. Treat the adapter like the data it learned from, and read [the lesson on data protection](/blog/ai-and-the-law-gdpr-ai-act/) first. ## Measure before and after A fine-tune that "feels better" is not a result. Run the base model and the tuned model on the same held-back test set with the same scoring, as in [testing a model on your own work](/blog/testing-a-model-on-your-own-work/). Check two things: did the target task improve, and did anything else get worse? Narrow training can make a model noticeably worse at the things you didn't train, so keep a handful of general questions in the test set as a canary. ## Licence and serving The adapter is only half a model. It runs on one specific base model and inherits its licence: whatever the base licence says about commercial use, naming or attribution still applies to your tuned version. Check it before you start, as in [open weights and licenses](/blog/open-weights-and-licenses/). For serving there are two options. Merge the adapter into the weights and ship one ordinary model. Or keep base and adapters separate: servers such as vLLM and llama.cpp can load LoRA adapters on top of a base model, so one GPU can host several tuned variants, one per task, for the memory cost of one base plus a few small files. ## Recommendation Fine-tune last, not first. Climb the ladder: prompt, retrieve, constrain, and only then train. When you do train, use LoRA or QLoRA on an open model, spend the effort on a few hundred excellent examples, and measure against the base model on a test set you never trained on. Done that way, fine-tuning is a modest, repeatable engineering task. Done as "let's train it on our wiki", it is expensive and produces a model that states wrong facts confidently, in your house style.