Fine-tuning means lightly retraining an existing AI model on your own examples, so that it performs one specific task better than a general-purpose model. Since LoRA and QLoRA, it has become affordable on a single server. But it is not always the right tool: to make an AI know your documents, RAG is often simpler, and a good prompt is sometimes enough.
Key takeaways
- Three ways to specialise an AI: the prompt (instructions and examples), RAG (the model looks up your documents), fine-tuning (the model learns a task).
- Fine-tuning mainly teaches a format, a style, a repetitive decision; it is poorly suited to information that changes.
- LoRA and QLoRA made it accessible: only a small fraction of the model is trained.
- Recent small open models, under Apache 2.0 or MIT licences, are well suited to this use.
- Before the first training run: a few dozen to a few hundred carefully written examples, a test set, and a checked licence.
Prompt, RAG or fine-tuning: three tools for three problems
The prompt is the first step, and it costs nothing: clear instructions, a few examples of good answers. Many projects stop there.
RAG (retrieval-augmented generation) meets a different need: making the AI know your procedures, contracts or catalogue. The model does not change; with each question, it is given the relevant passages from your documents. When a price changes, you update the document and the answer follows. I explained the principle in this article on RAG.
Fine-tuning changes the model itself. It helps when the problem is not what the model knows, but what it does: always following the same output format, sorting requests into your categories, adopting your company's tone, extracting the same fields from thousands of documents. A Meta study published in 2023 (LIMA) points the same way: its authors conclude that almost all of a model's knowledge comes from pre-training, and that little data is needed to teach it to answer well.
My rule of thumb: knowledge that changes, RAG; stable, repetitive behaviour, fine-tuning; and always the prompt first.
Why fine-tuning is the future of small specialised models
Retraining a whole model was out of reach for a small business. Two papers changed that. In 2021, a Microsoft team published LoRA: the original model is frozen and only small added matrices are trained. On GPT-3, the authors measured 10,000 times fewer trainable parameters and three times less GPU memory, for equal or better quality. In 2023, QLoRA (University of Washington) combined this method with a model compressed to 4 bits: a 65-billion-parameter model can then be fine-tuned on a single 48 GB GPU.
Meanwhile, small open models have multiplied. As of 15 November 2026, Google publishes Gemma 4 (from E2B to 31B) under the Apache 2.0 licence, Alibaba the Qwen3.5 series (from 0.8 to 9 billion parameters for the smallest) under Apache 2.0, Mistral AI the Ministral 3 models (3B, 8B, 14B) under Apache 2.0, and Microsoft its Phi-4 family under the MIT licence.
So the future I see for a small business is not one giant model that knows everything, but a few small fine-tuned models, each excellent at one task, running in-house. I cover choosing a general-purpose model in the article on large models, and hosting in the one on self-hosted AI.
Data and licence: what you need before you start
The data. You need little, but it must be good. OpenAI's documentation states a minimum of 10 examples, visible improvements from 50 to 100, and recommends starting with 50 well-crafted examples. LIMA achieved good results with 1,000 carefully chosen examples. Set part of your examples aside to test the model: without that test set, you cannot know whether it has improved. And if your examples contain personal data, data protection law applies to training like everything else.
The base model's licence. Your fine-tuned model inherits its terms. Apache 2.0 and MIT allow commercial use without special conditions. Other licences impose some: Llama 4's (Meta) requires you to display "Built with Llama", to start the name of any derived model you distribute with "Llama", and demands a special licence above 700 million monthly active users. Read the model card before you choose, not after.
Under the hood: for your IT person
A LoRA fine-tune of a small open model fits on a single GPU. Licences checked on the official repositories on 15 November 2026.
- Hugging Face PEFT (Apache 2.0): the reference implementation of LoRA and related methods, on top of Transformers.
- Hugging Face TRL (Apache 2.0): ready-made trainers for supervised fine-tuning and preference alignment.
- Unsloth: the core is Apache 2.0, but some optional components, including the Unsloth Studio interface, are AGPL-3.0. It can export to GGUF, readable by llama.cpp and Ollama.
- Axolotl (Apache 2.0): the whole pipeline (data preparation, training, evaluation, quantisation) described in a single YAML file, hence reproducible.
- Serving. vLLM can serve several LoRA adapters on the same base model: one model in memory, several specialities. Otherwise, merge the adapter and quantise.
- Pitfalls. Use the same chat template as in training; watch for overfitting on the test set; check that the model has not lost its general abilities. A model can reproduce passages from its training data: put nothing in it that must not come back out, or keep it strictly internal.
What I have learned
My DataTerra project made this choice for computer vision: rather than one large general-purpose model, small models trained on its own images, each for a specific defect such as cracks or corrosion. I describe that approach in the article on detection.
For my agents, on the other hand, I fine-tune no model at all. What they need to know changes every day: decisions, projects, preferences. That lives in a persistent memory, a search database I host, in other words RAG, and the working rules are written into their instructions. A stable task justifies training; living knowledge does not.
Where to start
1. Write down the task and gather 50 examples
One sentence for the task, then 50 "input, good answer" pairs taken from your real work, reviewed by the person who does that job today. Keep some of them for testing.
2. Try the prompt, then RAG
Measure the result on your test examples. If the model gets it wrong because it does not know your documents, that is a RAG problem; if it knows the answer but phrases it badly, it is a candidate for fine-tuning.
3. Fine-tune a small open model, and compare
Pick a model with a clear licence, train a LoRA adapter, then compare it with the original model on the same test set. Keep it only if it does better.
Frequently asked questions
Can fine-tuning teach the AI my catalogue?
Poorly. Prices and product codes change, and a fine-tuned model does not update itself. For a catalogue, RAG is the right tool; fine-tuning can then shape how the answer is presented.
Do I need a specialist to fine-tune a model?
On the technical side, an IT person comfortable with Python is enough. The real work is defining the task and producing good examples: that is done by your business.
Who owns the fine-tuned model?
If you train it in-house on an open model, the adapter is yours, within the limits of the base model's licence. If you use a provider's service, the fine-tuned model stays with the provider and follows its retirement schedule.
Going further
I design custom applications and automations, with a local model when confidentiality requires it and an external API when that is acceptable. The guides for my personal AI server detail the technical stack. To find out whether your case calls for a prompt, RAG or fine-tuning: a 20-minute discovery call.
Sources
- LoRA: Low-Rank Adaptation of Large Language Models, Hu et al., arXiv 2106.09685, 17 June 2021.
- QLoRA: Efficient Finetuning of Quantized LLMs, Dettmers et al., arXiv 2305.14314, 23 May 2023.
- LIMA: Less Is More for Alignment, Zhou et al., arXiv 2305.11206, 18 May 2023.
- Supervised fine-tuning, OpenAI (accessed 15 November 2026).
- Gemma 4, model card and Apache 2.0 licence, Google, March 2026 (accessed 15 November 2026).
- Qwen3.5-9B, model card, Qwen (Alibaba), February 2026 (accessed 15 November 2026).
- Ministral 3 8B Instruct, model card, Mistral AI (accessed 15 November 2026).
- Phi-4-reasoning-vision-15B, model card, Microsoft, January 2026 (accessed 15 November 2026).
- Llama 4 Community License Agreement, Meta, 5 April 2025.
- Official repositories and licences: PEFT, TRL, Unsloth, Axolotl, vLLM, GitHub (accessed 15 November 2026).
Cover photo: Yılmaz Kilim / Wikimedia Commons, public domain (CC0).
Tags: fine-tuning, LoRA, RAG, small language models, specialised AI, open source