RAG or fine-tuning? Two very different ways to make an AI model useful

The problem both approaches try to solve

A general-purpose AI language model is trained on a huge sweep of text up to some cut-off point, then left alone. It does not know about your company’s internal policies, last month’s court ruling, or the product catalogue your business updated yesterday. It also cannot always show its working or say where an answer came from.

Two established techniques address this: retrieval-augmented generation, usually shortened to RAG, and fine-tuning. They are often mentioned in the same breath because both aim to make a general model behave as if it knows your specific material. But they work in fundamentally different ways, solve different problems, and suit different situations. Understanding the difference matters if you are ever asked to choose between them, or simply want to know what is going on behind an AI tool you use at work.

What retrieval-augmented generation actually does

RAG leaves the underlying model untouched. Instead, it builds a search step in front of the model. When you ask a question, the system first searches a separate store of documents, such as a company handbook, a set of legal filings, or product manuals, and pulls out the passages that look most relevant. Those passages are then inserted into the prompt alongside your question, and the model is asked to answer using that material.

Think of it like giving an very well-read colleague a stack of relevant printouts just before they answer your question, rather than expecting them to have memorised your entire filing cabinet. The model’s core knowledge and reasoning ability stay the same. What changes is what it is shown at the moment it answers.

This has a useful side effect: because the answer is built from retrieved documents, a well-designed RAG system can cite which passage it used, which makes it easier to check the answer and spot when the model has strayed from the source material. It also means the underlying information store can be updated constantly, simply by adding or removing documents from the search index, without touching the model itself.

What fine-tuning actually does

Fine-tuning is a different kind of intervention. It involves further training an existing model on a curated set of examples, adjusting the internal parameters that determine how it responds. Instead of feeding it documents at question time, you feed it examples in advance, of the style, format, tone or type of task you want it to get better at, and the training process nudges the model’s behaviour to match.

Fine-tuning is good at teaching a model how to respond: to write in a particular house style, follow a specific output format, get better at a narrow specialised task such as classifying support tickets, or handle a domain’s jargon and conventions more fluently. It does not, by itself, give the model reliable access to a large, changing body of facts. Once a model is fine-tuned, that knowledge is baked into its parameters. Updating it means retraining, which is slower and more resource-intensive than adding a document to a search index.

The core distinction

A useful shorthand: RAG changes what the model can see. Fine-tuning changes what the model has learned to do. If the problem is that a model does not know something, and that something is written down somewhere and changes over time, RAG is generally the more natural fit. If the problem is that the model knows enough but responds in the wrong way, structure, tone or level of detail, or needs to get sharper at a narrow repeated task, fine-tuning is usually the better tool.

In practice, many production AI systems that need to answer questions about a specific organisation’s information use RAG rather than fine-tuning, precisely because organisational knowledge changes constantly, and RAG lets that knowledge be updated without retraining anything. Fine-tuning is more commonly reserved for shaping behaviour and style, or for narrow, well-defined tasks where a large volume of labelled examples exists.

The two are not mutually exclusive. It is entirely possible to fine-tune a model for tone and task-handling, and then also give it a retrieval step so it can pull in current facts. Many real deployments combine elements of both, using fine-tuning to shape how the model behaves and RAG to keep it grounded in accurate, up-to-date material.

Why this matters beyond the engineering

For anyone assessing an AI tool, whether as a business buyer, a policy-maker or simply a curious user, the distinction has practical consequences. A RAG-based system’s reliability depends heavily on the quality and currency of the document store behind it, and on whether it is actually restricted to using that material rather than falling back on general knowledge. A fine-tuned system’s reliability depends on the quality and representativeness of its training examples, and it can be harder to check exactly why it produced a given answer.

Neither approach is a guarantee against errors. RAG can still misread or misapply retrieved text, and fine-tuning can still produce confidently wrong answers if the training examples were flawed or too narrow. Knowing which technique underlies a tool is a reasonable first question to ask before relying on it for anything important, and organisations that build AI systems should be able to explain which approach, or combination, they use and why.

For further reading on how the UK approaches oversight of AI systems generally, see the sources below.