RAG or fine-tuning: deciding with evidence rather than preference
Retrieval and fine-tuning solve different problems, and most business requirements turn out to be retrieval problems wearing a fine-tuning costume. The distinction is simple: retrieval changes what the model knows, fine-tuning changes how it behaves. Deciding without that distinction produces expensive rework.
This is the first architectural decision on almost every applied AI project, and it is routinely made on instinct. The useful framing is not which technique is better — neither is — but which failure you are actually trying to fix.
What each one actually changes
That penultimate row is the one that reframes most projects. If an answer must be attributable to a source document — and in business contexts it usually must — fine-tuning cannot deliver it at all, regardless of quality.
Five questions that decide it
- Does the answer depend on data that changes? Policies, prices, inventory, documents. If yes, retrieval — otherwise every change requires retraining.
- Must the answer cite a source? If a user needs to verify the answer against a document, retrieval is the only option that provides it.
- Is the problem format rather than knowledge? Consistent JSON structure, a specific tone, a narrow classification taxonomy. That is behaviour, and fine-tuning addresses it directly.
- Is volume high enough that per-request cost dominates? A fine-tuned smaller model can be materially cheaper per call than a large model with long retrieved context.
- Do you have enough labelled examples? Fine-tuning needs hundreds to thousands of consistent examples. If you cannot produce them, the decision is made for you.
They are not alternatives
The framing as a binary choice is itself part of the confusion. The two compose, and the strongest production systems frequently use both for different jobs.
- Fine-tune a small model to reliably emit a strict output schema, then feed it retrieved context at request time
- Fine-tune for domain vocabulary and phrasing, retrieve for the facts that phrasing describes
- Fine-tune a cheap classifier to route requests, retrieve only on the branches that need grounding
- Use retrieval in production while accumulating the labelled examples that would make fine-tuning viable later
The work that dominates either path
Teams choose a technique and then discover the technique was the small part. For retrieval, the effort is in the retrieval quality; for fine-tuning, it is in the dataset.
That last row is the one that determines whether the project ships. Without it, "is it good enough?" is settled by whoever demonstrates most confidently.
Let the evaluation set decide
The defensible way to make this choice is to build the evaluation set first, implement the cheaper option, measure it, and only escalate if it falls short. This inverts the usual order, where the technique is chosen first and evaluated afterwards — or not at all.
- Write 50–200 representative inputs with expected outputs, including the awkward cases rather than only the clean ones.
- Define what passing means numerically — field-level accuracy, an acceptable refusal rate, a latency ceiling.
- Implement retrieval with a structured prompt. Measure against the set.
- If it passes, stop. You have a system you can update by reindexing a document.
- If it fails, examine how it fails: wrong facts retrieved points at retrieval quality, wrong format or tone points at fine-tuning.
- Escalate only on the dimension that failed, and re-measure against the same set.
Cost, honestly
Fine-tuning's cost advantage is real but arrives later than people expect. You pay training cost up front, then save per request. Retrieval has no setup cost but pays for retrieved context on every call.
Whether that crossover matters depends entirely on your volume, and it is straightforward to model before committing — measure tokens per request on your evaluation set, project to expected volume, and compare. What you should not do is assume either direction.
There is also an ongoing cost to fine-tuning that is easy to omit from the comparison: every base-model upgrade potentially invalidates your fine-tune, and you retrain to benefit from it. Retrieval systems inherit improvements to the underlying model for free.
Questions
Should I start with RAG or fine-tuning?
Retrieval, in almost every case. It is faster to build, cheaper to change, updates when your data changes, and produces citations. Fine-tuning is the right escalation when an evaluation set shows retrieval cannot meet the bar on format consistency, a narrow specialised task, or per-request cost at high volume.
Can fine-tuning teach a model new facts?
Poorly, and it is the wrong tool for it. Fine-tuning shapes behaviour — format, tone, task framing. Facts embedded in weights cannot be cited, cannot be updated without retraining, and tend to be recalled unreliably. If the requirement is factual grounding, that is a retrieval problem.
How many examples does fine-tuning need?
Typically hundreds to low thousands of consistent, high-quality examples, depending on task narrowness. Consistency matters more than volume — a few hundred examples that agree with each other outperform several thousand that contradict. If you cannot assemble that dataset, the decision is effectively made.
Does using both add too much complexity?
Not inherently. A fine-tuned model that reliably emits a strict schema, fed retrieved context at request time, is a common and sound production pattern. The complexity to avoid is adopting both before an evaluation set has shown that one alone is insufficient.