RAG vs Fine-Tuning: When to Fine Tune an LLM and When to Retrieve

The room relaxed. The demo looked good and the answers cited the right policy paragraph. Then the team tried to take the same system into production, and the work stalled.

I have watched this happen with retrieval-augmented generation, and I have watched it happen with a fine-tuned model. Teams often treat the stall as a tooling problem. In the cases I have seen, the tooling was fine enough for a demo but the missing piece was a decision about what the system was supposed to learn, and how you would measure it. The POC proved that a model can look competent on a staged question. It did not prove whether the product needed fresh knowledge, a stable behavior, or both.

Comparison pages will tell you, in one table, that RAG is for knowledge that changes and fine-tuning is for tone. That split is a useful start. It is also where a lot of teams stop thinking. The stall shows up later, on questions the demo never had to answer: how often the source of truth moves, how much labeled data you can actually afford, whether you have an eval set, what latency and cost per query you can live with, and who is allowed to see the knowledge once it lives in an index or in the weights.

I take a side here. Start by naming the job. If the job is knowledge freshness, I reach for retrieval, and I reach for it before I touch weights. If the job is format, tone, or a tight output contract, I work the prompt first, and I only fine-tune when the prompt cannot hold the behavior. If you need both, you can combine them. You should not combine them as a reflex.

That is the rag vs fine tuning decision your POC skipped. The rest of this piece shows when to fine tune an LLM, when to retrieve, and how to make that call on purpose.

What RAG and fine-tuning actually change

In Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, the authors describe RAG models as a combination of two memories. The parametric memory is a pre-trained sequence-to-sequence model. The non-parametric memory is a dense vector index, in their case Wikipedia, accessed with a neural retriever. They were explicit about why that split matters:

Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems.

And they stated the practical advantage of an inspectable index:

knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted.

They also showed that you can update what the model "knows" without retraining the generator. They built one Wikipedia index from December 2016 and compared it with an index from December 2018, then asked an open-domain question of the form "Who is the President of Peru?" for 82 world leaders who had changed between those dates. In their words:

RAG answers 70% correctly using the 2016 index for 2016 world leaders and 68% using the 2018 index for 2018 world leaders. Accuracy with mismatched indices is low (12% with the 2018 index and 2016 leaders, 4% with the 2016 index and 2018 leaders). This shows we can update RAG's world knowledge by simply replacing its non-parametric memory.

That is the core of retrieval. You keep facts outside the weights, you can swap the source, and you can often point at what was used. Microsoft's current guidance on retrieval-augmented generation makes the same operational point: large language models are trained on public data available at training time, and RAG is how you answer from private or frequently changing information and return citations back to source content. Anthropic's glossary puts it in one sentence: "RAG can be particularly useful for tasks that require up-to-date information, domain-specific knowledge, or explicit citation of sources."

Fine-tuning is a different lever. You update the model's weights with additional training data so the model starts to follow a pattern you care about. Anthropic's glossary is again plain: "Fine-tuning is the process of further training a pretrained language model using additional data. This causes the model to start representing and mimicking the patterns and characteristics of the fine-tuning dataset." Microsoft's guide to customizing LLMs focuses on supervised fine-tuning in particular, "an iterative process that adapts an existing large language model to a provided training set in order to improve performance, teach the model new skills, or reduce latency." Their good cases are "steering the model to output content in a specific and customized style, tone, or format, or tasks where the information needed to steer the model is too long or complex to fit into the prompt window."

Those two paragraphs already tell you when to fine tune an LLM, and when you should not. If the fact will change, keep it out of the weights. If the skill is a stable way of answering, writing, or formatting, the weights can hold it.

One research caveat belongs here, because it gets flattened in comparison posts. In Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, researchers at Microsoft compared unsupervised fine-tuning with RAG as methods for injecting knowledge. They wrote:

Our findings reveal that while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge.

And:

Moreover, we find that LLMs struggle to learn new factual information through unsupervised fine-tuning

Their current-events evaluation of Mistral 7B is the number I keep on a slide. RAG scored 0.875, regular unsupervised fine-tuning scored 0.504, and the base model scored 0.481. Combining regular unsupervised fine-tuning with RAG scored 0.810, which was worse than RAG alone. They also found that exposing the model to many paraphrases of the same fact helped fine-tuning, and they were careful about scope:

While fine-tuning can be useful for many use-cases, we found that RAG is a more reliable choice for knowledge injection.

They also say, later in the paper, that they focused on unsupervised training as their primary fine-tuning method, as opposed to instruction-tuning. That limit matters. Treat the result as evidence against baking changing facts into weights with unsupervised updates. Supervised fine-tuning for format, tone, or a consistent skill is a separate question, and this paper was not designed to settle it.

If you want a clear walkthrough of the three levers, watch Prompt Engineering, RAG, and Fine-tuning: Benefits and When to Use by Mark Hennings. For a shorter side-by-side, IBM Technology's RAG vs. Fine Tuning covers the same fork.

The decision your POC usually skips

Most POCs pick a method because a vendor demo, a blog table, or a weekend experiment made it look ready. The production decision is narrower. You are choosing which failure you are willing to own.

OpenAI's guide to optimizing LLM accuracy frames this as a matrix, not a ladder. You optimize for context when the model lacks knowledge, when that knowledge is out of date, or when the knowledge is proprietary. You optimize the model itself when results are inconsistently formatted, when the tone or style is wrong, or when reasoning is not followed consistently. I like that framing because it forces you to name the gap before you pick the tool.

The guide also has a sentence I quote to stakeholders who want to skip the boring work:

many of our largest customer deployments at OpenAI were done using only prompt engineering and RAG.

Microsoft makes a related point in the customizing-LLMs guide: prompt engineering, RAG, and fine-tuning can be used together on a single use case, and they are treated as complementary methods. The order still matters, because each step you add makes the system harder to iterate. OpenAI is direct about that cost. Once you introduce RAG, you have to tune retrieval as well as model behavior. Once you introduce fine-tuning, you have to rerun training and manage training and validation sets. Their closing advice is the one I wish more POC decks included:

squeeze as much accuracy out of basic methods as you can before reaching for more complex RAG or fine-tuning

The skipped decision, in practice, is almost always one of these:

• Whether the product mainly needs knowledge freshness, or a stable behavior and format.
• Whether you have enough labeled data, and what that labeling will cost.
• Whether you have an eval set that will still be meaningful after the demo.
• What latency and cost per query you can live with in production.
• Who may see the knowledge, and whether it should live in an index you can restrict or in weights you cannot easily inspect.
• Whether a hybrid (retrieval plus a light fine-tune) is justified, or just fashionable.

If you cannot answer those points in writing, you do not have a production plan. You have a demo that has not been asked a hard question yet.

A decision framework

I use a table like this in design reviews. Fill it in before you scale either path.

Two rows get skipped more often than the others, and they are the ones I would not ship without.

The first is the eval set. OpenAI's accuracy guide tells you to start with prompt engineering and "a solid evaluation set from prompt engineering which you can use as a baseline." Microsoft is just as blunt about fine-tuning: "Having a baseline for performance without fine-tuning is essential for knowing whether fine-tuning has improved model performance. Fine-tuning with bad data makes the base model worse, but without a baseline, it's hard to detect regressions." If your POC only has a handful of happy-path questions in a slide, you cannot tell a regression from a vibe.

The second is governance. Lewis and colleagues wanted accessed knowledge to be inspectable. That is easier when the knowledge sits in documents you can permission, log, and take down. It is harder when the same knowledge has been trained into a shared model that every user of the endpoint can tickle with a prompt. If a policy document is visible to legal and not to a frontline tool, do not put that document into a general fine-tune and hope the model keeps the secret. Retrieval with document-level access control is the more honest design. We already wrote about the product risk of ungrounded answers in 7 Practical Ways to Prevent LLM Hallucinations and Protect Your Product. The same instinct applies here: if you cannot point at the source, you do not have a defensible answer.

Failure modes I have seen after a good demo

These are the patterns that stall a POC that looked finished in the room.

You fine-tuned for a fact that moves. Rates, policies, inventory, and org charts do not belong in weights if they change on a schedule shorter than your retraining loop. Ovadia's results are the warning label for this path. The model can look fluent on last month's facts and still be wrong the week after a change.

You retrieved for a behavior problem. If the failure is schema, tone, or a stubborn refusal to follow a procedure, adding more documents will not fix it. OpenAI's Icelandic grammar-correction walkthrough in the same accuracy guide is the clean example. After fine-tuning GPT-4 on 1,000 examples, they reached a BLEU score of 87. Adding RAG then dropped the score to 83. They wrote: "RAG actually decreased accuracy, dropping four points from our GPT-4 fine-tuned model to 83." Their diagnosis was that this was a behavior-optimization problem, "where additional context will not necessarily help the model." Retrieval added noise. The eval caught it. A demo without that eval would have called the stacked pipeline more "advanced" and shipped the regression.

You had no eval set, so you could not leave the demo. This is the quiet stall. Legal, security, or a skeptical product owner asks how you will know the system still works after the corpus or the prompt changes. Nobody can answer, because the only test is a live prompt in a meeting.

You ignored latency and cost until traffic arrived. RAG is not free on each request. You pay for retrieval and for the tokens you stuff into the prompt. Fine-tuning is not free up front. Microsoft notes that "Fine-tuning has upfront costs for training the model. And additional hourly costs for hosting the custom model once it's deployed." They also note the other side: a fine-tune can reduce tokens, or let a smaller model reach the quality of a larger one on a narrow task. Their illustrative case starts with an IT department using GPT-4o to convert natural language queries to SQL, where "the responses are not always reliably grounded in their schema, and the cost is prohibitively high." They then "fine-tune GPT-4o mini with hundreds of requests and correct responses and produce a model that performs better than the base model with lower costs and latency." That trade only works if you measure cost per successful query, not cost per demo.

You treated access control as a later ticket. A shared index with no document-level permissions, or a fine-tune trained on a corpus that not every user is allowed to see, will stop a launch as soon as a security review is real.

You stacked RAG and a fine-tune because the internet said hybrid is best. Sometimes it is. Sometimes you have just added two systems you cannot debug. Have a hypothesis. Measure it. Drop the piece that does not earn its keep.

Illustrative composite. A product team in a regulated industry ran a two-week RAG demo over a policy corpus. Stakeholders liked the citations. When legal asked who would be allowed to see which documents after launch, and operations asked what would happen the night a policy was revised, the project froze. A second team, on a different product, fine-tuned a model on a few hundred "good answers" and got a cleaner tone in the demo. A month later the underlying rates had changed, and the model kept answering from the training week. Neither team had an eval set that would have caught the failure before demo day.

Prompt engineering first, then RAG, then a light fine-tune

I do not treat this as a purity test. I treat it as a way to spend complexity only when the eval says you need it.

Start with the prompt and a baseline eval. If the model already has the knowledge and only needs clearer instructions or a few carefully chosen examples, stop there. If the miss is missing, stale, or private context, add retrieval and spend your time on chunking, ranking, and citation. If the miss is still behavior (format, tone, tool use, or how the model reads retrieved context) and you have labeled examples, then consider a light fine-tune.

OpenAI's practical rule for that last step is one I have repeated in reviews:

if you have a RAG application, fine-tune the model with RAG examples in it so it isn't learning how to use the context zero-shot.

That is the honest hybrid: teach the model how to work with retrieved documents, and keep the documents themselves out of the weights. The research version of this idea is RAFT: Adapting Language Model to Domain Specific RAG. The authors train the model, given a question and a set of retrieved documents, "to ignore those documents that don't help in answering the question, which we call, distractor documents." RAFT "accomplishes this by citing verbatim the right sequence from the relevant document to help answer the question." They describe the setup as a domain-specific open-book exam: the model should use the right pages and should not be thrown by the wrong ones. Microsoft's Tech Community write-up of RAFT is a useful practitioner companion to the paper.

Two warnings travel with the hybrid.

First, do not fine-tune on a knowledge-injection problem and expect Ovadia-style unsupervised updates to beat a well-built index. If the job is new facts, retrieve them.

Second, do not add retrieval to a behavior problem and expect the score to rise. The Icelandic example is the counter-demo. Stacked pipelines need the same eval discipline as simple ones, and they need it more, because you now have two places to introduce a regression.

How you know the POC is ready to leave the lab

I would not take a RAG or fine-tuning POC to production until the team can show, in writing, the job the system is supposed to do, the decision table above, a frozen eval set with a baseline, and a plan for updates, access, latency, and cost. "The stakeholder liked the answer" is not one of those artifacts.

Once the system is live, you still need traces. You need to see which documents were retrieved, which examples the fine-tune was supposed to emulate, and where a failing answer came from. That is the job of AI observability, and it should be in the design before the second demo, not after the first incident. If the same team is also wiring agents to tools, you will meet the same ownership question in a different costume. We wrote about that failure mode in How to Orchestrate AI Agents Without Spaghetti Graphs. An agent that can retrieve or act without a named owner, a budget, and a stop condition will stall in the same way a knowledge POC stalls: the demo runs, the review starts, and nobody can explain the path.

If you cannot walk a skeptical reviewer through that flow with evidence, stay in the lab. The method can wait but the decision cannot.

FAQ

When should I use RAG vs fine-tuning?

Use RAG when the gap is knowledge: facts that were missing from training, knowledge that goes stale, or private documents the model should cite. Use fine-tuning when the gap is behavior: format, tone, a tight output contract, or a smaller model that needs to hold a skill at lower latency and cost. If both gaps are real, use both, and measure each one on an eval set before you stack them.

When to fine tune an LLM instead of retrieving?

Fine-tune when prompts cannot hold the behavior you need, when the knowledge is stable enough to live in weights, and when you have labeled examples plus a baseline eval that will show whether the fine-tune helped. Microsoft's guidance is to start from that baseline, because fine-tuning on bad data can make the base model worse. If the source of truth changes often, retrieve it. Do not bake a moving fact into the weights and hope a later training run keeps up.

Should I start with prompt engineering before RAG or fine-tuning?

Yes. OpenAI's own accuracy guide starts with prompt engineering and a solid evaluation set, and notes that many of its largest customer deployments were done using only prompt engineering and RAG. Squeeze accuracy out of the basic methods first. Add retrieval when the model lacks context. Add a fine-tune when consistency of behavior is still the gap.

Can I combine RAG and a light fine-tune?

Yes, and that is often the production answer when you need current knowledge and consistent behavior. OpenAI recommends fine-tuning a RAG application with RAG examples in the training data, so the model is not learning to use retrieved context zero-shot. Research such as RAFT goes further: the model is trained to ignore distractor documents and to cite the relevant passage. Combining the two should be a measured choice. It should not be the default. OpenAI's Icelandic correction example showed RAG lowering a fine-tuned model's BLEU from 87 to 83 when the problem was behavior rather than missing context.

Why do RAG and fine-tuning POCs stall before production?

The demo usually proves that a staged question can look good. Production asks questions the demo skipped: how often the knowledge changes, who can access it, how you will evaluate regressions, and what latency and cost per query you can live with. Teams that skip those criteria often pick a method for the room, then discover the method does not match the job once real traffic and real governance arrive.

How should latency, cost, and governance affect the choice?

RAG adds retrieval latency and extra prompt tokens on every query, and it keeps knowledge in an index you can update and, if you design it that way, restrict by user. Fine-tuning has upfront training cost and, on hosted custom models, a hosting cost, and it can reduce token use or let you run a smaller model. Knowledge in weights is harder to inspect, revoke, or segment by access policy than a document in an index. Put those constraints on the decision table before you scale the POC.

Sources

1. Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401; NeurIPS 2020. PDF.
2. Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs, EMNLP 2024. Also arXiv:2312.05934.
3. Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez, RAFT: Adapting Language Model to Domain Specific RAG, arXiv:2403.10131, 2024.
4. OpenAI, Optimizing LLM accuracy.
5. Microsoft Learn, Getting started with customizing a large language model (LLM).
6. Microsoft Learn, Retrieval augmented generation (RAG) in Azure AI Foundry.
7. Anthropic, Glossary.
8. Anthropic, Retrieval augmented generation guide.
9. Microsoft Tech Community, RAFT: A new way to teach LLMs to be better at RAG.
10. Databricks, RAG vs Fine Tuning.
11. Mark Hennings, Prompt Engineering, RAG, and Fine-tuning: Benefits and When to Use, YouTube.
12. IBM Technology, RAG vs. Fine Tuning, YouTube.
13. Roro, 7 Practical Ways to Prevent LLM Hallucinations and Protect Your Product.
14. Roro, AI Observability.
15. Roro, How to Orchestrate AI Agents Without Spaghetti Graphs.