What RAG and fine-tuning actually are
RAG (retrieval-augmented generation) connects a model to external data at query time. A user's question is embedded, a retriever searches a vector database for the most relevant chunks, and those chunks are injected into the prompt before the model answers — the technique was introduced by Meta AI in a 2020 paper and has since become the default way to ground LLM answers in an organization's own documents. The model's weights never change.
Fine-tuning takes a pretrained model and continues training it on a focused, labeled dataset of input/output examples, adjusting the model's internal weights so it consistently reproduces the pattern in those examples — a tone, an output schema, a narrow task. Per IBM's engineering writeup, it's best suited to stable, unchanging tasks that need a consistent output, not to knowledge that moves. As Red Hat's explainer puts it plainly: RAG augments the model by connecting it to external data; fine-tuning retrains the model itself on a focused dataset.
RAG vs fine-tuning: side by side
| Aspect | RAG | Fine-tuning |
|---|---|---|
| What it changes | What the model sees — retrieves context at inference time | The model itself — updates weights in a training run |
| Best for | Knowledge problems: facts that change, need citations | Behavior problems: tone, format, narrow task consistency |
| Knowledge freshness | Real-time — re-index a doc, the next answer reflects it | Frozen at training time — a new fact needs a new training run |
| Citations / provenance | Native — you know which chunk the answer used | Opaque — behavior lives in weights, not inspectable sources |
| Data needed | Your existing documents, any size | Hundreds to thousands of labeled input/output examples |
| Latency | Adds a retrieval hop (tens to hundreds of ms) | No retrieval step — direct model call |
| Cost pattern | Lower upfront, per-call token cost from injected context | Training cost upfront, cheaper per call at high volume |
| Typical build time | Days to a few weeks (embeddings, vector store, pipeline) | Weeks (labeled data collection is usually the bottleneck) |
The pattern across independent writeups from Databricks and IBM is consistent: RAG trades a small latency and per-call token cost for always-current, citable answers; fine-tuning trades an upfront training and data-labeling cost for consistent behavior and, at high volume, cheaper per-call inference.
When RAG is the right call
- • Knowledge changes often. Product docs, pricing, support content, or policy that updates weekly or faster — re-indexing a document is far cheaper than re-running a training loop every time it changes.
- • You need citations. Compliance, audit, or user trust requires pointing at the specific document a claim came from — something fine-tuned weights can't do.
- • The knowledge base is large or proprietary. Internal documentation, customer records, or a niche corpus the base model was never trained on.
- • You don't have labeled examples. RAG needs documents, which most teams already have; fine-tuning needs curated input/output pairs, which most teams don't.
When fine-tuning is the right call
- • You need a strict tone, voice, or output format — brand voice, a fixed JSON schema, or a specialized reasoning style — that prompting alone doesn't reliably enforce.
- • Latency is tight. No retrieval hop means a fine-tuned model answers in one call; that matters for voice interfaces or other sub-second budgets.
- • The knowledge is stable. A domain that changes on a quarterly cadence or slower doesn't fight the fact that a fine-tune is a frozen snapshot.
- • Volume is high and repetitive. A smaller fine-tuned model answering a narrow, high-volume task (ticket triage, structured extraction) can be meaningfully cheaper per call than a frontier model at scale.
One caution worth internalizing from IBM's guidance: don't fine-tune to teach a model facts. It doesn't reliably store them the way a retrieval index does — the more common outcome is a model that has memorized the style of the training examples and confidently invents details rather than accurately recalling them.
Get the weekly AI engineering brief
RAG, agents, evaluation, and the tools worth using — one practical email a week. Plus the free roadmap PDF.
By subscribing you agree to receive emails from AI Engineer Insights. Unsubscribe anytime. See our Privacy Policy.
The hybrid pattern: fine-tune for behavior, RAG for facts
In practice, "RAG vs fine-tuning" is often the wrong frame. RAG operates at the knowledge layer — what the model has access to. Fine-tuning operates at the behavior layer — how the model uses what it's given. Those are independent problems, which is why Databricks and IBM's field guidance both describe a hybrid as the strongest pattern for mature systems: fine-tune the model for tone, refusal calibration, and output structure, and layer RAG on top to supply the facts that change too fast to retrain on. A support assistant is the classic example — fine-tuning fixes the voice and the structured fields a ticketing system expects; RAG supplies the current product docs and policy so the answer is both on-brand and correct today. This combined approach is sometimes called RAFT (retrieval-augmented fine-tuning): fine-tuning a model specifically on the pattern of using retrieved context well.
The pragmatic sequence most guides converge on: ship RAG first because it's faster to stand up and reveals exactly where the base model's behavior actually breaks down, then fine-tune only the piece RAG can't reach — usually voice, structured output, or a narrow reasoning pattern.
Beyond classic fine-tuning: how model customization is evolving
"Fine-tuning" in 2026 rarely means retraining every weight. A handful of techniques have changed what the choice even looks like — most of them making customization cheaper, or removing the need for it entirely:
| Technique | What it does | Why it matters |
|---|---|---|
| LoRA / QLoRA (PEFT) | Freeze the base model, train tiny low-rank adapters | Fine-tunes an 8B–70B model on one GPU, ~0.1–1% of params, mergeable with no added latency |
| DPO (vs RLHF) | Align to preferences directly, no separate reward model | Simpler and more stable than RLHF; now the common way teams tune behavior |
| Distillation | Train a small student on a large teacher's outputs | Get near-frontier quality in a small, cheap-to-serve model |
| Reasoning / test-time compute | Spend compute at inference (o1/o3, DeepSeek-R1) | Buys reasoning ability without a task-specific fine-tune |
| Long context + RAG | Put knowledge in the prompt instead of the weights | Often replaces fine-tuning for knowledge; RAG stays the cost-efficient default |
The one that reset expectations is LoRA (Hu et al., 2021) and its 4-bit cousin QLoRA: by training small adapter matrices instead of the full model, they cut fine-tuning from a data-center job to something you can run on a single GPU (see the HuggingFace PEFT library). Preference tuning shifted too — Direct Preference Optimization (Rafailov et al., 2023) aligns a model to human preferences without the separate reward model and RL loop that made RLHF hard to run. And reasoning models (OpenAI's o-series, DeepSeek-R1) plus long context and RAG increasingly deliver, at inference time, capability that teams used to chase with a fine-tune — which is why recent research frames RAG and long context as the cost-efficient default for knowledge, with fine-tuning reserved for behavior.
Where small fine-tuned models win (with real examples)
The most durable use of fine-tuning in 2026 isn't making a frontier model smarter — it's making a small, open model good enough at one narrow task to replace a big one, at a fraction of the cost and latency. LoRA is what made this practical, and the production case studies are now concrete:
| Company | Model | Use case | Reported outcome |
|---|---|---|---|
| Checkr | Fine-tuned small Llama (via Predibase) | Background-check adjudication — 230 categories | ~5× lower cost, ~0.15s response, 90% accuracy on the hardest 2% of cases; replaced GPT-4 |
| Together AI (benchmark) | Fine-tuned Llama-3-8B on math data | Mathematical problem solving | 47.2% → 65.2% — beat Llama-3-70B (64.2%), ~91% of GPT-4o, ~50× cheaper, <$100 to train |
| Convirza | Multi-LoRA Llama (via Predibase) | Call-center conversation analysis at scale | Serves many fine-tuned adapters on shared infra; sub-second inference on millions of calls/month |
Checkr's team documented replacing GPT-4 with fine-tuned small open models for background-check classification — covered independently by Computerworld and in Predibase's case study — cutting cost roughly 5× while holding 90% accuracy on their hardest cases. Together AI's published benchmark shows the same shape: a fine-tuned Llama-3-8B beat the 70B base on math and reached ~91% of GPT-4o at ~50× lower cost. The pattern is consistent — narrow task, small model, big cost win — with the caveat that these outcomes are self-reported and your task's numbers will differ.
What AI engineers actually say about this
Practitioner sentiment on Hacker News and r/LocalLLaMA has converged on a few hard-won opinions worth knowing before you spend a training budget:
- • "Fine-tuning is for form, not facts." The most repeated lesson: teams that fine-tuned to make a model know something got confident hallucinations, then fixed it with RAG in a fraction of the time.
- • LoRA is the default, not full fine-tuning. The community treats parameter-efficient tuning as the normal path; full fine-tunes are seen as rarely worth the cost or the catastrophic-forgetting risk.
- • "Prompt → RAG → fine-tune → distill," in that order. A widely shared sequence: exhaust prompting and RAG and write evals first; fine-tune a small model only for the narrow piece that's left; distill if you need it smaller still.
- • The market agrees. Menlo Ventures' enterprise survey put RAG at 51% adoption versus just 9% for fine-tuning — a useful reality check against fine-tuning hype.
Source for the market figures: Menlo Ventures — 2024: The State of Generative AI in the Enterprise (RAG 51%, up from 31%; fine-tuning 9%).
Tips for fine-tuning a small model well
- Earn the fine-tune first. Ship prompting + RAG and write evals before you train anything — it tells you exactly which narrow gap a fine-tune actually needs to close.
- Use LoRA/QLoRA, not a full fine-tune. Same task quality for a tiny fraction of the compute and memory, mergeable with no inference-latency cost, and lower catastrophic-forgetting risk.
- Prioritize data quality over volume. A few hundred to a few thousand clean, human-reviewed examples typically beat tens of thousands scraped from logs.
- Consider distillation. If you need frontier-level quality in a small model, training the small model on a larger one's outputs is often a better lever than fine-tuning on hand-labeled data.
- Evaluate before and after, on held-out data. Confirm the new checkpoint improved the target task and didn't regress on tasks it used to handle — small models memorize fast, so watch for overfitting.
- Match the method to the layer. Facts change → RAG. Behavior needs enforcing → fine-tune. Need reasoning → reach for a reasoning model before you assume a fine-tune is required.
What it takes to build each
A minimal RAG pipeline needs an embedding model, a vector database, a retrieval/reranking step, and prompt-injection logic — infrastructure most AI-engineering teams already know how to stand up. It pairs naturally with tool-calling standards: wrapping a retrieval pipeline as an MCP server lets any AI host search your knowledge base as a standard tool, with zero custom integration per app.
A minimal fine-tuning pipeline needs a base model, a curated set of labeled input/output examples — commonly hundreds to low thousands for a supervised fine-tune — and a training job, plus an evaluation harness to confirm the new checkpoint didn't regress on tasks it used to handle. The data collection and labeling step is usually the real bottleneck, not the training compute itself. Building tool-using AI agents and evaluating them well is core AI-engineering work either way — if you're leveling up toward it, our AI engineering roadmap covers the fundamentals underneath both approaches.
Frequently Asked Questions
Is RAG cheaper than fine-tuning?
Usually to start, no — not always to run. RAG typically has a lower upfront build cost and no training run, but every call pays a token cost for the injected context plus a retrieval hop. Fine-tuning costs more upfront (data labeling and a training job) but a fine-tuned smaller model can get cheaper per call at high, repetitive volume. Compare total cost across build, run, and maintain, not just the price of one training run.
Can you use RAG and fine-tuning together?
Yes, and most production systems that reach real scale do. The common pattern is to fine-tune for behavior — tone, refusal style, output format — and use RAG for facts, so the model talks in a consistent voice while grounding its answers in current, cited documents. This combined pattern is sometimes called RAFT (retrieval-augmented fine-tuning).
Does fine-tuning teach a model new facts?
Not reliably. Fine-tuning adjusts weights toward the style and structure of the training examples; it does not give a model a dependable, inspectable store of facts the way a retrieval index does. Teams that fine-tune on a document set to make the model "know" it often end up with a model that confidently invents details rather than accurately recalling them. For facts, use RAG.
When should I choose RAG over fine-tuning?
Choose RAG when your knowledge changes often, you need to cite the source of an answer for compliance or trust, your knowledge base is large or proprietary, or you don't have labeled training examples. RAG is also the standard starting point even for teams that expect to add fine-tuning later.
When should I choose fine-tuning over RAG?
Choose fine-tuning when you need consistent behavior — a specific tone, refusal pattern, or output format — that prompting alone doesn't reliably enforce; when your latency budget can't absorb a retrieval hop; or when the underlying knowledge is stable and query volume is high enough that a smaller fine-tuned model is meaningfully cheaper to run than a frontier model with RAG.
How much data does fine-tuning need?
There's no universal number, but production guides commonly cite ranges in the hundreds to low thousands of clean, labeled input/output examples for a supervised fine-tune, with quality mattering more than raw count. RAG, by contrast, works directly off documents you likely already have, with no labeling step.
What is LoRA / QLoRA?
LoRA (Low-Rank Adaptation) fine-tunes a model by freezing its original weights and training small low-rank adapter matrices instead — roughly 0.1–1% of the parameters — so quality stays close to a full fine-tune at a fraction of the compute and memory, with no added inference latency once merged. QLoRA adds 4-bit quantization of the base model, making it possible to fine-tune large models on a single GPU. Both are parameter-efficient fine-tuning (PEFT) methods and are the default way teams fine-tune in 2026.
Can a small fine-tuned model beat a large model?
For a narrow, well-defined task, often yes. Publicly documented examples show fine-tuned small open models (8B-class) matching or beating much larger models on a specific task at far lower cost — e.g. Together AI reported a fine-tuned Llama-3-8B beating the 70B base on math at ~50× lower cost than GPT-4o, and Checkr replaced GPT-4 with fine-tuned small models for background-check classification at roughly 5× lower cost. The win is task-specific: a small fine-tuned model does not become generally smarter, just very good at the one job it was tuned for.
References
- • IBM — RAG vs. Fine-Tuning
- • IBM Developer — RAG vs. Fine Tuning: Which AI Strategy Should You Choose?
- • IBM Community — RAG vs. Fine-Tuning: Best Practices
- • Databricks — RAG vs Fine Tuning: Enterprise Decisions for AI Models
- • Red Hat — RAG vs. Fine-Tuning
- • Hu et al. — LoRA: Low-Rank Adaptation of Large Language Models
- • Rafailov et al. — Direct Preference Optimization (DPO)
- • HuggingFace — PEFT (LoRA/QLoRA) documentation
- • In Defense of RAG in the Era of Long-Context Language Models
- • Menlo Ventures — 2024: The State of Generative AI in the Enterprise (RAG 51% vs fine-tuning 9%)
- • Together AI — Fine-tuning Llama-3-8B (math benchmark, cost)
- • Predibase — How Checkr streamlines background checks with fine-tuned small models
- • Computerworld — Checkr ditches GPT-4 for a smaller genAI model