If you want an AI system to know things about your business (pricing, policies, products), the answer is almost always RAG. If you need it to behave in a very specific way (output format, tone, consistent classification) and a good prompt is not enough, then fine-tuning is worth evaluating. For most projects, the right starting point is well-measured prompt engineering. This guide gives you concrete criteria to decide.
Three techniques, three different levers
The most common mistake is treating these as interchangeable solutions to the same problem. They act on different things:
- Prompt engineering is designing the instructions, examples and context the model receives on every call, without changing the model itself. It is fast, cheap and reversible.
- RAG (retrieval-augmented generation) is searching your sources for relevant information and handing it to the model at answer time. It changes what the model knows for that answer. Our enterprise RAG guide covers how it works.
- Fine-tuning is further training a model on your own examples to adjust its weights. It changes how the model behaves: format, style, classification criteria or a narrow task.
OpenAI's guide to LLM accuracy frames it as two axes: optimizing context (what the model needs to know, solved with RAG) and optimizing the model (how it needs to act, solved with fine-tuning). The same guide recommends starting with a solid prompt and an evaluation set as the baseline.
Comparison: cost, update speed, data, traceability and risk
| Criterion | Prompt engineering | RAG | Fine-tuning |
|---|---|---|---|
| What it changes | Instructions and examples per call | Knowledge available at answer time | Model behavior (its weights) |
| Upfront cost | Low: days of work | Medium: ingestion, index, permissions | Medium to high: data prep, training, evaluation |
| Cost per call | Grows with long prompts | Adds retrieval and context tokens | Can drop with shorter prompts or a smaller model |
| Update speed | Immediate | Minutes to hours (re-index) | Days to weeks (retrain and re-evaluate) |
| Data needed | A handful of good examples | Your current documents | Dozens to thousands of curated input-output pairs |
| Traceability | High: the prompt is readable | High: every answer cites a source | Low: you cannot tell which example produced an answer |
| Per-user permissions | Not applicable | Yes, by filtering retrieval | No: what is learned is the same for everyone |
| Main risk | Brittle on unforeseen cases | Retrieving wrong or stale passages | Memorizing sensitive data, losing general skills, lock-in to one base model |
One market signal worth knowing: in 2026 OpenAI announced it is winding down its self-serve fine-tuning platform and no longer accepts new users, citing how much better recent models follow instructions and formats. For most companies, fine-tuning a closed model is no longer the first tool to reach for. It remains relevant with open-weight models (Llama, Mistral and similar), where techniques like LoRA adapt a model by training only a small fraction of its parameters.
A five-question decision tree
- Do you have a measured baseline? If not, write a clear prompt with examples and build a set of 50 to 100 real questions with expected answers. Without it you cannot tell whether any other technique helps. See our guide to evaluating AI assistants.
- Are errors caused by information the model does not have or that changes? (pricing, inventory, policies, contracts). You need RAG or live queries to your systems.
- Are errors about form rather than content? (broken JSON, mixed-up categories, off-brand tone). Improve the prompt, add examples and use structured outputs first. If problems persist, consider fine-tuning.
- Do you need citations or per-user permissions? Only RAG supports them. A fine-tuned model cannot say where an answer came from.
- Is volume high enough that cost per call matters? A small model fine-tuned for a narrow task (classification, extraction) can replace a large one and cut cost and latency. This is fine-tuning's strongest business case.
When to combine them
These techniques stack. A common pattern in mature systems is a fine-tuned model for format and judgment, fed by RAG for current knowledge, with a prompt that sets rules and boundaries. A 2024 Microsoft study on agricultural Q&A found fine-tuning improved accuracy by more than 6 percentage points and RAG added another 5 points on top, cumulatively.
Combining makes sense once you have measured each piece on its own. If you start with all three at once, you will not know which one is helping and which one is hurting.
Common misconceptions
- "Fine-tuning teaches the model our documents." This is the most expensive misunderstanding. A 2023 study (Ovadia et al.) compared both approaches for knowledge injection and found RAG consistently outperformed unsupervised fine-tuning, and that models struggle to learn new facts through fine-tuning. And every time a document changes, you would need to retrain.
- "Fine-tuning eliminates hallucinations." It does not. A fine-tuned model can make things up more convincingly, because it now sounds like your company. What reduces verifiable errors is cited evidence plus evaluation.
- "We need our own model for privacy." Privacy depends on vendor contracts, data residency and access controls, not on training a model. See our guide to choosing an LLM: Claude, GPT or open source.
- "RAG is only for chatbots." RAG also drafts proposals from approved clauses, checks contracts against policy and gives an agent context before it acts.
- "Fine-tuning open models is free." The weights may be; data preparation, training, evaluation, hosting and maintenance are not.
Examples by use case
| Use case | Recommended approach | Why |
|---|---|---|
| Internal assistant for HR policies and SOPs | RAG + prompt | Content changes; answers must cite sources and respect permissions |
| Customer support bot with live pricing and stock | RAG + live API calls + prompt | Prices and inventory change daily; see our AI chatbot on WhatsApp Business guide |
| Routing thousands of emails or tickets by category | Prompt first; fine-tune a small model if volume justifies it | Narrow, repetitive, cost- and latency-sensitive |
| Extracting fields from invoices or contracts into a fixed schema | Prompt + structured outputs; fine-tuning rarely | Current models follow schemas well; validation rules matter more than tuning |
| Drafting replies in an exact brand voice | Prompt with examples; fine-tune if you have hundreds of strong examples and prompting falls short | A style problem, not a knowledge problem |
| Questions about regulations (HIPAA, state privacy laws, tax rules) | RAG over dated, official sources | Rules get amended; a trained model would go stale |
What to prepare before deciding
- A use case with an owner: which task, for whom and how success is measured (time saved, tickets resolved, errors avoided).
- Real data: actual questions or documents, not invented ones. For fine-tuning, correct input-output pairs reviewed by someone from the business.
- Constraints: what personal or regulated data is involved (PHI, financial data, data covered by state privacy laws), where it may be processed and who can see what.
- An operating budget: cost does not end at launch; tokens, search infrastructure and maintenance are monthly. We break this down in the cost of implementing AI in business.
Through our AI development services, we validate this decision with a prototype on your real data, starting at USD 4,000 in 2 to 4 weeks, before you commit to a larger investment. Our nearshore team in Quito works in UTC-5, so reviews happen during your business day.
Frequently asked questions
Which is better, RAG or fine-tuning?
Neither is better in general; they solve different problems. RAG wins when the model needs proprietary or changing knowledge and you must cite sources or enforce permissions. Fine-tuning wins when you need highly consistent behavior (format, classification, style) that prompting cannot achieve, or when a small model must handle a narrow task at lower cost. Many companies only need prompt engineering plus RAG.
Can I train ChatGPT or Claude on my company documents?
Some models can technically be fine-tuned, but it is not the recommended way to make them know your documents: the model does not memorize reliably, cannot cite its source and goes stale with every change. RAG is the standard approach, searching your documents at answer time. Also note that in 2026 OpenAI announced it is winding down its self-serve fine-tuning platform.
How much data do I need for fine-tuning?
It depends on the task. Adjusting format or style may take dozens to a few hundred high-quality examples; more complex tasks can need thousands. Quality matters more than quantity: reviewed, consistent examples that reflect real cases, including the hard ones. Without a separate evaluation set, you cannot tell whether fine-tuning made the model better or worse.
Is RAG or fine-tuning cheaper?
RAG is usually cheaper and faster to get started, and it does not require retraining when data changes. Fine-tuning has higher upfront cost (data, training, evaluation) but can lower cost per call if it lets you use a smaller model or shorter prompts on high-volume tasks. The right comparison is total cost of ownership over 12 months, not the build cost alone.
Can I use RAG and fine-tuning together?
Yes, and mature systems often do: fine-tuning locks in behavior and format, while RAG supplies current knowledge for every answer. The recommended path is to get there step by step, measuring each technique's contribution against the same evaluation questions, rather than implementing everything on day one and guessing which part works.
Conclusion
The rule of thumb: knowledge goes in RAG, behavior goes in fine-tuning, and everything starts with a good prompt and a way to measure. Most enterprise projects are solved with well-built prompt engineering and RAG; fine-tuning earns its place on narrow, high-volume tasks or when style is critical and simpler options are exhausted.
If you are unsure which approach your use case needs, explore our AI development services or book a free consultation and we will assess it with your data.
Sources and references
- OpenAI: Optimizing LLM accuracy
- OpenAI: Model optimization (includes the fine-tuning wind-down notice)
- Ovadia et al. (2023): Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs
- Microsoft (2024): RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture
- Hu et al. (2021): LoRA, Low-Rank Adaptation of Large Language Models
- Claude documentation: Prompt engineering overview