When teams decide to build an AI feature on top of a large language model, they hit the same fork in the road: how do you make a general-purpose model behave the way your business needs? There are three common answers—prompting, retrieval-augmented generation (RAG), and fine-tuning—and a surprising amount of wasted effort comes from reaching for the most complex one first.
The short version: most teams should start with prompting, graduate to RAG when they need the model to know things, and fine-tune only when they have a specific, well-justified reason. Here’s how to tell which situation you’re in.
Prompting: start here, always
Prompting means giving the model clear instructions, examples, and context in the request itself. No training, no infrastructure beyond an API call. It’s the cheapest, fastest, and most flexible approach—and it’s dramatically more capable than most people assume.
A well-constructed prompt can define a role, set a tone, enforce a format, walk through a reasoning process, and include a few examples of good output (“few-shot” prompting). Modern models follow detailed instructions remarkably well. A huge fraction of “we need a custom AI model” requests are actually solved by a better prompt.
Use prompting when: the model already has the general knowledge and skill to do the task, and you mainly need to shape how it responds. Drafting emails, classifying text, extracting structured data, rewriting content, answering general questions—prompting handles all of these.
The limits: prompting can’t give the model knowledge it doesn’t have. It doesn’t know your internal documentation, your latest pricing, or yesterday’s support tickets. You can paste some of that into the prompt, but there’s a context-window ceiling and a cost to sending large amounts of text on every request. When the bottleneck becomes knowledge rather than behavior, you’ve outgrown prompting alone.
RAG: when the model needs to know your stuff
Retrieval-augmented generation solves the knowledge problem. Instead of relying on what the model learned during training, you store your content—documents, policies, product data, past tickets—in a searchable form (usually a vector database). At query time, you retrieve the most relevant pieces and include them in the prompt. The model answers using that retrieved context.
This is the workhorse pattern for business AI, and for good reason:
Your data stays current. Update a document and the next answer reflects it. No retraining. For anything that changes—prices, policies, inventory, documentation—this is essential.
Answers are grounded and citable. Because the model is answering from specific retrieved passages, you can show where an answer came from. That’s the difference between a trustworthy assistant and a confident hallucination. For most business uses, citations aren’t a nice-to-have; they’re what makes the feature safe to deploy.
You control the knowledge boundary. The model answers from your content, not the open internet. You decide what it can see.
Use RAG when: the task depends on knowledge specific to your business, that knowledge changes over time, or you need answers traceable to a source. Internal knowledge assistants, documentation search, customer support over your own help center, “chat with your contracts”—these are all RAG.
The limits: RAG is only as good as your retrieval. If the system fetches the wrong passages, the model answers from the wrong context. The hard engineering in RAG isn’t the model—it’s chunking documents sensibly, generating good embeddings, and tuning retrieval so the right content surfaces. It’s very doable, but it’s where the real work lives. (For a deeper look at the data layer this depends on, see our piece on AI data foundations.)
Fine-tuning: the specialized tool, not the default
Fine-tuning means continuing to train a model on your own examples so it internalizes a behavior or style. It’s the approach people reach for first by reputation and should reach for last in practice.
Fine-tuning is genuinely useful for a narrow set of needs:
Consistent style or format at scale. If you need output to match a very specific voice or structure every single time, and prompting gets you 90% there, fine-tuning can close the gap reliably.
Teaching a specialized task or format the model handles awkwardly. Niche classification schemes, domain-specific output structures, or proprietary formats can benefit from examples baked into the model.
Latency and cost at very high volume. A smaller fine-tuned model can sometimes match a larger general model on a narrow task, running cheaper and faster—but you need the volume to justify it.
What fine-tuning does not do well is teach the model facts. Fine-tuning changes behavior, not knowledge. If you fine-tune a model on your documentation expecting it to “know” your policies, you’ll get a model that confidently makes up plausible-sounding answers, because it learned the style of your docs, not the content. For knowledge, you want RAG.
Fine-tuning also has real costs: you need a quality dataset (often hundreds to thousands of examples), the work to create and maintain it, and a retraining cycle every time things change. It adds an ongoing maintenance burden that prompting and RAG don’t.
Use fine-tuning when: you have a specific behavior or format need that prompting can’t reliably hit, you have the volume to justify it, and you’re prepared to maintain a training dataset.
They combine—and usually should
These approaches aren’t mutually exclusive. The most robust production systems often use all three: a carefully engineered prompt, RAG to supply current and grounded knowledge, and—occasionally—a fine-tuned model for a specialized step. A common, effective architecture is “RAG with a strong prompt,” with fine-tuning reserved for the rare case that genuinely needs it.
A practical decision path
- Try prompting first. Invest real effort in the prompt before concluding it isn’t enough. Most needs stop here.
- Hitting a knowledge wall? If the model needs to know things specific to your business or that change over time, add RAG.
- Still missing consistent behavior or format? Only now consider fine-tuning—and confirm it’s a behavior gap, not a knowledge gap.
The instinct to fine-tune first usually comes from associating “custom AI” with “training a model.” But for the overwhelming majority of business applications, the right answer is a good prompt and a well-built retrieval layer. That gets you a system that’s current, grounded, explainable, and far cheaper to run and maintain.
If you’re weighing how to add AI to an existing product and want to skip the expensive detours, we build exactly these systems—and we’ll tell you honestly when the simplest approach is the right one.