When would you fine tune instead of using RAG?

Answer

The distinction that matters: retrieval supplies knowledge, fine tuning changes behaviour.

Fine tune when you need:

• A consistent output format that prompting does not reliably produce
• A specific style or tone at scale
• A narrow classification task where a small tuned model beats a large prompted one on cost and latency
• Domain vocabulary the model handles poorly

Use retrieval when you need:

• Facts the model does not have
• Anything that changes, since retraining for a document update is absurd
• Citations back to sources
• Per user or per tenant data

Fine tuning does not reliably teach facts. Training a model on your documentation does not make it answer questions about that documentation accurately. It shifts style and makes the model sound like your docs while still inventing specifics. This is the most expensive misunderstanding in the space.

They also compose. A fine tuned model that follows your output format, fed retrieved context for facts, is a reasonable design.

Practically, most problems brought to fine tuning are solved more cheaply by better prompts, better context, or better retrieval. Fine tuning adds a training pipeline, an evaluation burden, model versioning, and a re-tuning cost every time the base model updates.

Common wrong answer

"Fine tune the model on our company data so it knows our business." This is the classic error. It produces a model that writes in your voice and still fabricates details, because fine tuning adjusts weights toward a distribution rather than storing retrievable facts. Put the facts in context instead.