When Fine-Tuning Beats Prompting, and When It Does Not

On this page

Fine-tuning changes behaviour, prompting changes instructions
Climb five rungs before you consider tuning
What fine-tuning is genuinely good at
What fine-tuning will not fix
How to tell which side of the line you are on
LoRA, full fine-tuning and preference tuning are three different tools
The arithmetic that usually decides it
The dataset is the project
Running the comparison honestly
What you are signing up for, on each side

Fine-tuning changes behaviour, prompting changes instructions

The short answer

Prompting tells a model what to do on this call. Fine-tuning adjusts weights on examples so the model tends toward that behaviour on every call, without being told. The practical rule follows from that difference: tune when you can show the behaviour but cannot write the rule for it, and do not tune when the model is missing a fact, because facts belong in retrieval and no amount of training will keep them current. Most teams who ask for fine-tuning have a context assembly problem wearing a training costume.

Both approaches produce better output, so they look like two routes to the same place. They are not. A prompt is an instruction read at run time, changed in minutes, reviewed in a pull request, reverted the moment it misbehaves. A tuned model is a build artifact with a training run behind it, a dataset somebody has to own, an eval suite that has to stay current, and a base model underneath that will be deprecated on a schedule nobody asked you about.

So the honest framing is a reversible change against a permanent one. Exhaust the reversible options first, not because prompting is superior, but because you learn what the problem actually is while doing it. Teams that tune early usually end up training a model to compensate for a task definition that was never settled.

Behaviour, not knowledgeTuning shifts how a model responds. It is a poor and expensive way to install facts, and a hopeless way to keep them fresh.
Few-shot is the previewIf thirty demonstrations in the prompt do not move quality, thousands of examples in the weights are unlikely to either.
The dataset is the deliverableThe training run takes hours. Curating, splitting and disagreement-checking the data is most of the project and all of the risk.
Every base model change reopens itA tuned artifact is pinned to the model it was trained on. Deprecation is a scheduled re-run, not a config change.
Keep the prompted baseline aliveWithout a maintained prompted version beside it, you cannot tell whether the tuned model still earns its upkeep.

Fine-tuning: Continuing to train an existing model on a set of input and output examples so that its default behaviour shifts toward the pattern those examples demonstrate, without changing the task it is asked to do at run time.
LoRA: Low-rank adaptation, a method that freezes the original weights and trains a small set of additional parameters instead, producing a lightweight adapter that can be swapped in at serving time rather than a whole new model.
Preference tuning: Training on pairs of a better and a worse response to the same input, which teaches ranking rather than exact wording, and suits tasks where many outputs are acceptable but some are clearly preferable.
Distillation: Using a large model to produce outputs that are then used to train a smaller model, so the smaller one imitates the larger one on a narrow task at lower cost and latency.
Catastrophic forgetting: The loss of general ability that occurs when training pushes a model too hard toward a narrow set of examples, visible as a model that nails the trained task and becomes noticeably worse at everything adjacent to it.

Climb five rungs before you consider tuning

Most problems that arrive labelled as fine-tuning problems are solved two rungs lower at a fraction of the cost. Climb the ladder in order and write down what each rung bought you, because that record is the only honest justification for a training run and the first thing a reviewer should ask for.

Sharpen the instructionhours
Most production prompts are a first draft nobody revisited. State the task, the exact output shape, the tie-break rules, and what to do when the input is unusable. Write the negative cases explicitly, since a model cannot infer a policy you kept in your head. This rung closes more gaps than anything below it.
Show examples in the prompta day
Put twenty to forty demonstrations in the prompt, chosen to cover the confusing cases rather than the easy ones. This is the most informative experiment in the whole decision, for reasons covered two sections down. Measure on a held-out set, never on the examples you picked.
Constrain the outputa day
If the failures are shape failures, malformed JSON, missing fields, invented enum values, a schema or grammar removes that class at the decoding layer. Doing it first stops you reading shape errors as reasoning errors. See structured output from LLMs.
Give it the missing evidenceone to three weeks
If the model is wrong about your products, your policies or anything that changed last quarter, it is missing context rather than training. Retrieval fixes that, keeps working when the facts change, and produces citations a human can check. RAG explained for builders is the starting point.
Split the taska week
One call doing extraction, judgement and formatting together fails in ways you cannot attribute. Two calls with a deterministic step between them are easier to test and often close the gap that looked like a capability limit.

Record the ladder, not just the outcome

For each rung, write what changed and by how much on a fixed evaluation set. When somebody proposes tuning six months later, that record tells you whether the cheap options were genuinely exhausted or merely tried once by somebody in a hurry. It also tells you what the tuned model has to beat.

What fine-tuning is genuinely good at

Fine-tuning earns its place in five situations. Each has the same shape: a behaviour that is consistent, demonstrable, and expensive to specify in words on every single call.

Format and protocol adherence at volume. Not the shape of the JSON, which constrained decoding handles, but the conventions inside it: which of two overlapping fields to populate, when a value is left null rather than guessed. House rules with long tails are exactly what a prompt cannot carry cheaply.
A voice reviewers recognise but cannot specify. If three editors reliably agree a draft is or is not on-voice and none can write the rule, that agreement is a training signal.
Classification into an idiosyncratic taxonomy. Forty internal codes with several near-neighbour pairs, where the distinction rests on precedent rather than definition. Prompts get long and still lose those pairs, because they are the ones no definition separates.
Unit cost and latency, by moving down a model size. If a smaller model can be trained to hit the quality bar on a narrow task, you drop both price per call and tail latency. This is distillation, the most common legitimate reason to tune.
Domain notation the base model mangles. Specialist markup, structured identifiers, unusual units, or a language variant it handles awkwardly. Tuning helps because the correction is systematic and appears on every call.

Symptom	Usual real cause	Cheaper fix that works	Tune only if
Output JSON breaks occasionally	Unconstrained decoding	Schema-constrained decoding or a tool schema	Shape holds but field choice is wrong in a patterned way
Wrong about your products or policies	Missing evidence in context	Retrieval with citations	Never. Facts in weights go stale silently
Answers drift off-voice	An instruction that describes taste in adjectives	Few-shot with strong exemplars	Reviewers agree on examples but not on the rule
The prompt has grown very long and costly	Instruction and demonstration tokens on every call	Prefix caching, then trimming	Volume is high and the behaviour has been stable for months
Latency is above the interaction budget	A large model doing a narrow job	Smaller model with a better prompt	The smaller model closes the gap only after training
Confuses two adjacent categories	A taxonomy defined by precedent, not definition	Two-stage coarse then fine classification	The confusion survives both the split and the examples

Symptoms that look like tuning problems, and what they usually are

What fine-tuning will not fix

Four categories of problem survive training completely intact, and money spent on them is simply lost. Each one has a tell you can check in an afternoon.

Facts that change. Prices, policies, inventory, staff, anything with a date on it. A tuned model is a snapshot with no expiry warning, which is worse than being wrong, because it is confidently wrong in a way retrieval would have caught.
Arithmetic and anything needing a guarantee. Totals, thresholds, entitlements, identifier lookups. Training moves the probability of a correct answer, and probability is not a guarantee. Put these in code permanently and let the model choose only which path to call.
A task nobody has defined. If two reviewers label the same example differently, the model will learn to average them, and the averaged behaviour will satisfy neither. Measure reviewer disagreement before you collect a single training row.
A retrieval layer returning the wrong passages. Tuning the generator only makes it more fluent about the wrong evidence, the failure described in confidently wrong RAG answers. Training makes it harder to spot, not easier.

The disagreement check comes first

Take fifty representative cases, have two experienced reviewers label them independently, and count how often they differ. If they differ often, you do not have a model problem. You have an unresolved policy question, and every hour spent on training before that is settled is an hour spent making the ambiguity permanent and much harder to see.

How to tell which side of the line you are on

The decision has one useful axis. Behaviour you can describe belongs in a prompt. Behaviour you can only demonstrate belongs in weights. Everything else in the debate is noise, and the four checks below tell you where you are standing in about a day of work.

Framework

The Describe or Demonstrate Line

A prompt carries rules. Weights carry patterns. These four checks tell you which one your behaviour is, before you spend anything on a training run.

Write the rule and hand it over

Give the written instruction to a competent colleague who has not seen the task, along with twenty real cases. If their output is close to what you wanted, the behaviour is describable and belongs in a prompt. If they follow your rule faithfully and still get it wrong, your rule is not the real rule, and the real one is living in somebody's judgement.

Count the exceptions the rule needs

Take the rule to fifty real cases and note every carve-out you add to make it survive. A rule needing three exceptions is a rule. A rule needing twenty is a memorised list wearing a rule's clothes, and memorised lists are what training is for. The count measures how much of the behaviour is precedent rather than principle.

Check agreement in both currencies

Ask your reviewers two separate questions. Do they agree on the written rule, and do they agree on the labels for individual examples? Agreement on labels combined with disagreement on the rule is the exact signature of a tuning problem. Disagreement on both means the task is undefined and no technique will save it. Agreement on both means write the rule down properly and stop.

Run the few-shot preview

Put twenty to forty of your best demonstrations directly in the prompt and measure on a held-out set. This is the closest cheap approximation of what training does, and it is predictive. A clear jump means the behaviour is learnable from demonstration, and tuning is mainly a way to make those demonstrations permanent and stop paying for them on every call. No movement is the important result: it says the gap is not a demonstration gap, and weights will not close what examples could not open.

Price the permanence before you cross

Crossing means owning a dataset, an eval suite, a serving path and a re-tuning cycle for as long as the feature exists. If nobody is named as owner of those four, the correct decision is a better prompt whatever the accuracy numbers say. An unowned tuned model becomes a box nobody dares change, which is the worst object to keep in production.

The fourth check deserves emphasis because it is cheap and almost nobody runs it. Few-shot demonstration and supervised fine-tuning teach through the same channel, correct input and output pairs. The prompt version is temporary, costly per call and capped by how many examples fit. The weights version is permanent and effectively unlimited. If the temporary version moves nothing, the channel itself is not where your problem lives.

Tuning decision record, one per taskyaml

# tuning-decision-record.yaml
# Written before any training run. Reread whenever the base model changes.

task: "classify inbound supplier email into 41 internal exception codes"
owner: "named engineer, not a team"
decided_on: 2026-08-26

ladder_record:                 # what each cheaper rung actually bought
  - rung: sharpened_instruction
    result: "clear gain on the easy 20 codes, nothing on the confusable pairs"
  - rung: few_shot_in_prompt
    examples: 32
    result: "jump on the near-neighbour codes, added ~2,100 input tokens per call"
  - rung: constrained_decoding
    result: "shape failures to zero, value errors unchanged"
  - rung: retrieval
    result: "no change, not missing facts, missing our conventions"
  - rung: task_split
    result: "coarse then fine helped the top confusion pair"

line_test:                     # the describe or demonstrate check
  rule_written: true
  rule_exceptions_needed: 19   # a rule needing this many carve-outs is not a rule
  reviewers_agree_on_rule: false
  reviewers_agree_on_labels: true      # <- the signature of a real tuning problem
  few_shot_preview_moved_quality: true # if false, stop here, weights will not help either

economics:
  monthly_calls: 200000
  tokens_saved_per_call_if_tuned: 2200   # instruction plus the few-shot block
  payback_months: 3.1                    # recompute with today's prices, never inherit this
  price_source: "our own billing export, not a published rate card"

dataset:
  size_at_first_run: 1200
  split_by: "supplier_id"      # never by row, near-duplicates leak across the boundary
  labelled_by: 2
  disagreement_rate_measured: true
  contains_model_generated_labels: false

commitments_accepted:          # tick all four or do not tune
  - "a versioned dataset with a named owner"
  - "an eval suite that runs on every candidate base model"
  - "a re-tuning budget for the next deprecation notice"
  - "a prompted baseline kept alive and re-tested each release"

revisit_when:
  - "base model deprecation notice"
  - "taxonomy changes by more than three codes"
  - "prompted baseline closes the gap on the eval suite"

LoRA, full fine-tuning and preference tuning are three different tools

The method matters less than the decision to tune at all, but picking the wrong one wastes a cycle. Start with the lightest method that can express the change you need, and only escalate when you have evidence that the lighter one plateaued.

Method	What it adjusts	Data it needs	Where it fails
LoRA or other adapter tuning	A small set of added parameters, base weights frozen	Input and output pairs, the smallest viable set	Cannot express very large behavioural shifts, and stacking many adapters gets operationally messy
Full supervised fine-tuning	All weights	The same pairs, usually many more of them	Catastrophic forgetting, higher cost, a separate deployment per variant
Preference tuning	A ranking over acceptable outputs	Pairs of better and worse responses to the same input	Needs consistent preference judgements, and rewards whatever your labellers actually preferred rather than what you meant
Continued pretraining	All weights, on raw domain text with no labels	Large volumes of unlabelled in-domain text	Expensive, slow, and rarely what a product team needs. Reach for it only for genuinely unusual domains

Tuning methods by what they adjust and how they fail

One operational detail decides more than the accuracy comparison. Adapters can usually be swapped at serving time against a shared base, so several variants coexist behind one deployment and a rollback is a pointer change. A fully tuned model is its own artifact that has to be hosted, versioned and paid for separately. If you expect several variants, one per customer segment or per document type, that difference will shape your infrastructure more than any quality gap between the two methods.

The arithmetic that usually decides it

The tuning cases that survive contact with a finance review are almost always about unit economics rather than quality. Prompting pays for the instruction and the demonstration block on every single call. Tuning pays once for the training and the data work, then pays less per call, either because the prompt is much shorter or because the work moved to a smaller model.

Two things distort this in practice. Prefix caching cuts the per-call cost of a long stable prompt, pushing break-even out and sometimes removing the case entirely, so measure your cached cost rather than the nominal one. And the one-off column is not the training run, which is the smallest line. It is labelling, disagreement resolution, the eval suite and the serving path. Put engineering hours in or the calculation will lie to you.

Break-even on a tuning project

The two prices are starting defaults, not quotes. Replace them with figures from your own billing export for last month, since published rates change and your effective rate depends on caching and discounts. Hours should cover labelling, disagreement resolution and the eval suite, not just the training run.

Calls per monthInput tokens per call today, prompted designInput tokens per call after tuning, shorter promptYour current price per million input tokensPrice per million input tokens on the target tuned modelOne-off training and hosting setup costHuman hours for data, labelling and evalsFully loaded hourly rate

0Prompted design, monthly input cost

0Tuned design, monthly input cost

0Monthly saving

0Months to pay back the one-off cost

Read the payback figure against the expected life of the feature or the base model, whichever is shorter. A payback measured in many months, on a model generation that may be superseded before then, is a bet that the world will hold still. Token cost arithmetic works through the general case.

The line item nobody budgets

Base models are deprecated on the provider's schedule, not yours. When that notice arrives, you re-run the training on a new base, re-run the eval suite, discover which behaviours the new base already has and which it lost, and re-tune the prompt around it. Budget that as a recurring project rather than a surprise, and keep the prompted baseline maintained so you have something to fall back to while the re-run is in progress.

The dataset is the project

The training run takes hours. The dataset takes the rest of the schedule, and every serious failure traces back to it. Four disciplines matter more than set size.

Split before you dedupe, and split by group

Near-duplicate rows are everywhere in real corpora, because real processes repeat. Split randomly by row and near-duplicates land on both sides of the boundary, so your held-out score is measuring memorisation. Split by a natural group key instead, the customer, the supplier, the document, the month, and hold that group out entirely. The score will drop, and the lower number is the true one.

Grow the set in doublings and measure each time

There is no universal number of examples, and any figure you have been quoted was measured on somebody else's task. Train on a small set, measure, double it, measure again. The curve rises steeply and then flattens, and the point where doubling stops paying is your answer. Guessing a target up front only records how much labelling you were willing to buy.

Include the cases that are hard for the right reason

A set of clean, easy examples teaches a model to be confident on cases it already handled. What moves the score is the confusable pairs, the ambiguous inputs your reviewers argued about, and the negatives that look positive. Record the reason alongside the label, because in six months the reason is the only thing that lets somebody re-adjudicate the row instead of deleting it.

Never train on unreviewed model output

Using a stronger model to draft labels is a legitimate accelerator. Using its output without review closes a loop, and the model's systematic errors become training targets that then look like ground truth forever. Mark every generated row, review it, and keep the flag so a later audit can separate the two populations.

Training row schema, with the metadata that keeps it maintainablejson

// one training row, with the metadata that makes the dataset maintainable
// store as JSONL, one object per line, in version control or an object store with immutable versions

{
  "id": "sup-exc-000412",
  "split": "train",                       // train | dev | holdout, assigned BEFORE dedupe
  "group_key": "supplier_88213",          // split on this, not on the row
  "dedupe_hash": "sha1:9c1f...",          // near-duplicate detection, computed on normalised text
  "messages": [
    {"role": "system", "content": "<the exact system prompt that will be used at serving time>"},
    {"role": "user", "content": "<the raw supplier email, unedited, including the signature block>"},
    {"role": "assistant", "content": "{\"code\":\"E17_PARTIAL_DELIVERY\",\"confidence\":\"high\"}"}
  ],
  "meta": {
    "labelled_by": ["ops_reviewer_2", "ops_reviewer_5"],
    "agreed": true,
    "label_reason": "two of six lines shipped, remainder acknowledged, no price dispute",
    "difficulty": "near_neighbour",        // easy | near_neighbour | adversarial
    "confusable_with": ["E19_SHORT_SHIP", "E22_BACKORDER"],
    "model_generated": false,              // if true, a human reviewed it and said so here
    "added_on": "2026-05-14"
  }
}

Running the comparison honestly

A tuned model will almost always beat a lazy prompt, and that comparison proves nothing. The result only means something if the prompted baseline received the same effort as the tuned candidate, on the same held-out cases, measured on the same metric, with cost and latency included.

The rigged comparisons repeat across teams: evaluating against the first-draft prompt rather than the best one, drawing eval cases from the same pool the training set came from, measuring exact string match where several outputs are correct, and reporting accuracy while ignoring that the tuned path is slower to change. The quiet one is forgetting to re-test the prompted baseline on the newest base model, which frequently closes the gap on its own. Writing evals for LLM systems covers the discipline behind all of them.

Before you accept a fine-tuning result

The prompted baseline was improved by the same engineer, with the same effort, before the comparison ran.
The held-out set was split by group key and never touched during development.
Reviewer disagreement on the task was measured and is lower than the gap you are claiming.
The metric reflects the decision the output feeds, not string similarity for its own sake.
Cost per call and tail latency are reported next to accuracy, using your own billing figures.
The prompted baseline was re-run on the newest available base model within the last month.
An owner is named for the dataset, the eval suite and the re-tuning cycle.
A rollback path to the prompted system exists and has been tested, not merely described.

0 of 8 done

What you are signing up for, on each side

The operational difference outlives the accuracy difference. Before choosing, read the two columns below as a description of your next two years rather than your next sprint.

Prompted systemTuned model

Time to change behaviourMinutes, with a reviewA data change, a training run and an eval cycle

Who can safely change itAnyone who can pass the eval suiteWhoever owns the dataset and the pipeline

RollbackRevert a versionRedeploy the previous artifact, if you kept it

Cost per callHigher, instruction tokens on every callLower, if you actually dropped a model size

New base model appearsRe-test, usually a small prompt editRe-train, re-evaluate, re-deploy

Where the behaviour is visibleIn readable text under reviewOnly in the dataset and the eval results

Handling a fact that changedRetrieval picks it upStale until the next training run

Moving to another providerRe-test and adjustStart the tuning project again

None of this makes tuning a bad choice. It makes it a commitment. Teams who get value from it treat the tuned model as a service: owned, versioned, monitored, and periodically justified against the cheaper alternative still running beside it. Our model fine-tuning work starts with the ladder record and the disagreement check rather than the training run.

Cite this

Fine-tuning teaches behaviour, not facts. Tune when reviewers agree on the examples but cannot agree on the rule, and only after a few-shot preview shows that demonstration moves quality at all.

Questions readers ask next

How many examples do I need to fine-tune a model?

There is no transferable number, and any figure quoted to you was measured on a different task with a different base model. Find yours empirically: train on a small set, measure on a held-out split, double the set, measure again. The curve rises steeply then flattens, and the point where doubling stops paying is your answer. Task difficulty and the distance from the base model's default behaviour matter far more than raw count.

Can fine-tuning teach a model my company's information?

It can absorb some of it, and you should not do it that way. Facts baked into weights cannot be updated without another training run, cannot be cited, and leave you unable to tell whether an answer came from your document or the model's own priors. Retrieval keeps facts current, produces a passage a human can check, and lets the system say it does not know. Tune for behaviour, retrieve for knowledge.

Is fine-tuning cheaper than a long prompt?

Sometimes. Break-even depends on call volume, the token count you genuinely remove, and whether prefix caching already cuts the cost of your long prompt. Run the arithmetic with your own billing figures rather than published rates, and put labelling hours and the eval suite in the one-off column, because the training run itself is the smallest cost. Add a re-training cycle for the next deprecation.

Will fine-tuning make the model worse at other things?

It can, and the effect is called catastrophic forgetting. Training hard on a narrow set of examples degrades abilities adjacent to the trained task, which is easy to miss because your eval suite tests the task you tuned for. Keep a small general regression set covering behaviours you rely on but did not train, and prefer adapter methods that leave base weights untouched when the required shift is modest.

How do I know if my problem needs tuning or better prompting?

Run the few-shot preview. Put twenty to forty of your best demonstrations in the prompt and measure on a held-out set. A clear improvement says the behaviour is learnable from demonstration, so tuning would mainly make those demonstrations permanent and stop you paying for them per call. No improvement says the gap lies elsewhere, usually in missing evidence, an undefined task, or reviewers who disagree with each other.

What happens to my fine-tuned model when the base model is deprecated?

You re-run the project on a new base. The dataset carries over, which is why it is the real asset, but the eval results do not, and the new base will already have some behaviours you trained for and will have lost others. Plan a scheduled re-tune rather than an emergency, keep the prompted baseline maintained to cover the gap, and store the exact dataset version and training configuration behind the current artifact.

Should I fine-tune a small model or prompt a large one?

Try the large model with a strong prompt first, because it costs a day and gives you a quality ceiling to aim at. If quality is adequate but cost or latency is not, distilling that behaviour into a smaller tuned model is the clearest legitimate use of tuning there is. If a large model with a good prompt cannot do the job, a smaller tuned model rarely rescues it, and the problem is the task definition or the evidence rather than model size.