Why you cannot eyeball quality
The normal way people evaluate a prompt change is to try it a few times and see if it looks better. This does not work, and it fails in a specific way worth understanding.
Three reasons it fails
Outputs are not deterministic. The same prompt can produce different answers. Two good results after a change might be luck. Two bad results before it might also have been luck.
You test what you just fixed. After changing the prompt to handle empty input, you try empty input. You do not retry the twelve cases that used to work, so you do not notice that three now fail.
Fluent text reads as correct. A well structured, confident answer feels right. Judging quality by reading is largely judging presentation, and models are very good at presentation.
The specific danger
Prompt changes are not local. Adding a rule to fix one case changes behaviour on cases you were not thinking about.
Without a fixed set of cases you re-run every time, you cannot see this happening. You improve one thing, quietly break two, and conclude the prompt is getting better because the case you are watching improved.
What to do instead
Write down cases. Run them every time. That is the whole discipline, and even twenty cases changes everything.
The set does not need to be large or clever. It needs to be fixed, so that today's run is comparable with last week's.
What a case looks like
An input, and something checkable about the output.
Checkable is the important word. Not "is the answer good" but something a script or a person can answer without debate:
- Does it contain the order id
- Does it refuse when the context lacks the answer
- Is it valid JSON
- Does it call the search tool rather than answering from memory
- Is it under 200 words
Vague criteria produce evaluations that drift as your mood does.
Where cases come from
Real failures. Every time the agent does something wrong, that becomes a case.
This is the same principle as the "do not" section in a rules file: entries earn their place by having happened. A suite built from real failures is worth far more than one built from imagined edge cases, and it grows naturally as you use the thing.
Accept a pass rate
You will not get a hundred percent, and chasing it will make you overfit to your own suite.
Track the number over time. Twenty six out of thirty this week against twenty two last week is real information. "It seems better" is not.
Start now, small
Ten cases you actually run beat a hundred you plan to write. Add one every time something breaks.
Building a regression set covers how to assemble and run one.