How do you choose which model to use for a task?

Answer

Decide against your own workload, because model rankings on public benchmarks do not reliably predict performance on a specific task.

The process:

1. Build a small evaluation set from real tasks with checkable outcomes. Twenty cases is enough to start.
2. Run the candidates on it, several times each because of nondeterminism.
3. Compare success rate, cost per task, and latency. All three, not just quality.
4. Pick the cheapest model that clears your quality bar, not the highest scoring one.

Considerations that often decide it:

• Latency matters if a human is waiting. A slower, slightly better model can be the wrong choice in an interactive path.
• Context window must fit your typical task.
• Tool calling reliability varies between models more than general capability does, and matters enormously for agents.
• Structured output support, if you need schema conformance.
• Data and deployment constraints may rule out providers regardless of quality.

Mixed routing is normal. Use a strong model for planning and hard reasoning, a cheaper one for mechanical steps.

Re-evaluate periodically. Models are updated, prices change, and the ordering shifts. Anything you decided a year ago is probably stale.

Common wrong answer

Picking based on a leaderboard. Benchmarks measure narrow tasks, are frequently contaminated by training data, and average over workloads that are not yours. They are useful for a shortlist and useless as a decision.

Also common is choosing the most capable model for everything as a safety measure. It is expensive and often slower, and for many steps in a loop the difference in outcome is nil.