Where to check AI benchmarks
The sources worth checking for model comparisons, what each is good for, and why we do not host benchmark numbers here.
We do not host benchmark data. It changes weekly, and a stale leaderboard is worse than no leaderboard because people act on it. Here is where to look instead, and what each source is actually good for.
Artificial Analysis
Independent benchmarking across quality, speed, and price, covering most commercial models and many open ones.
Good for: comparing cost against capability, and finding tokens per second and time to first token. The price and speed data is the part you cannot easily get elsewhere.
Watch for: aggregate quality indices combine several benchmarks into one number, which hides that a model may be strong at reasoning and weak at instruction following.
LMArena
Human preference rankings from blind pairwise comparisons. People are shown two anonymous responses and pick the better one.
Good for: a sense of which models people actually prefer in open ended conversation, which correlates with subjective quality better than most automatic benchmarks.
Watch for: it measures preference, not correctness. Verbose, confident, well formatted answers win votes even when wrong. It also does not test tool use or long context work, so it says little about agent performance.
Provider pricing pages
The only authoritative source for what a model costs.
Good for: current price per million input and output tokens, context window sizes, and rate limits. Third party pricing tables go stale quickly.
- Anthropic: https://www.anthropic.com/pricing
- OpenAI: https://openai.com/api/pricing
- Google: https://ai.google.dev/pricing
Watch for: output tokens usually cost several times input tokens, and cached input is cheaper again. A single blended figure will mislead you.
SWE-bench
Real GitHub issues from real repositories, scored on whether the generated patch makes the tests pass.
Good for: the closest public proxy for coding agent capability, because the task is end to end rather than a snippet completion.
Watch for: contamination is a live concern, since the issues are public and may appear in training data. Scores also depend heavily on the scaffolding around the model, so a number reflects a system, not just a model.
How to actually use benchmarks
Use them to build a shortlist of two or three candidates. Then test those on your own tasks, because that is the only measurement that predicts your results.
Public benchmarks average over workloads that are not yours. A model topping a leaderboard can be worse than a lower ranked one on your specific job, particularly for tool calling reliability, which most benchmarks barely test and which matters enormously for agents.
Re-check every few months. Prices drop, models update, and rankings move.
For how to run your own evaluation, see the interview bank on evaluation, which covers building a regression set.