Reading AI Benchmarks

Every model launch quotes benchmark results. They are not fabricated and they are selected, framed and occasionally contaminated. Reading them properly takes a few habits.

What the common ones measure

Knowledge benchmarks — large multiple-choice sets across academic subjects. They measure recall and recognition. They correlate loosely with usefulness.

Mathematics benchmarks — from school level to competition problems. These separate models sharply and are where reasoning modes shine.

Coding benchmarks — generating functions that pass tests, or resolving real issues in real repositories. The repository-based ones are considerably more informative than the function-writing ones, because they involve navigating a codebase rather than writing an isolated snippet.

Reasoning benchmarks — multi-step logic and inference.

Human preference rankings — people compare anonymous outputs and vote. These track general usefulness better than academic tests and reward style, formatting and length as well as substance.

Agentic benchmarks — completing multi-step tasks with tools. Newer, more relevant to how the tools are actually used, and less standardised.

The problems

Contamination. Benchmark questions leak into training data. A model that has seen the test scores higher without being better. Providers work to detect this and it cannot be fully verified from outside. Suspiciously large jumps on an old benchmark are worth scepticism.

Selective reporting. A provider quotes the benchmarks it leads. Absence of a common benchmark from a launch announcement is informative.

Inconsistent conditions. Scores vary with prompting technique, number of attempts, and whether tools were available. A score achieved with elaborate prompting and multiple samples is not comparable to a single-shot result, and the methodology is frequently in a footnote.

Saturation. Once every model scores above 90, the benchmark stops discriminating and small differences are noise.

Narrowness. Competition mathematics tells you nothing about drafting a report.

Reading a launch claim

What is the baseline? Improvement over the previous version, over a competitor, or over an old competitor version.

Single attempt or best of several?

With tools or without?

What prompting? Elaborate techniques inflate scores.

Which benchmarks are missing?

Is the evaluation independent or the vendor's own run?

What actually predicts your experience

Your own five tasks. Nothing beats running representative work through candidates and comparing. An hour of this outweighs every published number.

Human preference rankings, as a rough guide to general usefulness.

Repository-level coding benchmarks, if you write software.

Long-context evaluations, if you work with large documents. These measure whether a model uses the middle of its context window, which several do poorly despite advertising large windows.

Reports from people doing your kind of work, with specifics.

The category error to avoid

A model that improves 15 points on a mathematics benchmark has improved at mathematics. It has probably not improved at summarising your meeting notes, and the marketing will imply otherwise by presenting the mathematics score as evidence of general advancement.

Ask what the benchmark measures and whether you do that. For most people the answer is no, and the appropriate response to a headline capability gain is to keep using what works until your own tasks show a difference.

A reasonable position

Use benchmarks to exclude the clearly weaker options. Use preference rankings for a general sense. Use your own tasks to choose. Re-examine every six months rather than at every launch.

That approach costs an hour twice a year and produces better decisions than following announcements.

When a launch announcement should change your behaviour

Most launches should change nothing about what you use. A few should. The distinguishing questions are short.

Did a capability appear that you specifically lack? Not a better score on something you already do adequately — a thing you could not do before.

Did the price of something you use heavily fall materially?

Did a data or retention policy change in a way that affects whether you can use it for work?

Did a limitation you work around routinely get fixed? Context length, file handling, a language.

Has anyone using it for your kind of work reported a difference after a month?

If none of these, do nothing. The cost of switching — relearning the behaviour, rebuilding prompts, migrating history — is real, and a capability improvement you cannot feel in your own tasks is not worth paying it for.

Re-examine on your schedule rather than the industry's.