What a published benchmark score leaves out

The number is the last step of a long pipeline

Every benchmark result is produced by a chain: a fixed set of items, a template that wraps each item into a request, sampling settings, some procedure for pulling an answer out of free-form text, and a rule for deciding whether that answer counts as correct. The headline figure is the final aggregation of that chain. Two groups running the same named benchmark on the same system can report materially different figures without either of them making a mistake, because they made different reasonable choices at each link.

Answer extraction does more work than it looks

On free-response benchmarks, the extractor is often the largest hidden variable. A strict rule that requires the answer on its own line will mark down a system that habitually explains before it answers. A permissive rule that searches the whole response for a matching string will credit a system that mentioned the right value while arguing for the wrong one. Neither is wrong in principle; both change the ranking.

Contamination and the shelf life of a test set

Public benchmarks are published, discussed, quoted and mirrored, which means their items circulate in exactly the kind of text that later training corpora are drawn from. Once that has happened, a score partly measures exposure rather than capability, and it does so invisibly - there is no signal in the output that distinguishes recall from reasoning. This is the strongest practical argument for maintaining a small private set alongside any public numbers you track.

Aggregate scores hide the shape of the failures

A single percentage collapses two very different profiles: a system that is slightly weak everywhere, and a system that is strong across most of the set and fails completely on one category. For a real deployment those are not interchangeable, and which one you have is usually recoverable from a per-category breakdown that the headline figure omits. If the breakdown is published, read it first and the aggregate second.

How to use a published score honestly

Treat it as a filter, not a decision. A published benchmark is good evidence about the rough band a system occupies and poor evidence about how it will behave on your task. Use it to build a shortlist worth spending evaluation time on, then decide with a set you built and can inspect.