How to compare two language models without fooling yourself

Start with the task, not the shortlist

The common failure in model comparison is ordering the work backwards: two or three candidates are chosen first, a handful of prompts are typed into each, and a preference emerges from whichever answers happened to read well that afternoon. The preference is real, but it is a preference about those particular prompts on that particular day, and it does not survive contact with the work the system is actually meant to do.

The alternative costs an hour. Write the task down as three separate statements. First, the input: what exactly arrives, in what format, at what typical and worst-case length. Second, the output contract: what shape the answer must take, whether it must be parseable, what must never appear in it. Third, the judgement: how you will decide that a given output is acceptable, stated clearly enough that a second person applying it would agree with you most of the time.

Only once those three exist does the shortlist matter. The task statement is what makes the comparison portable: it can be rerun in six months against whatever is available then, and the results will still be commensurable with today's.

Hold everything else constant

A model comparison is a controlled experiment with one variable, and language model systems offer an unusual number of ways to lose control of it. The prompt template must be byte-identical across candidates, including whitespace and the order of any examples. Sampling settings must be fixed and recorded. Retry and fallback logic must be off, because a silent second attempt turns a failure into a success without telling you. Any tool access, retrieval step or system instruction must either be present for every candidate or absent from all of them.

Where a candidate genuinely cannot accept the identical prompt - a different instruction format, a different way of requesting structured output - record the deviation explicitly rather than quietly adapting the prompt. A comparison with one documented deviation is usable. A comparison where each candidate got the prompt that suited it best is a comparison of your prompt-writing.

Judge blind, and judge twice

Whoever scores the outputs should not know which candidate produced them. This is not a formality. Expectations about a particular system leak into borderline judgements, and borderline judgements are most of the interesting ones. Shuffle the outputs, strip the labels, and keep the mapping in a separate file until scoring is finished.

Score a subset twice, at least a day apart, and compare your own two passes before comparing the candidates. If you disagree with yourself on a meaningful share of the examples, the judgement criterion is too loose, and no difference between models will be readable through that much noise. Tighten the criterion and rescore; do not average the disagreement away.

Report the comparison, not the verdict

A useful write-up records the task statement, the number of examples, the fixed settings, the judgement rule, the date, and the raw counts of acceptable and unacceptable outputs per candidate. A verdict without those is unfalsifiable and expires quietly the next time any candidate is updated. The counts, by contrast, can be recomputed, disputed and extended, which is the whole point of running the comparison in the first place.