--- title: "aimodelscompare" url: https://aimodelscompare.com/ description: "Dated working notes on how language models are compared: fixing the task, building a small evaluation set, reading benchmark scores, context windows, latency and cost." --- # aimodelscompare ## Latest notes on comparing models MONDAY 17 AUGUST 2026 17 AUG 2026 · 09:20 · EVALUATION ### [How to compare two language models without fooling yourself](https://aimodelscompare.com/notes/how-to-compare-language-models/) A comparison only means something once the task is fixed. Write down the input you will send, the output shape you will accept and the judgement you will apply, then hold all three constant while the model changes. Everything that is easy to vary later - wording, sampling settings, retry behaviour - has to be nailed down before the first request, or the result measures your setup rather than the model. [Read the full note](https://aimodelscompare.com/notes/how-to-compare-language-models/) 17 AUG 2026 · 08:05 · CONTEXT ### [Token counts are not word counts](https://aimodelscompare.com/notes/token-counts-are-not-word-counts/) FRIDAY 14 AUGUST 2026 14 AUG 2026 · 16:40 · BENCHMARKS ### [What a published benchmark score leaves out](https://aimodelscompare.com/notes/reading-a-benchmark-score/) A published score is the output of a pipeline: a dataset, a prompt format, sampling settings, an answer extractor and a scoring rule. Change any one of those and the number moves, often by more than the gap between the systems being ranked. Reading a benchmark well means reconstructing the pipeline before accepting the summary it produced. [Read the full note](https://aimodelscompare.com/notes/reading-a-benchmark-score/) 14 AUG 2026 · 11:15 · BENCHMARKS ### [A tie is a result, not a failed test](https://aimodelscompare.com/notes/a-tie-is-a-result/) 14 AUG 2026 · 09:00 · EVALUATION ### [Building an evaluation set from fifty examples you care about](https://aimodelscompare.com/notes/building-your-own-evaluation-set/) Fifty examples pulled from work you have actually done will separate candidates better than any public leaderboard, because they carry the input distribution, the failure modes and the acceptance standard that your deployment will face. The cost is a day of collection and a written judgement rule; the return is a test you can rerun every time anything changes. [Read the full note](https://aimodelscompare.com/notes/building-your-own-evaluation-set/) TUESDAY 11 AUGUST 2026 11 AUG 2026 · 15:30 · CONTEXT ### [Context windows: capacity is not attention](https://aimodelscompare.com/notes/context-windows-explained/) A context limit is a capacity figure: the largest amount of input and output a single request may contain. It is not a promise that material placed anywhere inside that span will be weighted equally, and comparing two candidates on their stated limits alone compares the least informative property they publish. [Read the full note](https://aimodelscompare.com/notes/context-windows-explained/) 11 AUG 2026 · 10:10 · PROMPTING ### [Sampling temperature changes what you are measuring](https://aimodelscompare.com/notes/prompt-sensitivity-and-variance/) THURSDAY 6 AUGUST 2026 6 AUG 2026 · 14:25 · COST & LATENCY ### [Latency, throughput and cost move in different directions](https://aimodelscompare.com/notes/latency-throughput-and-cost/) Speed is at least two measurements - how long until the first token appears, and how fast tokens arrive afterwards - and neither is throughput, which is what a system sustains under concurrent load. Cost is a third axis that moves against both. A comparison that reports one number for speed has already lost the trade-off it was meant to expose. [Read the full note](https://aimodelscompare.com/notes/latency-throughput-and-cost/) 6 AUG 2026 · 11:40 · COST & LATENCY ### [Cached prompts distort a latency comparison](https://aimodelscompare.com/notes/cached-prompts-and-latency/) 6 AUG 2026 · 09:45 · DEPLOYMENT ### [What changes when the weights are on your own machine](https://aimodelscompare.com/notes/open-weight-and-hosted-models/)