Monday, 17 August 2026
17 AUG 2026 · 09:20 · EVALUATION
A comparison only means something once the task is fixed. Write down the input you will send, the output shape you will accept and the judgement you will apply, then hold all three constant while the model changes. Everything that is easy to vary later - wording, sampling settings, retry behaviour - has to be nailed down before the first request, or the result measures your setup rather than the model.
17 AUG 2026 · 08:05 · CONTEXT
14 AUG 2026 · 16:40 · BENCHMARKS
A published score is the output of a pipeline: a dataset, a prompt format, sampling settings, an answer extractor and a scoring rule. Change any one of those and the number moves, often by more than the gap between the systems being ranked. Reading a benchmark well means reconstructing the pipeline before accepting the summary it produced.
14 AUG 2026 · 11:15 · BENCHMARKS
14 AUG 2026 · 09:00 · EVALUATION
Fifty examples pulled from work you have actually done will separate candidates better than any public leaderboard, because they carry the input distribution, the failure modes and the acceptance standard that your deployment will face. The cost is a day of collection and a written judgement rule; the return is a test you can rerun every time anything changes.
11 AUG 2026 · 15:30 · CONTEXT
A context limit is a capacity figure: the largest amount of input and output a single request may contain. It is not a promise that material placed anywhere inside that span will be weighted equally, and comparing two candidates on their stated limits alone compares the least informative property they publish.
11 AUG 2026 · 10:10 · PROMPTING
6 AUG 2026 · 14:25 · COST & LATENCY
Speed is at least two measurements - how long until the first token appears, and how fast tokens arrive afterwards - and neither is throughput, which is what a system sustains under concurrent load. Cost is a third axis that moves against both. A comparison that reports one number for speed has already lost the trade-off it was meant to expose.
6 AUG 2026 · 11:40 · COST & LATENCY
6 AUG 2026 · 09:45 · DEPLOYMENT
The notes in this river are written to be read independently of any particular model release. Every claim in them is about method — what a comparison must hold constant, what a measurement can support — rather than about the standing of a named system.
Figures are deliberately absent. A published number without the harness that produced it cannot be checked by a reader, so the notes describe how to obtain your own instead. The vocabulary they assume is collected in the glossary, and the scope of the river is set out in about these notes.