Reference
Glossary
The vocabulary the notes assume, defined once so the notes themselves can stay short.
- Aggregate score
- A single figure summarising performance across a whole benchmark. It collapses per-category behaviour, so two systems with the same aggregate can fail in completely different places.
- Answer extraction
- The step in a benchmark harness that pulls a gradeable answer out of free-form text. Strict and permissive extractors can rank the same systems differently.
- Benchmark harness
- The code around a benchmark: prompt template, sampling settings, extraction and scoring. Reported numbers are properties of the harness as much as of the system.
- Blind scoring
- Grading outputs with the identity of the producing system hidden, so expectations cannot leak into borderline judgements.
- Ceiling effect
- The condition where nearly every candidate scores near the maximum, leaving the set unable to resolve differences. The fix is harder examples, not more of the same ones.
- Cold start
- The first run of a request path, before any cache or processed prefix can be reused. Cold and warm timings are not comparable.
- Contamination
- The presence of benchmark items in the material a system was trained on, which turns part of a score into recall rather than capability.
- Context window
- The maximum span of tokens a single request may occupy, shared between instruction, retrieved material, conversation history and the response.
- Determinism
- The property of returning the same output for the same input and settings. Sampled generation is not deterministic by default, and reruns establish the noise floor.
- Evaluation set
- A fixed collection of inputs with a written acceptance rule, held constant so that reruns across versions remain comparable.
- Holdout
- Examples deliberately kept out of sight while prompts are tuned, checked only at the end to detect fitting to the visible set.
- Input distribution
- The statistical shape of the inputs a deployment actually receives, including lengths, languages and malformed cases. Invented examples rarely reproduce it.
- Judgement rule
- The written statement of what makes an output acceptable or unacceptable. If applying it twice gives different answers, it is underspecified.
- Noise floor
- The run-to-run variation of a single candidate on a fixed prompt. Differences smaller than it are not findings.
- Open weights
- Model parameters distributed so they can be run on hardware you control, which fixes the target under test and transfers the serving stack to you.
- Output contract
- The required shape of an answer: format, length bounds, parseability and prohibited content. Stating it before testing keeps candidates comparable.
- Prefix cache
- Reuse of an already-processed shared beginning of a prompt, which reduces latency and cost for repeated prefixes and distorts naive timing tests.
- Quantisation
- Storing weights at reduced numerical precision to lower memory and cost. It is a quality variable, not only a deployment setting.
- Tail latency
- The slow end of the response-time distribution, which determines timeouts and the experience of the least fortunate requests. Means conceal it.
- Temperature
- The sampling setting governing how much randomness enters generation. It changes the spread of outputs and therefore what a comparison measures.
- Time to first token
- The delay before the first piece of a response appears, dominated by input processing and felt directly in any streaming interface.
- Tokenizer
- The learned segmentation that turns text into the tokens a system counts and charges for. Its behaviour on code, digits and non-Latin scripts departs sharply from word counts.