---
title: "Glossary"
url: https://aimodelscompare.com/glossary/
description: "Twenty-two terms used when comparing language models, from answer extraction to tokenizer, defined in plain language."
---

# Glossary

Reference

The vocabulary the notes assume, defined once so the notes themselves can stay short.

**Aggregate score**: A single figure summarising performance across a whole benchmark. It collapses per-category behaviour, so two systems with the same aggregate can fail in completely different places.

**Answer extraction**: The step in a benchmark harness that pulls a gradeable answer out of free-form text. Strict and permissive extractors can rank the same systems differently.

**Benchmark harness**: The code around a benchmark: prompt template, sampling settings, extraction and scoring. Reported numbers are properties of the harness as much as of the system.

**Blind scoring**: Grading outputs with the identity of the producing system hidden, so expectations cannot leak into borderline judgements.

**Ceiling effect**: The condition where nearly every candidate scores near the maximum, leaving the set unable to resolve differences. The fix is harder examples, not more of the same ones.

**Cold start**: The first run of a request path, before any cache or processed prefix can be reused. Cold and warm timings are not comparable.

**Contamination**: The presence of benchmark items in the material a system was trained on, which turns part of a score into recall rather than capability.

**Context window**: The maximum span of tokens a single request may occupy, shared between instruction, retrieved material, conversation history and the response.

**Determinism**: The property of returning the same output for the same input and settings. Sampled generation is not deterministic by default, and reruns establish the noise floor.

**Evaluation set**: A fixed collection of inputs with a written acceptance rule, held constant so that reruns across versions remain comparable.

**Holdout**: Examples deliberately kept out of sight while prompts are tuned, checked only at the end to detect fitting to the visible set.

**Input distribution**: The statistical shape of the inputs a deployment actually receives, including lengths, languages and malformed cases. Invented examples rarely reproduce it.

**Judgement rule**: The written statement of what makes an output acceptable or unacceptable. If applying it twice gives different answers, it is underspecified.

**Noise floor**: The run-to-run variation of a single candidate on a fixed prompt. Differences smaller than it are not findings.

**Open weights**: Model parameters distributed so they can be run on hardware you control, which fixes the target under test and transfers the serving stack to you.

**Output contract**: The required shape of an answer: format, length bounds, parseability and prohibited content. Stating it before testing keeps candidates comparable.

**Prefix cache**: Reuse of an already-processed shared beginning of a prompt, which reduces latency and cost for repeated prefixes and distorts naive timing tests.

**Quantisation**: Storing weights at reduced numerical precision to lower memory and cost. It is a quality variable, not only a deployment setting.

**Tail latency**: The slow end of the response-time distribution, which determines timeouts and the experience of the least fortunate requests. Means conceal it.

**Temperature**: The sampling setting governing how much randomness enters generation. It changes the spread of outputs and therefore what a comparison measures.

**Time to first token**: The delay before the first piece of a response appears, dominated by input processing and felt directly in any streaming interface.

**Tokenizer**: The learned segmentation that turns text into the tokens a system counts and charges for. Its behaviour on code, digits and non-Latin scripts departs sharply from word counts.

[Back to the notes](https://aimodelscompare.com/notes/)