Building an evaluation set from fifty examples you care about

Collect from the record, not from imagination

The examples should come from work that already happened: logged requests, past tickets, documents you have processed, questions colleagues actually asked. Invented examples drift toward the clean middle of the distribution, and the clean middle is where every candidate looks fine. Real inputs bring the truncated pastes, the mixed languages, the internal abbreviations and the person who asked three questions in one sentence.

Fifty is a practical floor for seeing a large difference and a practical ceiling for scoring by hand in an afternoon. Weight the set roughly the way reality is weighted, then deliberately over-sample the cases that would be expensive to get wrong, and mark those separately so you can report them on their own as well as inside the total.

Write the judgement rule down before scoring anything

The rule is the part that decays if it is left in your head. Write it as a short list of conditions an acceptable output must meet and a short list of things that make an output unacceptable regardless of the rest. Prefer binary acceptable/unacceptable over a five-point scale: graded scales feel more informative and in practice mostly record how generous the scorer was feeling.

Then test the rule on ten examples before you use it on fifty. If applying it twice gives you different answers, the rule is underspecified and the whole evaluation will inherit that noise.

Freeze, version and rerun

Once the set is built, freeze it. Store the inputs, the fixed prompt, the settings and the rule together, with a version number and a date. The value of the set compounds with reuse: rerunning the identical set after a version change, a prompt edit or a retrieval change turns a vague impression that something got worse into a countable difference.

Keep a small holdout of examples you never look at while tuning prompts, and check it only when you think you are finished. Prompts get fitted to the examples in front of you, quickly and without anyone intending it, and the holdout is the only cheap way to notice.