A benchmark is a standardized test suite that measures model capability on fixed tasks, letting teams compare scores across models and over time.
How it works
A suite ships prompts with answers or graders, runs each model under identical settings, and publishes aggregate scores with splits by skill. Designers refresh items to fight contamination, report variance across seeds, and pair automatic grading with human review where judgment matters. Scores climb as models memorize or genuinely improve, so a single number never tells which happened without checking the method.
Builders reach for benchmarks whenever a model choice must be defensible: shortlisting candidates on public suites, then confirming with their own evals on production-like tasks before committing.