Solutions for
Please select
Try the Web App
Back to glossary

Term

Benchmark (AI)

An AI benchmark is a standardized task suite that makes language models comparable — same prompts, same scoring, transparent results for reasoning, knowledge, or coding.

Benchmark (AI) — explained in detail

AI benchmarks consist of fixed task sets with unambiguous scoring. Models receive identical inputs, and their responses are checked against gold solutions (multiple-choice accuracy, test-suite pass rate, judge score). This makes models comparable and forms the basis for leaderboards. The most important benchmarks in 2026 include MMLU (knowledge), GPQA-Diamond (PhD-level reasoning), SWE-bench Verified (coding agents), MATH and HumanEval (code), as well as the Chatbot Arena with Elo rankings from real user votes.

Example / Practical use

When selecting a model for a project, teams typically look at a mix: GPQA for reasoning, SWE-bench Verified for engineering tasks, Arena Elo for general answer quality. MMLU has lost its discriminating power because top models now score above 90 percent — the benchmark has “saturated”. Newer tests such as HLE (Humanity’s Last Exam) and LiveCodeBench are replacing it as differentiators.

A benchmark is static and curated once; an eval suite (see Evals) is typically project-specific and continuously adjusted. A leaderboard aggregates benchmark results but is not a benchmark itself. Risks: benchmark contamination (training data contains test items) and overfitting to popular suites — hence the move to verified variants and private hold-out sets.

More on this topic

See everything in one place:Evaluation
Encyclopedia · 05 Sept 2026Agent Evaluation — Task Success Rate, Eval Harnesses, BenchmarksHow to evaluate AI agents: eval harnesses, task success rate, benchmarks like GAIA and Tau-Bench, LLM-as-a-judge — and how it differs from pure model quality.Encyclopedia · 03 May 2026Measuring LLM Quality — Evals, Benchmarks, JudgesHow to tell whether an LLM system actually works: three evaluation layers from benchmarks to CI evals, plus pitfalls like Goodhart and judge bias.News · 09 May 2026Apple's 'Illusion of Thinking' — what's left of it one year later?A year ago Apple published its reasoning critique 'Illusion of Thinking'. We look at which findings held up and which were refuted.