Main area
Agency
ki-konzepte

Measuring LLM Quality — Evals, Benchmarks, Judges

Editorial ·

Why LLM evaluation matters

“Is our RAG system any good?” is the only question that really counts — and the one most teams quietly avoid answering. Instead, decisions get made on demo-day vibes, on a screenshot of three good answers, or on public-benchmark scores that don’t actually reflect the team’s problem. Three weeks later, nobody knows why answer quality has tanked in production — because there was never a yardstick in the first place.

LLM evaluation is the answer. It works differently from classical software testing, because a language model’s output isn’t binary right or wrong. An answer can be factually correct but irrelevant. It can sound great and still be hallucinated. It can outperform the previous version in 80 percent of cases and regress catastrophically in the remaining 20. To measure that, you need three layers — public benchmarks, custom evals, LLM-as-a-judge — and clarity about what each layer can and cannot do.

The three layers of LLM evaluation

Layer 1 — Public benchmarks

Public benchmarks are standardized test sets that model providers use to grade their models. They answer the question “How smart is model X in general?” — not “How well does it work for my use case?”. Even so, they’re the natural starting point for model selection.

The big names:

  • MMLU (Massive Multitask Language Understanding): 57 multiple-choice subjects from math to law. Measures factual knowledge and basic reasoning.
  • HumanEval: 164 Python programming problems with unit tests. Measures small-scale code generation.
  • SWE-Bench Verified: 500 real GitHub issues from open-source Python repos. The model gets the issue plus the full repo and has to produce a diff that makes the test suite pass. Measures coding-agent ability on a realistic task.
  • GPQA: Graduate-level questions in physics, chemistry, biology — designed so Google won’t help.
  • Chatbot Arena (Arena): Crowd-sourced pairwise comparison where users vote between two anonymous model answers. Output is an Elo rating. Measures human preference rather than factuality.

What public benchmarks don’t do: tell you anything about your concrete use case. A model with 90 percent on MMLU can be terrible on your support tickets, because your domain vocabulary wasn’t part of MMLU’s training distribution.

Layer 2 — Custom evals

Custom evals are the tool most teams need and few build. The core idea: a fixed dataset with expected outputs (or at least expected properties) that every model or prompt change runs against.

Build:

  1. Dataset — 50 to 500 real cases from your domain. Don’t curate one-sided: you need easy cases (sanity check), typical cases (volume) and the nasty ones (edge cases that show up every few days in production).
  2. Metrics — one measurable property per case. For classification: accuracy, F1. For free-text answers: keyword checks (“answer contains the phone number”), regex patterns, or LLM-as-a-judge (see Layer 3).
  3. Pipeline — automated run that pushes dataset × model version through and persists results. Ideally a CI job that surfaces the diff against the previous version on every prompt PR.

The jump from “20 manual tests in a notebook” to “CI eval runs on every merge” is the most important maturity step. Without it, every model or prompt change is a gut call.

Layer 3 — LLM-as-a-judge

For free-text answers, manual scoring doesn’t scale. The fix: a second LLM scores the first model’s outputs. The judge gets the question, the answer and a rubric as a prompt and returns a score.

It works surprisingly well — and surprises you with how many bias classes it has. LLM-as-a-judge is not a free lunch.

Pitfalls — where evaluation breaks down in practice

Benchmark saturation and data contamination

Public benchmarks age fast. Frontier models in 2026 sit at 97–99 percent on MMLU — the numbers don’t separate models any more. Worse: studies show that MMLU questions appear verbatim or near-verbatim in modern training corpora, inflating scores by an estimated 8–15 points. HumanEval has been public since 2021 — a model scoring 92 percent there says little about how it handles novel code. An audit in early 2026 found that models with 92 percent HumanEval scores produced production-ready code on new enterprise codebases only 61–74 percent of the time.

The takeaway: frontier comparisons need contamination-resistant benchmarks — LiveCodeBench with rolling questions, SWE-Bench Verified with real issues, Chatbot Arena with human preference. The classics still work as sanity checks at the lower end of the model spectrum, but not for “which of the top three models do we ship”.

Judge bias

LLM-as-a-judge has three documented bias classes:

  • Position bias: in A/B comparisons the judge systematically favours the answer in a particular position (often the first or the last). Mitigation: run every comparison twice with positions swapped.
  • Verbosity bias: the judge rewards longer, more structurally complex answers — even when the actual content is thin. A concise correct answer loses to a sprawling correct answer. Mitigation: length constraints in the judge prompt, or length normalisation.
  • Self-preference bias: a judge model favours outputs that resemble its own style — the same model used as judge and generator scores disproportionately well. Research traces this back to lower perplexity on stylistically related text. Mitigation: don’t use the same model for generation and judging; ideally use multiple judges from different model families.

A 2026 RAND study found that no judge is reliably accurate across benchmarks — frontier models exceeded 50 percent error rates on challenging bias benchmarks. So: judge scores are an indicator, not ground truth.

The Goodhart effect

“When a measure becomes a target, it ceases to be a good measure.” The moment you have an eval set and start optimising against the score, the system drifts toward the score — not necessarily toward real quality. A prompt that reliably pleases the judge is not automatically a prompt that reliably pleases real users. Mitigation: split your eval set into “dev” (for iteration) and “holdout” (for occasional reality checks), and periodically sample real user logs.

Eval aging

Eval datasets get stale. New product features, new domain terms, new use cases appear — the eval still represents the old world. Rule of thumb: refresh 10–20 percent of the set quarterly with new cases, and retire old cases that no longer cover an edge.

RAG-specific — the RAGAS metrics

RAG systems have an extra problem: two components can fail independently — the retriever finds the wrong documents, or the generator misinterprets the right ones. Generic “is the answer good?” metrics can’t tell you which part broke.

The RAGAS framework addresses this with four core metrics:

| Metric | What it measures | Component | |---|---|---| | Faithfulness | Are all claims in the answer supported by the retrieved context? | Generator | | Answer Relevancy | Does the answer actually address the question? | Generator | | Context Precision | Are relevant chunks ranked high, not low? | Retriever | | Context Recall | Do the retrieved chunks cover all aspects of the question? | Retriever |

This separation lets you localise problems: low faithfulness → the model is hallucinating or drifting from context. Low context recall → the retriever isn’t finding enough. Low context precision → the retriever is drowning the generator in noise.

RAGAS ships further metrics (Context Entities Recall, Noise Sensitivity, Response Relevancy) but the four above are the pragmatic entry point. One thing to note: RAGAS itself uses LLM-as-a-judge under the hood — every Layer-3 caveat still applies.

From first test to CI eval — a maturity ladder

| Stage | What’s in place | What’s missing | Next step | |---|---|---|---| | 0 — Vibes | A demo run, three screenshots | Reproducibility | 20 real cases in a spreadsheet | | 1 — Manual eval | 20–50 cases run by hand on every change | Time, inconsistency | Script that automates model × dataset | | 2 — Automated eval | Eval script runs on demand, results as JSON/CSV | Output scoring | Add metrics (regex, keywords, LLM-as-a-judge) | | 3 — CI eval | Eval runs automatically on every prompt/model PR, blocks regressions | Production reality | Periodically sample real user logs into the set | | 4 — Production-aware eval | Set is continuously refreshed with real cases, drift is monitored | — | Maintain pipeline, rotate set |

Most teams sit at stage 0 or 1 and skip the next stages, justifying it with “our outputs are too open-ended to test”. That’s almost always wrong — open-ended outputs are scoreable, just not with assertEqual.

FAQ

Do I need custom evals if I'm just using the OpenAI API?
Yes, the moment you start iterating on prompts or switching models. Public benchmarks won't tell you whether your prompt still handles your edge cases after the last tweak.
How big should my eval dataset be?
50 cases is enough for early iterations. 200–500 is solid for CI. Larger only helps if cases bring real diversity — 5,000 variations of the same pattern don't.
Which model should I use as judge?
Ideally a different one from what you're evaluating — and feel free to use a cheaper one. Judge calls add up. A Sonnet judge on Haiku outputs is a reasonable default. Avoid self-judging.
What does running evals cost?
On a 200-case set with judge scoring per case, expect roughly 20–80 cents per run depending on the judge model. Ten runs a day adds up to a few euros a month — negligible against the engineering time you save.
Isn't Chatbot Arena enough as an eval?
No. Arena Elo tells you which model humans prefer on average. It tells you nothing about how a model performs on your data. Arena is for model selection; custom evals are for model operation.

Conclusion

Anyone running LLM systems in production needs custom evals. Public benchmarks help with the initial model pick, but they don’t measure your problem. LLM-as-a-judge scales scoring but comes with documented bias classes you have to know. For RAG systems, RAGAS metrics add the crucial separation between retriever and generator failures.

A pragmatic on-ramp: drop 50 real cases into a spreadsheet, three weeks later wrap them in a script, three months later have a CI eval that diffs every prompt merge against the previous version. That puts you in the league where model and prompt changes are defensible rather than guessed.

And keep Goodhart in mind: the moment you optimise a score, you optimise the score, not reality. Hold out a separate dataset, sample real user logs, occasionally read the outputs with your own eyes. Eval frameworks are tools, not truth.

Themenuebersicht

KI-KonzepteEvaluation