Term
Evals
Evals are systematic tests for LLM applications — fixed test cases, automatic scoring of the responses, result as a score. The foundation for objectively comparing prompt or model changes.
Evals — explained in more detail
Anyone building an LLM application knows the problem: a tiny prompt tweak improves one example and breaks three others — without test coverage, that stays invisible. Evals solve this by running a curated set of input/expectation pairs against the system and scoring each response. Scoring can be deterministic (exact match, regex), structural (valid JSON schema?), via LLM-as-a-judge (another model assesses) or hybrid. Individual scores aggregate into a single number that is compared across builds.
Example / Practical context
OpenAI Evals, Promptfoo, Langfuse, Braintrust and LangSmith offer frameworks for this; Anthropic provides its own tooling with anthropic-evals and the Workbench. A typical dataset for a support bot: 50–500 sample tickets with expected tags, tonality criteria and escalation flags. CI pipelines can run eval suites on every pull request and block a merge on score regression.
How it differs from related terms
Public benchmarks like MMLU, HumanEval or Terminal-Bench measure general model capabilities — evals, by contrast, test the concrete application with its own prompt stack. Unit tests deterministically verify a function; evals tolerate variance in the response and measure statistically across many examples. Observability tools collect live data that often feeds back into eval datasets.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarRecall
Recall (hit rate, sensitivity) measures the share of all actually existing relevant cases that a system finds. Formula: relevant cases found divided by all relevant cases that truly exist. Requires a known ground truth.
LexikonMeasuring LLM Quality — Evals, Benchmarks, Judges
How to tell whether an LLM system actually works: three evaluation layers from benchmarks to CI evals, plus pitfalls like Goodhart and judge bias.