Term
LLM-as-a-Judge
LLM-as-a-Judge is the practice of using a language model as an evaluator — it compares responses from other models or scores outputs against given criteria.
LLM-as-a-Judge — explained in detail
Instead of having humans grade every model output, a second, often stronger LLM takes the role of referee. It receives a task, the response under review, and a scoring rubric — for example, “rate factual faithfulness from 1 to 5” or “which of the two responses is better?”. Typical variants: single grading (one response, absolute score), pairwise comparison (two responses, pick a winner), and reference-based judging against a gold answer.
Example / Practical use
In RAG pipelines, judges automatically score faithfulness, answer relevancy, and context precision across thousands of examples. Frameworks like RAGAS, DeepEval, and Promptfoo run judges in production. Chatbot Arena also uses a judge model as a pre-filter. Rule of thumb: a judge should be at least one model class stronger than the system under test, otherwise its reliability drops sharply.
Distinction from related terms
LLM-as-a-Judge is a tool inside an eval pipeline, not a replacement for human evaluation — judges inherit training-data bias and tend toward position bias (favoring the first response) and length bias (favoring longer answers). Mitigations: calibrate the judge, use a jury of multiple models, or validate against human samples. Distinction: a classic benchmark has objective gold answers, while a judge can also score open-ended responses.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarRecall
Recall (hit rate, sensitivity) measures the share of all actually existing relevant cases that a system finds. Formula: relevant cases found divided by all relevant cases that truly exist. Requires a known ground truth.
LexikonMeasuring LLM Quality — Evals, Benchmarks, Judges
How to tell whether an LLM system actually works: three evaluation layers from benchmarks to CI evals, plus pitfalls like Goodhart and judge bias.