Term
RAGAS
RAGAS is an open-source framework for automated evaluation of RAG and agent pipelines — with built-in metrics like faithfulness, context precision, and answer relevancy.
RAGAS — explained in detail
RAGAS (Retrieval-Augmented Generation Assessment) ships ready-made metrics to compare RAG systems objectively — without arranging a manual review for every change. Most metrics use LLM-as-a-Judge: a model checks, given the question, retrieved contexts, and final answer, whether the answer is actually grounded in the contexts (faithfulness) or whether the retriever delivered the relevant chunks (context precision/recall). It plugs into CI pipelines so each change is scored against an eval set automatically.
Example / Practical use
A common setup: build an eval set of question, gold answer, and gold contexts, run the pipeline, and let RAGAS compute per-example scores for faithfulness, answer relevancy, context precision, and context recall. Engineering teams use it to compare retrievers (vector vs. hybrid vs. reranked) or to gate prompt changes. Beyond RAG metrics, RAGAS also covers agent metrics (tool-call accuracy, goal achievement) and classical NLP scores (BLEU, ROUGE, semantic similarity).
Distinction from related terms
RAGAS is a specific library, not a concept — comparable to DeepEval, TruLens, and Promptfoo. The metrics themselves (faithfulness, context precision …) are universal and implemented in several frameworks. RAGAS is also not a benchmark: it is a measurement tool that runs on your own eval data. Weaknesses: LLM judges are expensive and slow, and can become unreliable in narrow domains — so production teams typically combine it with human spot checks.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarRecall
Recall (hit rate, sensitivity) measures the share of all actually existing relevant cases that a system finds. Formula: relevant cases found divided by all relevant cases that truly exist. Requires a known ground truth.
LexikonMeasuring LLM Quality — Evals, Benchmarks, Judges
How to tell whether an LLM system actually works: three evaluation layers from benchmarks to CI evals, plus pitfalls like Goodhart and judge bias.