Solutions for
Please select
Try the Web App
ki-konzepte

Understanding RAG — From Idea to Production System

Redaktion ·

Why RAG — and why not just bigger context windows

An LLM expected to know everything ever written inside your company fails on three points at once: the model wasn’t trained on your data, the context window is finite, and a fine-tune is expensive and ages with every new document. Retrieval-Augmented Generation flips the context-window trick on its head: instead of training everything into the model or stuffing everything into the prompt, at inference time the relevant material is pulled from a collection and only those excerpts go into the prompt.

Since 2023/2024, this has been the de-facto standard for “LLM on your own data” — internal support bots, doc search, research assistants. This article walks the path from the bare pipeline to the optimizations that make the difference in production, plus a worked cost example showing where the money actually goes.

The base mechanics — five steps, one loop

A minimal RAG pipeline has five stations. Once you can see them clearly, every advanced technique slots underneath.

1. Indexing — documents into vector space

Before the first query, your corpus has to be processed once. Texts are split into pieces — see Chunking — and every chunk is run through an embedding model. Out comes a vector per chunk, typically 768 to 3,072 dimensions, a numeric fingerprint of the text’s meaning. Details in the glossary entry Embedding.

These vectors land in a vector database, together with the original text and any metadata (source, date, author, tags). Indexing happens once at load time and again on every update — not a runtime step but a batch job.

2. Query embedding

When a user asks a question, the question goes through the same embedding model as the documents. Crucial point: the same one — vectors from two different models are not comparable. Out comes a single query vector.

3. Retrieval — finding the top K

The vector DB now answers a simple question: “Which K vectors lie closest to my query vector?” The usual distance metric is cosine similarity (two vectors are more similar the smaller the angle between them). Typical K values: 5 to 50.

This is semantic search: hits often don’t contain the literal search term but texts that are similar in meaning. “How do I cancel my subscription?” will find a passage that doesn’t say “cancel” but says “terminate the contract.”

4. Reranking — from coarse sieve to fine sieve (optional but powerful)

The top-K from vector retrieval is a fast but coarse pre-selection. A reranking model — usually a cross-encoder like Cohere Rerank or a local bge-reranker — is shown the query and each of the K hits as a pair and scores actual relevance. Top-50 becomes top-5, and only those go into the prompt.

Rerankers are slower (they run through the model once per hit, not just a vector comparison), but they deliver markedly better orderings. In practice, reranking is the lever that turns naive RAG from “okay” into “usable.”

5. Generation — assemble the prompt, the LLM answers

The final top-N chunks are stitched together with the original question into a prompt, typically with a system instruction along the lines of “Answer only based on the following context passages. If the information isn’t there, say so.” The LLM generates the answer.

Once you can see this as a loop — query → embedding → retrieval → rerank → prompt → answer — you have the skeleton of 90% of all production RAG systems.

Pitfalls almost everyone hits

The pipeline sounds straight, but the spots where you least expect to trip are the most expensive ones.

Chunk size is not a detail. Chunks too small (50 tokens) tear sentences and arguments apart, chunks too large (2,000 tokens) dilute embeddings because a single vector has to encode too many topics. Rule of thumb: 200–500 tokens with 10–20% overlap between chunks. Anyone indexing tables, lists, or code often needs custom splitters — markdown-heading-based or recursive-character splitters.

Switching embedding models without reindexing. If you move from an old embedding model to a new one, you have to rebuild the entire index. Vectors from two models are incompatible — mixed mode produces silent garbage.

Query and document look too different. Users ask short questions (“Vacation days part-time?”), documents are long and bureaucratic. The embedding of the question then often lies far from the answer in vector space, even though semantically they match. Fix: query expansion or HyDE (next section).

Top-K too small. Pulling only top-3 without reranking regularly misses the right hit. Better: pull top-30, rerank to top-5.

Hallucinations despite context. Even when you tell the model explicitly “only from context,” it sometimes fills in. Mitigations: mandatory citations in the prompt, faithfulness eval (see below), stricter models (Claude Sonnet/Opus tend to be more disciplined than smaller open-source models in our observation).

Advanced techniques — what you need when naive isn’t enough

When the baseline doesn’t deliver, a handful of levers tend to make the difference in production systems.

Pure semantic search struggles with exact terms — product names, version numbers, IDs. Hybrid search combines vector retrieval with classical keyword retrieval (BM25) and merges results, often via Reciprocal Rank Fusion. If your corpus has lots of proper names or technical terms, hybrid is almost always an improvement.

Query expansion

Instead of the original question, additional generated variants (“vacation entitlement part-time,” “pro-rata vacation days,” “how is leave calculated under reduced hours”) are run against the index and hits are merged. Costs an extra LLM call but measurably helps with terse user queries.

HyDE (Hypothetical Document Embeddings)

The trick: instead of embedding the question itself, you let the LLM first generate a hypothetical answer and embed that. The hypothetical answer sits closer to real answers in vector space than the question does. Particularly effective when question style and corpus style diverge.

Contextual retrieval

Anthropic’s variant (2024): before embedding, every chunk gets a short LLM-generated context summary prepended (“This passage is from the HR handbook, vacation policy chapter, and describes …”). According to Anthropic’s study, retrieval error rates drop by double-digit percentages. Cost: indexing gets heavier (every chunk runs through the LLM once), runtime stays the same.

GraphRAG

When your knowledge is networked (people, organizations, relationships) and questions need multiple hops (“Which subsidiaries of X work on project Y?”), pure similarity search often falls short. GraphRAG additionally builds a knowledge graph from the corpus and combines graph traversal with vector retrieval. Effort: significantly higher (entity extraction, graph construction, hybrid querying). Worth it for networked knowledge domains, overkill for FAQ bots.

Agentic RAG

Instead of a fixed pipeline (retrieve → generate), an agent gets retrieval as a tool and decides itself whether, how often, and with what reformulation to search. In complex research tasks it can resolve multi-step reasoning that a one-shot RAG misses. Trade-off: higher latency, higher cost, harder to debug — and agents have their own pitfalls (infinite loops, tool hallucinations).

Vector databases compared

Three options cover 95% of cases.

| Tool | Type | Strengths | Weaknesses | When to pick | |---|---|---|---|---| | Pinecone | Managed SaaS | Zero ops, fast, scales well | Cost, vendor lock | Quick start, small team, no DBA | | Qdrant | Self-hosted (Rust) | Open source, performant, native hybrid search, cloud variant available | Own infra, backups on you | Mid-scale, on-prem requirement | | pgvector | PostgreSQL extension | Relational data + vectors in one DB, simple ops | Slower on very large indexes (> 10M vectors) | Existing Postgres footprint, < 5M chunks |

Beyond these: Weaviate (similar to Qdrant, with built-in hybrid and schema system), Milvus (very large scale, more complexity), and for quick starts also Chroma and LanceDB.

Rule of thumb: start with pgvector if you already run Postgres and stay under a few million chunks. Move to Qdrant or Weaviate once native hybrid search matters or volume grows. Pinecone if ops effort is the deciding factor and budget isn’t.

Worked example — naive vs. optimized pipeline

So the cost side doesn’t stay abstract, a concrete comparison for 10,000 queries per month. Assumptions: embedding model text-embedding-3-small ($0.02 / 1M tokens), generation with gpt-5-mini ($0.15 / 1M input, $0.60 / 1M output), average question 50 tokens, chunk size 400 tokens, answer 300 tokens.

Naive pipeline — top-3 without reranking:

| Item | Tokens | Cost | |---|---|---| | Query embeddings | 10,000 × 50 = 500k | $0.01 | | Generation input (question + 3×400-token context) | 10,000 × 1,250 = 12.5M | $1.88 | | Generation output | 10,000 × 300 = 3M | $1.80 | | Total | | $3.69 |

Reranking-optimized — pull top-30, rerank to top-5 (Cohere Rerank: $1 / 1k requests), then generate:

| Item | Tokens / Calls | Cost | |---|---|---| | Query embeddings | 500k tokens | $0.01 | | Reranking | 10,000 calls × 30 docs | $10.00 | | Generation input (question + 5×400-token context) | 10,000 × 2,050 = 20.5M | $3.08 | | Generation output | 3M | $1.80 | | Total | | $14.89 |

Reranking quadruples cost — and in almost every production test it produces measurably better answers (typically +15 to +30 percentage points on faithfulness and context precision). At 10k queries, that ~$11 delta is usually cheaper than a second support agent fixing wrong answers. At 10M queries the math shifts — that’s where a local reranker pays off.

Measuring quality — quick primer

RAG without eval is hope with extra steps. Three metrics from the RAGAS framework are enough to start:

  • Faithfulness — does the answer actually rest on the retrieved context, or is the model hallucinating? Measured by a judge LLM checking each statement in the answer against the context.
  • Context precision — are the retrieved chunks really relevant to the question, or is there a lot of noise? Measures how high in the top-K the actual hits sit.
  • Context recall — are relevant chunks missing from retrieval? Requires a gold standard (which chunks would be the right ones), so usually measured on a curated eval set.

Eval setups deserve their own article — dataset hygiene, judge bias, Goodhart effects. Here the takeaway is: before you bolt on reranking, HyDE, or GraphRAG, you need an eval metric, otherwise you can’t tell whether the optimization is actually better or merely different.

Conclusion — what to do first, what can wait

The shortest path to a usable RAG: pgvector + top-30 + Cohere Rerank + top-5 into the prompt + faithfulness eval on 50 manually curated questions. That isn’t a day-one setup, but two weeks for a small team. From there you can decide data-driven which lever to pull next.

HyDE and query expansion are cheap improvements when your users ask in short, keyword-style queries. Contextual retrieval pays off once you can absorb the indexing overhead and hallucinations are your main problem. GraphRAG and agentic RAG are specialty tools — wire them in when you can demonstrably show the naive pipeline hitting its limits, not preemptively.

What you do not need before the baseline stands: your own embedding model, an exotic vector-DB swap, a homemade reranking architecture. The biggest jumps in RAG quality come from chunking, reranking, and evaluation — not from the model zoo.

Themenuebersicht

KI-KonzepteRAG