Solutions for
Please select
Try the Web App
ki-orchestrierung

Evaluator-Optimizer Loop

Redaktion ·

The evaluator-optimizer loop is really just this: you let one AI agent produce something, and a second one looks at it and says what’s still off. Then it goes back to the first one, which fixes it up. And this back and forth runs in a circle until the result meets a quality bar you set in advance.

The first agent is called the generator or optimizer — it produces. The second is called the evaluator or critic — it judges. In other contexts you’ll see this as “reflection” or “self-critique”. It always means the same pattern: generate, check, refine, repeat.

Why a second agent at all?

A single language model that answers in one go gets no second shot. It writes its text and that’s it. If the first version is mediocre, mediocre is what you keep.

The trick in the evaluator-optimizer loop is that you deliberately split the two roles. The generator gets to be generous and just throw something out there. The evaluator gets to be skeptical and look closely. That small bit of friction — one builds, the other pushes back constructively — often carries the result past a point where a single pass would have stalled.

It sounds like overhead, and for a quick answer it is. But the moment quality matters, it suddenly makes real sense.

How the loop actually runs

The flow is the same in every variant:

  1. Generate. The generator produces a first version — a piece of text, some code, a translation.
  2. Evaluate. The evaluator checks that version against clear criteria and writes down its verdict: what’s good, what’s missing, what needs to change.
  3. Decide. Does the version meet the bar? Then you’re done. If not, the feedback goes back to the generator.
  4. Refine. The generator works the critique in and delivers a new version. Back to step 2.

What matters is that the evaluator doesn’t just say “good” or “bad” but gives concrete, actionable feedback. That’s the only thing the generator can target. A vague “I don’t like it” doesn’t move the loop forward.

When the pattern is worth it

The evaluator-optimizer loop doesn’t fit everywhere. It fits where two things come together:

  • There’s a clear criterion you can pin quality to. Does the code meet the security rules? Does the text hit the right tone? Are the facts right?
  • A second round measurably improves the result. If the first draft already does the job, you save yourself the whole effort.

Typical real-world cases: code that has to pass security standards. Customer communication where the tone needs to land. Translations where fine nuances count. Anywhere a human could clearly put their feedback into words — and the model can produce exactly that feedback itself.

It’s closely related to multi-agent orchestration, where several agents work together. The evaluator-optimizer loop is one of its clearest and most useful flavors.

The stopping condition — the most important part

A loop that never stops is dangerous. So every evaluator-optimizer loop needs a clean stopping condition. In practice you usually combine a few:

  • Quality reached. The evaluator gives the green light — that’s the ideal case.
  • Max rounds. After three or five passes it stops, no matter what. This guards against endless loops when the generator simply can’t hit the bar.
  • No more progress. If two rounds in a row barely improve anything, the next one usually isn’t worth it.

Without these brakes the loop can run forever in the worst case — and every round costs tokens, and therefore money.

The price: more effort, more cost

Honestly, the evaluator-optimizer loop is more expensive than a single call. Each round means at least two model calls — one to generate, one to evaluate. At five rounds you’re quickly at ten times the tokens.

That’s the deal: you trade speed and cost for quality. For a throwaway answer that’s waste. For a result that genuinely has to be right, it’s often exactly the right move.

FAQ

What's the difference between generator and evaluator?
The generator produces the content — it writes, builds, generates. The evaluator judges that content against fixed criteria and feeds back. They're separate roles, often even separate models, so the reviewer brings a genuinely independent perspective.
Do the generator and evaluator have to be the same model?
No. You'll often deliberately use different models, because a different model gives a more independent second opinion. It costs a bit more but produces more honest feedback, since the reviewer doesn't share the same blind spots as the generator.
When should I NOT use this loop?
When there's no clear quality criterion for the evaluator to anchor on. And when the first draft is already good enough — then the whole effort is just wasted time and tokens.
How do I keep the loop from running forever?
With a fixed stopping condition. In practice you best combine three: quality reached, max round count exceeded, or no meaningful progress between two rounds.
Is evaluator-optimizer the same as reflection?
At its core, yes. „Reflection" and „self-critique" describe the same basic pattern — an AI checks and improves its own output iteratively. „Evaluator-optimizer" just puts more emphasis on the evaluator and generator working as two separate roles.

Themenuebersicht

KI-OrchestrierungWorkflow-Automatisierung