Wire a judge into your eve evals

An eve enquiries agent with two judge variants for one failure mode, redundant enquiries. Variant A shells out to claude -p on your subscription and records the verdict with t.score; variant B asks the same question with t.judge through AI Gateway and skips cleanly without credentials. Five stubbed tasks, critiques saved in the eval artifacts.

Back to the exercise

Before you start

Before you start:Durable agents with eve Get an OpenRouter key Local models with Ollama Install Claude Code

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

The agent drafts enquiries; the eval stubs read_document with a synthetic pack, takes the structured draft and judges each enquiry separately against the TA6 and the other documents. Variant A is the one you can always run: claudeJudge() returns { score, message, metadata }, so the critique lands in eve's artifacts next to the score. Variant B is the same criterion through t.judge, which needs AI Gateway credentials and returns a probability with no rationale. The decision that matters most is making unknown and a missing verdict never count as a pass: unknown scores 0 and is flagged for a human, and a judge that returns nothing throws, which errors the assertion.

Answers Lesson 4: LLM-as-judge

Step-by-step answer

Stretch 60 min8 steps