Module 1 · Foundations of evaluation · Lesson 3

Code-based graders

Grade everything that can be checked mechanically, from schemas and tool calls to grounded quotes, approval gates and final state, using eve evals, tool stubs and Claude Code headless runs.

75 min 1 video 3 exercises 7-question quiz Answers

Setup for the exercises:Install Claude Code Durable agents with eve Get an OpenRouter key Local models with Ollama Node.js, pnpm and TypeScript

By the end of this lesson you can

Design agent outputs so that the important properties can be checked with code
Write eve evals with defineEval, scoped tool assertions, value assertions, t.score and t.require, and choose gate or soft severity for each
Use tool stubs to serve fixtures, inject tool errors and keep side effects from running
Assert that a high-stakes side effect is waiting for human approval rather than executed
Grade a Claude Code headless run from its structured output and event stream, and test the graders themselves

Most of what can go wrong with an agent can be checked by a few lines of code. Did the output validate? Did it read the official copy before drafting? Does every evidence quote actually appear in the document it cites? Is the call that would send enquiries to the seller's solicitor waiting for a solicitor's approval, or did it go out? These are mechanical facts, and a code-based grader checks them deterministically, in milliseconds, for free. This lesson writes them, mainly in eve, with the Claude Code equivalent at the end.

Code-based graders

Grade everything that can be checked mechanically, from schemas and tool calls to grounded quotes, approval gates and final state, using eve evals, tool stubs and Claude Code headless runs.

Code-based graders

Code-based graders

Sign in to keep reading.

Sign in to keep reading.