Build and run a 10-case eval

Ten synthetic official copies, expected outcomes as data in cases.json, small code graders, and a runner that calls claude -p with a JSON schema per case and keeps the report, tool-call log and transcript of every trial, so each failure can be read and explained.

Back to the exercise

Before you start

Before you start:Install Claude Code Node.js, pnpm and TypeScript

Put the expected outcomes in data (eval/cases.json) and the logic in small graders, so adding a case never means writing code. Every trial gets the same general checks (schema, title number, quotes in the cited entry, coverage, consistency) plus its case graders, and the runner keeps everything you need to read a failure afterwards. The decisions that matter most are in the expectations themselves: each one has to be something two experts would agree on and code can check.

Answers Lesson 6: Risk and first evals

Step-by-step answer

Core 90 min8 steps

Build and run a 10-case eval

Back to the exercise

Before you start

Before you start:Install Claude Code Node.js, pnpm and TypeScript

Free · sign in to start

Sign in to keep reading the answer.

Free, no card. We use your account to save progress, quiz scores, bookmarks and occasionally send updates from our team.

8 more sections
Progress, quiz scores and bookmarks