Module 1 · Foundations of evaluation · Lesson 2
Error analysis from traces
Read agent traces systematically, turn your notes into a failure taxonomy, and build a first balanced dataset of 20–50 tasks from what actually goes wrong.
By the end of this lesson you can
- Find and read eve and Claude Code traces, and identify the first upstream failure in each
- Run open coding and axial coding on a sample of traces to produce a counted failure taxonomy
- Decide, for each failure mode, whether to fix it directly, grade it with code, or grade it with a judge
- Build a balanced 20–50 task dataset with reference solutions, including negative cases, using synthetic inputs when real traffic is scarce