Module 1 · Designing prompts systematically · Lesson 3
Eval-driven prompt iteration
The loop that turns prompt edits into measured changes - splits, one change at a time, repeated trials, paired comparisons and a changelog with a reason for every line - and how to estimate how much of a dev-set win is just selection noise.
By the end of this lesson you can
- Run a prompt iteration loop with a frozen test split, a dev split and repeated trials, and decide when to promote a version
- Compute the expected inflation of the best of k prompt variants on a small dev set, and explain why the winner's dev score is biased upward
- Keep a prompt changelog that ties every instruction to the failures it fixed, so that later migrations can remove it safely
- Use Claude to propose prompt edits from failure transcripts without letting it grade its own work