Module 1 · Simulated worlds · Lesson 3
Validating user simulators
Measure how far a simulated user is from real ones and how much a score depends on the simulator model, then attribute every trial to the agent, the simulator, the environment or the grader before reporting a number.
By the end of this lesson you can
- Name the documented failure modes of LLM user simulators and predict their effect on reported success
- Measure simulator sensitivity by swapping the simulator model, and judge whether the spread is larger than the noise
- Compare human-played and simulated conversations on behaviour features and outcomes
- Attribute each trial's outcome to the agent, the simulator, the environment or the grader, and report raw and corrected success
- Validate a simulator-adherence judge against your own labels before using it to filter trials