Module 3 · Benchmarks you can trust · Lesson 8
Item analysis, difficulty and time horizons
Treat a benchmark as a test: fit item response models to a configs-by-tasks success matrix, find saturated, broken and useless tasks, choose smaller subsets that keep the ranking, and see how 50% time horizons are computed and where they stop being reliable.
By the end of this lesson you can
- Fit Rasch and 2PL item response models to a configs-by-tasks matrix of trial outcomes, and interpret ability, difficulty and discrimination
- Flag saturated, all-fail, flat and negatively discriminating tasks, and name the evidence that tells a broken task from a hard one
- Choose a smaller task subset, estimate full-suite scores from it, and measure its ranking fidelity on held-out configurations
- Compute 50% and 80% time horizons from a logistic fit of success on log human time, with a bootstrap interval, and explain where the measure breaks