Step-by-step answer
Core 60–90 min8 stepsBuild a calibration toolkit and test it on two models
A dependency-free calibration toolkit (Brier, reliability bins, ECE and TypeSafe's three confidence formulas) run on two simulated models whose truth you control. Both make the same calls with 73.3% accuracy, yet ECE is 0.026 for the calibrated one and 0.113 for the overconfident one, which is the gap accuracy can't see.
Back to the exercise