Module 1 · Decisions, not text · Lesson 2
Calibrated probabilities
What it means for a model's probabilities to be calibrated, how to measure it with reliability diagrams, ECE and the Brier score, why modern networks and LLMs are often overconfident, and how Jev's confidence values are computed from its probabilities.
By the end of this lesson you can
- Define calibration as a property of groups of predictions and distinguish it from accuracy and discrimination
- Build a reliability diagram and compute expected calibration error and the Brier score by hand for a small set of predictions
- Explain why modern neural networks and LLMs' verbalised confidence tend to be overconfident, and what temperature scaling does
- Compute Jev's confidence for Noul, Choice and Score answers from their probabilities, and explain why confidence is not correctness