Module 1 · Human judgement at scale · Lesson 3
Pairwise preferences and rankings
Turn many blind A/B votes between versions into a ranking with honest intervals using a Bradley–Terry model, separate substance from length and style, and see how public leaderboards mislead.
By the end of this lesson you can
- Fit a Bradley–Terry model as a logistic regression on pairwise votes, handle ties, and convert a rating difference into a win probability
- Explain why online Elo depends on vote order and K, and put confidence intervals on ratings and on the difference between two versions
- Add style covariates such as length to the regression and decide when controlling for them is right
- Decide which pairs to collect next, and critique a public leaderboard for private testing, sampling imbalance, rigging and sensitivity to a handful of votes