Step-by-step answer
Stretch 60 min7 stepsPick a t.judge threshold and correct a reported rate
Two ways to finish calibrating a judge. Part A runs one t.judge boolean question over your labelled dev items with no threshold, reads the scores from eve's JSON and sweeps thresholds 0.1 to 0.9 to pick one for a flagging judge, then confirms it on test. Part B corrects a fresh batch's observed pass rate with Rogan-Gladen and puts a 2,000-resample bootstrap interval on it.
Back to the exercise