Agreement and disagreement among many raters

The pilot from Designing human evaluations has grown into a real labelling round. Four of Quillmoor's support agents have labelled 240 tickets, each ticket by three of them, with two labels: the ticket's category (billing, refund, account-access and six more) and the draft's editing effort on a four-step scale (0 send as is, 1 minor edit, 2 major edit, 3 discard). Before anyone uses these labels to calibrate a judge or count failures by category, you need to know whether they are reliable. Calibrating judges answered that for one judge against one expert with Cohen's kappa, TPR and TNR. None of its assumptions hold here: there are four raters, not two; every ticket is missing one of them; and the effort scale is ordered, so a 0-versus-3 disagreement is worse than a 0-versus-1. This lesson measures agreement in that setting, finds where the disagreement comes from, and decides what to do about it.