Quillmoor's support assistant drafts replies for 14 support agents and answers questions in the help-centre chat, about 5,000 conversations a week. Prompt v4 is live, v5 is a candidate, and v3 is the version v4 replaced. The senior support lead has been the only expert anyone trusts, and one person can't keep up. The team needs to know how good the assistant is this week, whether v5 should ship, which user signals mean anything, what customers ask about, and where review time should go. This project builds the system that answers those questions and leaves you with a weekly routine you could run on real traffic.
It uses every module. From human judgement at scale: a guideline, a multi-rater pilot measured with Krippendorff's alpha, and a Bradley–Terry ranking from blind pairwise votes. From signals from real use: feedback events validated against the lead's labels and corrected for who chooses to rate, and a usage map built without anyone reading a conversation. From closing the loop with few labels: a prediction-powered estimate from a judge on all traffic plus 150 human labels, and the review queue and dataset promotion that keep those labels coming.
Every dataset is synthetic and generated by a script with the truth planted in it (a careless rater, a gap in the guideline, a judge that is worse on one topic, participation that depends on mood, one identifiable cluster), so you can check every estimate against the answer. Model calls go through Claude Code in headless mode (claude -p) on your Claude subscription: about 55 calls on Haiku, all in the usage map. Expect 8–12 hours.