Module 2 · Scoring that you can trust · Lesson 6

Rubrics per task

Grade open-ended outputs whose must-cover points change with every input, with a rubric of weighted, binary criteria written for each task, and check that rubric criterion by criterion against an expert.

75 min 1 video 3 exercises 8-question quiz Answers

Setup for the exercises:Install Claude Code Node.js, pnpm and TypeScript

By the end of this lesson you can

Write a per-task rubric of weighted positive and penalty criteria, each a binary check of one claim
Compute a rubric score from criterion verdicts, including negative task scores and clipping of the suite mean
Choose between grading each criterion in its own call and all criteria in one call, from cost and measured accuracy
Draft rubrics with a model from a reference answer for an expert to review, and measure each criterion's agreement with expert labels
Recognise stuffed, missing, compound and over-specified criteria, and explain how a policy trained against rubrics exploits them

The enquiries agent has a second job: it drafts the covering email to the buyer: a plain-English summary of what the contract pack shows and what you've asked the seller's solicitor. A solicitor approves it before it's sent. What makes that email good depends on the pack. For pack 0007, the freehold house whose register carries a covenant against any trade or business while the seller works from a home office, a good email explains the covenant and asks whether the buyer plans to run a business there. For a leasehold flat whose ground rent doubles every ten years, the ground rent is the headline. The redundancy judge from LLM-as-judge works because "redundant" means the same thing in every pack; "left out what matters in this pack" doesn't. This lesson grades outputs like that with a rubric written for each task.

Rubrics per task

Rubrics per task

Rubrics per task

Sign in to keep reading.

Sign in to keep reading.