A full eval suite for your drafting agent

A complete eve evaluation suite for the pre-contract enquiries agent. An error-analysis taxonomy turned into 34 generated tasks (a third of them negative), pure code graders checked against planted faults, a claude -p redundancy judge calibrated against 80 of your labels, repeated trials with clustered intervals, unbiased pass^3 and a paired comparison, and a two-tier CI gate that catches a broken prompt.

Back to the project

The suite is one eve project with four layers, each built on the one before. Tasks come from the error analysis, one or more per failure mode, generated so they're reproducible. Code graders are pure functions that run inside eve eval and in a plain script, so you can prove each one rejects the mistake it targets. The judge handles the one failure mode code can't check, and it is only trusted as far as its calibration on your labels allows. Statistics and CI turn repeated runs into numbers with intervals and into a merge gate. The decision that matters most is the order: graders and calibration before statistics, because an interval around a broken grader is a precise measurement of nothing. Every document is synthetic; the packs, names and title numbers are invented.

Back to the project

A full eval suite for your drafting agent

A full eval suite for your drafting agent

Sign in to keep reading the answer.

Sign in to keep reading the answer.