Coders In Hoods

Courses
Answers
Setup
Bookmarks

Coders In Hoods Academy

Every course is free. Sign in to start, and your progress, quiz scores and bookmarks follow you.

Academy

Courses
Paths
Exercise answers
Setup guides
Bookmarks

Coders In Hoods

The agency
Coaching for teams

Social

LinkedIn
X (Twitter)

Contact

codersinhoods@gmail.com

© Coders In Hoods · since 2019

Free · no card

Coders In Hoods Academy

Sign in to start learning

Free, no card. We use your account to save progress, quiz scores, bookmarks and occasionally send updates from our team.

Course overview

System One Models with Jev

Free · sign in to start and track progress

1 · Decisions, not text

1What a System One model is
2Calibrated probabilities

2 · The Jev API

3Your first Jev request
4Choice, Score and Noul
5State and decomposition

3 · Building systems

6Routing on confidence
7Jev alongside LLMs
8Evaluating Jev on your data
Exercise answers
- Answer: Build an eval harness for System One answers
- Answer: Consistency and version regression checks

CapstoneTriage a busy inbox with Jev Reference implementation

Answers Lesson 8: Evaluating Jev on your data

Step-by-step answer

Core 75–90 min6 steps

Build an eval harness for System One answers

An eval harness that sends one Choice per enquiry, saves every answer to JSONL and scores the saved file: accuracy with a Wilson interval, a confusion matrix, ECE, Brier and a risk-coverage table. On 54 synthetic enquiries a word-overlap baseline scores 70.4% with an ECE of 0.47, and Claude Haiku through claude -p scores 92.6% with an ECE of 0.091 and a mixed reliability table.

Back to the exercise

Before you start

Before you start:Python with uv Jev, or its local stand-in Install Claude Code

Separate the run from the scoring. harness.py sends one Choice per item through TypeSafeClient, saves every answer (gold, top answer, top probability, full distribution, model name, tokens, seconds) to JSONL, and prints a report; --from re-scores a saved file with no model call, so metric changes never cost a run. The same harness runs the fake backend (a deliberately dumb word-overlap baseline), the claude backend and, with a key, Jev pinned to a version.

Previous answerSweep a cascade's gate and find the frontier Next answer Consistency and version regression checks

On this page

Before you start
The exercise
1. Create the project
2. Write the eval set and rules
3. Write the metrics
4. Separate the run from scoring
5. Run it
6. Write up the deliverable
Common mistakes
All files

Separate the run from the scoring. harness.py sends one Choice per item through TypeSafeClient, saves every answer (gold, top answer, top probability, full distribution, model name, tokens, seconds) to JSONL, and prints a report; --from re-scores a saved file with no model call, so metric changes never cost a run. The same harness runs the fake backend (a deliberately dumb word-overlap baseline), the claude backend and, with a key, Jev pinned to a version.

Free · sign in to start

Sign in to keep reading the answer.

Free, no card. We use your account to save progress, quiz scores, bookmarks and occasionally send updates from our team.

6 more sections
Progress, quiz scores and bookmarks