Coders In Hoods

Courses
Answers
Setup
Bookmarks

Coders In Hoods Academy

Every course is free. Sign in to start, and your progress, quiz scores and bookmarks follow you.

Academy

Courses
Paths
Exercise answers
Setup guides
Bookmarks

Coders In Hoods

The agency
Coaching for teams

Social

LinkedIn
X (Twitter)

Contact

codersinhoods@gmail.com

© Coders In Hoods · since 2019

Free · no card

Coders In Hoods Academy

Sign in to start learning

Free, no card. We use your account to save progress, quiz scores, bookmarks and occasionally send updates from our team.

Course overview

Agent Benchmarks and Simulations

Free · sign in to start and track progress

1 · Simulated worlds

1Stateful environments and end-state grading
2Building user simulators
3Validating user simulators

2 · Running benchmarks

4Task specs, harnesses and sandboxes
5Noise, reproducibility and eval infrastructure

3 · Benchmarks you can trust

6Contamination, saturation and broken tasks
Exercise answers
- Answer: Re-annotate a Quillmoor suite with planted flaws
- Answer: Probe a public benchmark for memorisation
7Auditing benchmarks for exploits
8Item analysis, difficulty and time horizons
9Reporting results and reading model cards

CapstoneA benchmark for the account-support agent you'd defend Reference implementation

Answers Lesson 6: Contamination, saturation and broken tasks

Step-by-step answer

Stretch 60–90 min6 steps

Probe a public benchmark for memorisation

Back to the exercise

Previous answerRe-annotate a Quillmoor suite with planted flaws Next answer Red-team your Quillmoor benchmark

Free · sign in to start

Sign in to read the answer.

Free, no card. We use your account to save progress, quiz scores, bookmarks and occasionally send updates from our team.

6 more sections
Progress, quiz scores and bookmarks