Think Like a PhD Student
Ask a question, design the experiment, annotate the data, doubt the result, write it up and get it reviewed: the research discipline for AI agent work.
3 modules · 10 lessons~26 hoursUpdated 2026-11-13
- Lessons10
- Quizzes10
- Sources68
- Of study~26h
About this course
About this course
Engineering ships demos; research answers questions. This course teaches the craft a PhD trains, applied to AI agents: turning a vague worry into a falsifiable hypothesis, reading the literature in three passes, making sure a metric measures the thing, designing an experiment with baselines, controls and enough trials to beat run-to-run variance, pre-registering the analysis, building a failure taxonomy by open coding traces, running an annotation study with a codebook and measured agreement, packaging the work so someone else can rerun it, analysing and reporting honestly, and writing a report that survives peer review. The statistics come from Machine Learning Foundations for LLM Work and the graders from Evaluating and Scoring Agents; this course is about the method that uses them. The capstone is one complete small study, from question to self-review.
About the running example
You are the one researcher inside Quillmoor, the fictional project-management SaaS of this path, and the question running through the course is whether giving the triage agent retrieval over past resolved tickets reduces escalations without raising the wrong-priority rate. Experiments run through Claude Code headless over the Quillmoor triage kit's synthetic tickets, with a small set of resolved tickets, pre-generated traces and a flawed report added for the annotation and review lessons. Everything is synthetic.
What you'll learn.
- 1Turn a vague worry about an agent into a research question, a falsifiable hypothesis and a measurable prediction with a pre-registered analysis plan
- 2Search and read the literature in three passes, chase citations, grade a paper's evidence and keep an annotated bibliography
- 3Design an experiment on an LLM system with baselines, controls, ablations and enough trials to separate an effect from run-to-run variance, and run it from a reproducible harness
- 4Build a failure taxonomy by open coding agent traces, turn it into a codebook, pilot an annotation study and report agreement with an interval
- 5Analyse results honestly: effect sizes with intervals, no forking paths, corrected multiple comparisons, negative results and figures that don't mislead
- 6Write a tech report whose claims match its evidence, package it so someone else can rerun it, review another report with a rubric and red-team your own
3 modules, 10 lessons.
13h 15m of lessons
Module 1 · 3 lessons · 4h
Asking the question
What research trains that engineering doesn't, how to turn "is our agent good?" into a falsifiable claim, how to read what's already known, and how to make sure the metric measures the thing.
Module 2 · 4 lessons · 5h 25m
Designing and running the study
Controls, baselines, ablations, confounds and variance; pre-registration and trials; exploratory open coding of failures; a confirmatory annotation study with a codebook and measured agreement.
Module 3 · 3 lessons · 3h 50m
Trusting and telling
Reproducibility packages, honest analysis and reporting, writing the tech report, reviewing others' work and red-teaming your own claims, with the research ethics basics for AI work.
A small study on the triage agent
Run one complete, honest study on Quillmoor's triage agent, from a research question and a tagged pre-registration through the experiment, an annotation study, a pre-registered analysis and a reproducibility package, to a tech report you review yourself.
The project brief is coming soon
More in Evaluation and Optimisation.
- IntermediateAI agents · Evals
Evaluating and Scoring Agents
Turn "it seems better" into numbers you can defend: error analysis, code graders, calibrated LLM judges and honest statistics.
7 lessons ~20hFree - AdvancedPrompt engineering · DSPy and GEPA
Prompt and Context Optimisation
Treat prompts, examples, tool descriptions and skills as versioned, measured artefacts, and improve them by hand, by optimiser and in production without breaking what already works.
7 lessons ~17hFree - AdvancedEvals · Human feedback
Human Feedback at Scale
Collect, measure and trust the judgements of reviewers and users: study design, rater agreement, preference rankings, real-use signals and quality estimates from a few labels.
7 lessons ~28hFree - AdvancedFine-tuning · Reinforcement learning
Fine-Tuning and RL for Agents
Train a small open model for one agent step with supervised fine-tuning, preference optimisation and reinforcement learning, and prove with your eval suite whether it beats an optimised prompt.
8 lessons ~38hFree