Module 3 · Reinforcement learning for agents · Lesson 8
Reward hacking, regressions and shipping
Catch a policy that games its training reward, find out what else training changed, compare the trained model with a frontier-model baseline on paired outcomes and cost per completed case, and serve adapters you can version and roll back.
By the end of this lesson you can
- Detect reward hacking with held-out graders that differ from the training reward, canary tasks and stratified trace review, and name the common agent hacks
- Design capability and safety regression tiers for a fine-tuned model, and explain why narrow training can change behaviour far outside the task
- Compare a trained model with a baseline using paired outcomes, negative flips by tier and an exact test, and decide when a higher average still blocks a release
- Compute cost per completed case including amortised training, serving and retraining, and find the volume at which training pays
- Serve several LoRA adapters on one base model, version them, roll back by moving a label, and plan for base-model deprecation