Module 3 · Reinforcement learning for agents · Lesson 5
GRPO in practice
Run reinforcement learning with verifiable rewards on a small open model, from configuring TRL's GRPO trainer and its loss variants to keeping groups informative, reading the training curves and telling sharpening from new capability.
By the end of this lesson you can
- Configure a GRPO run (group size, completion length, loss type, clipping, KL coefficient, LoRA learning rate) and explain the problem each loss variant fixes
- Compute the share of zero-variance groups from per-prompt success rates, and use difficulty filtering to keep a run learning
- Diagnose healthy, stalled and collapsing runs from reward, reward spread, entropy, KL and completion-length curves
- Compare pass@1, pass@k and pass^k before and after RL, and explain the evidence on whether RL adds capability or sharpens it
- Explain why LoRA is enough for policy-gradient RL and why online RL tends to forget less than supervised fine-tuning