Module 2 · How models are trained · Lesson 5
Post-training an assistant
How supervised fine-tuning, preference learning (RLHF and DPO), constitutional training and reinforcement learning with verifiable rewards turn a base model into an assistant, and the side effects each one leaves behind.
By the end of this lesson you can
- Describe the stages of a modern post-training pipeline and what each one adds
- Compute a Bradley–Terry preference probability and explain how a reward model is trained from comparisons
- Explain the KL penalty in RLHF and how DPO removes the explicit reward model
- Compute group-relative advantages for RL with verifiable rewards (GRPO) and pick real tasks that have verifiable rewards
- Connect sycophancy, reward hacking and over-refusal to the training signals that cause them