Module 2 · Supervised and preference fine-tuning · Lesson 4
Preference optimisation
When you have comparisons or thumbs rather than gold outputs, train on them with DPO, KTO, ORPO or SimPO, build the pairs from your own model's samples ranked by a calibrated open-weight judge, and catch the model when it learns to please the judge instead of the user.
By the end of this lesson you can
- Build on-policy preference pairs from a fine-tuned model's own samples, using code checks first and a calibrated open-weight judge in both orders for the rest
- Turn binary thumbs-up and thumbs-down signals into KTO training data, correcting the class imbalance that selective feedback creates
- Choose between DPO, KTO, ORPO and SimPO from the shape of the data, the need for a reference model and the risk of length bias, and set beta and the learning rate
- Read implicit-reward margins, reward accuracy and log-probabilities during training, and explain why the loss and the margin disagree
- Detect length and judge exploitation after training with paired, length-controlled comparisons and a fresh check of the judge against your own labels