Module 2 · Changing the model, and governing the spend · Lesson 6
Distilling into small models
When training a small open-weight model on a frontier model's outputs pays for itself, how the pipeline works from teacher labels to LoRA fine-tuning and serving, and the break-even arithmetic that often says no.
Setup for the exercises:Install Claude CodePython with uvRun GPU code on ModalRun Hugging Face models on CPU
By the end of this lesson you can
- Explain sequence-level distillation and LoRA fine-tuning, and compute the size of a LoRA adapter for a given model
- Design a distillation pipeline from teacher labelling to evaluation and serving, using routing logs and verifiers as data sources
- Compute the fixed and marginal costs of distillation and the break-even volume against both the teacher and an off-the-shelf small model
- Identify the contractual, data-protection and quality risks of training on client documents and on a provider's outputs