28 upvotes
New Distillation Method Teaches Students Without Teacher’s Confidence Bias
What is Order-Only On-Policy Distillation?
Order-Only On-Policy Distillation trains a student model by copying only the ranking of the teacher’s token probabilities, ignoring the teacher’s confidence levels. This prevents the student from inheriting the teacher’s calibration quirks while still learning its preferences.
Who it is for
Anyone building smaller or faster language models can apply this to compress models without passing on over-confidence from the original.