11 upvotes
Every Token Gets Its Own Teacher with Adaptive Self-distillation
What is Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation?
The technique trains a language model by building a separate teacher for each token, using privileged reference data to guide predictions. These teachers are updated on the fly, so the student model always learns from the most relevant guidance.
Who it is for
Anyone fine-tuning large language models can use this to boost performance without extra data.
How to try it
Open the repository linked on the paper’s Hugging Face page