13 upvotes
Muon-RT cuts memory for LLM self-distillation
What is MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation?
Muon-RT reuses the EMA teacher’s own weight history instead of storing a separate momentum buffer. The result is a lighter self-distillation process that needs less GPU RAM.
Who it is for
Anyone training large language models can save memory and speed up self-distillation runs.