16 upvotes
On-Policy Distillation Fails to Fully Merge Multiple RL Teachers
What is On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation?
The authors examine a method that tries to combine several trained reinforcement-learning agents into one by copying their behavior while training. They find that, although the merged agent looks perfect on tasks it was trained on, it often misses key abilities when faced with new situations.
Who it is for
RL engineers and researchers who rely on distillation to compress or consolidate policies should note that the technique may not generalize, prompting more thorough testing.
How to try it
Read the paper at https://huggingface.co/papers/2611.05609