6 upvotes

Sparse Autoencoders Uncover Human-Readable Specialties in MoE Experts

Paper · SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization

What is SAE Interprets MoE: Sparse AutoEncoders Reveal That MoE Experts Exhibit Human-Interpretable Fine-Grained Specialization?

The authors train tiny sparse autoencoders on the activations of each MoE expert to see what they actually respond to. The resulting probes reveal that experts fire on fine-grained, human-interpretable linguistic patterns rather than just broad domains.

Who it is for

Anyone building, debugging, or pruning large mixture-of-experts models can use these insights to understand and refine individual expert behavior.