6 upvotes
Muir introduces linear-time attention with gated state updates
What is Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout?
Muir compresses the entire context into a fixed-size key-value memory, letting attention run in linear time and constant memory. It uses a sensitivity-allocated gating mechanism and a matched-filter readout to decay old entries and write new ones efficiently.
Who it is for
Developers of transformer-based models who need low-memory or faster inference can try Muir as a more efficient attention drop-in.