Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Linear attention promises a way around one of the Transformer’s central costs: softmax attention requires work that grows quadratically with sequence length during training and a key-value cache that grows with context during inference. Instead of retaining every past token, linear-attention models compress history into a fixed-size associative state. The trade-off is memory interference: as more keys share that state, older associations can overwrite or obscure one another.
In this paper, Tushaar Gangavarapu, Gil Tabak, and Clemens JS Schaefer introduce Muir, a gated delta-rule model designed to make that fixed-size memory more effective. The authors build on Gated DeltaNet-2 (GDN-2), but argue that its gates do not deserve equal parameter capacity. Muir reallocates capacity according to how sensitive the state is to each gate, improves how the state is read, and adds a chunkwise training algorithm intended for wide matrix-multiply accelerators.
The problem: not all state channels behave alike
GDN-2 uses channel-wise decay, erasure, and write gates. This gives each state row or value channel independent control, but the dense projections used for erasure and writing consume substantial parameters. Under an iso-parameter budget, those parameters reduce the feed-forward network (FFN), which the authors identify as an important store of factual and token-level knowledge.
The paper’s analysis finds an asymmetry between erasure and writing. Queries and keys are passed through SiLU and normalized to unit length, concentrating key energy on a relatively small subset of channels. An erase gate therefore affects the state mainly where both the current key and an existing state row have energy. A write gate, by contrast, scales the incoming value directly across all value channels. In the authors’ trained-model diagnostics, the state was about 13 times more sensitive to write-gate perturbations than erase-gate perturbations.
There is a second problem. Channel-wise decay creates rows with very different norms. Slow-decaying rows accumulate more writes and can dominate an unweighted readout, even when their stored content is not the best match for the current query. Suppressing that interference during writing requires stronger erasure, but stronger erasure also shortens the memory horizon.
What Muir changes
Muir combines five gates—decay, erasure, output, writing, and query readout—through one fused low-rank projection. The rank allocation is deliberately asymmetric: decay and erase receive less rank, while the output, write, and readout gates receive more. In the paper’s primary configuration, the ranks are 64, 64, 128, 128, and 128 respectively, for a total rank of 512.
The saved parameters are reinvested in the FFN. At a fixed parameter budget, the authors report a 30.4% wider FFN than GDN-2 at the 2,048-dimensional model width used in their largest comparison. This is not merely a compression trick: Muir trades dense gate flexibility for additional representation capacity where the authors believe it has greater impact.
The readout change is the paper’s other central idea. Muir multiplies the normalized query by a learned channel-wise gain before reading the state. The static component approximates a matched filter: it downweights high-energy, noisy rows while preserving channels whose signal is likely to survive over the relevant horizon. A dynamic component then adapts the gain per token, allowing the model to use different channel profiles for short- and long-lag retrieval.
For training, the authors introduce a boundary-pivoted chunkwise algorithm. Standard parallel formulations can divide by very small cumulative decay prefixes, causing overflow, or materialize large decay tensors and fall back to slow vector units. Muir splits decay factors at boundaries inside a recursive block decomposition. The resulting computations use matrix multiplications without prefix division and route most chunkwise arithmetic to the matrix units of the TPU7x systems evaluated in the paper.
Evidence from controlled memory tests and language modeling
The synthetic experiments support the proposed mechanism. In one stationary-memory setup, the authors’ row-energy formula tracked measured channel energy across a 52.4-fold range, with reported Spearman and log-Pearson correlations of 1.0. Full inverse-RMS whitening hurt retrieval because it amplified fast-decaying rows whose target signal had already disappeared. A fractional or horizon-aware filter performed better and reduced the optimal erase strength from 0.60 to 0.50 in that experiment.
In end-to-end associative recall, low-rank erase gates outperformed dense erase gates across the tested ranks, while overly aggressive compression of the write gate reduced long-lag recall. Adding static or dynamic readout gains improved recall over unit readout; the dynamic version was strongest in the authors’ reported low-rank configuration.
The main pretraining comparison covers 166M to 1.34B parameters, using 100B tokens from FineWeb-Edu. Across these scales, Muir lowers validation negative log-likelihood by 18.0–20.4 millinats relative to GDN-2 and by 25.0–36.2 millinats relative to the Transformer baseline. At 1.34B parameters, the authors report a 1.38-point improvement in the eight-task commonsense average over GDN-2.
The gains are particularly relevant to retrieval. At 1.34B, Muir improves the reported multi-hop and multi-needle results over both baselines and raises the six-task real-world extractive recall average to 34.60%, compared with 32.08% for GDN-2. The paper also reports 90.0% zero-shot passkey retrieval at one million tokens for the static-readout variant, and 99.4% after a 64K-token cooldown. Its recurrent state remains fixed at 25 MB in the described evaluation, rather than growing with the stream.
Efficiency and limitations
On eight TPU7x chips, Muir matches GDN-2 throughput at the 4K training length and is slightly faster at 16K and 32K. Compared with the Transformer, the authors report 1.64 times higher throughput at 16K and 2.74 times at 32K. The advantage is hardware- and kernel-dependent: the Transformer is faster below roughly 11K tokens in these experiments.
The paper’s limitations are important. Static readout preserves more write-gate capacity and is excellent for single-horizon passkey lookup, but it cannot adapt optimally to multiple retrieval lags. Dynamic readout helps multi-hop and mixed-lag tasks, yet it consumes rank that could otherwise support writing; in the reported setup, dynamic Muir tapers on very long single-needle retrieval where static Muir remains strong. The kernel is also not universally superior: the pure-JAX implementation leaves a short-context gap compared with a hand-tuned attention kernel, and synthetic stress tests show that the triangular solve becomes inaccurate when conditioning is substantially worse than in the trained checkpoints.
For practitioners, Muir suggests a useful design rule: allocate parameters according to measured sensitivity, not by treating every gate symmetrically. It also highlights readout as a distinct control surface for recurrent memory. If long-context workloads mix local prediction, multi-hop reasoning, and distant retrieval, a learned, horizon-aware readout may reduce the need to erase memory aggressively. The authors’ results make that a promising direction—but they do not establish that Muir replaces attention for every workload or hardware stack.
Sources
AI-assisted research summary. Check the original paper for complete methods, results, and limitations.