Abstract

In a controlled fixed-budget study of linear attention decay mechanisms, we find that the delta rule is the largest observed factor under our protocol. We evaluate seven linear-attention variants plus a standard softmax baseline on two datasets (TinyStories, WikiText-103) with 3 random seeds each at both 18M and 125M parameters, and 5 variants at 42M on TinyStories. The data-independent subset, where decay gates are learned parameters rather than token-conditioned functions, forms a complete 2×2 comparison over granularity (scalar vs. channel-wise) and delta rule (absent vs. present), while the data-dependent variants provide additional controlled pairwise comparisons. We measure the delta rule's dominance by rank separation and loss gaps: all delta-rule variants (ranks 1–4) beat every model without a delta-rule write, including the softmax baseline (ranks 5–8), at 18M and 125M on both datasets; at 125M the worst delta variant beats the best model without a delta-rule write by 0.057 nats on TinyStories and 0.072 on WikiText-103 (0.217 versus softmax). A *granularity×delta interaction* provides the second key finding at 18M on both datasets: channel-wise decay hurts without the delta rule but helps with it. At 125M the picture is dataset-dependent: on TinyStories granularity effects fall below cross-seed variability, while on WikiText-103 channel-wise decay retains a robust benefit within delta (−0.036 ± 0.003, every seed) and the non-delta penalty falls within seed-to-seed variability — so the collapse of granularity effects is specific to TinyStories rather than a general property of 125M scale, consistent with dataset saturation. Within the top-4 delta variants, data-independent *StaticChannelDelta* leads at 18M on both datasets, while at 125M the top two are not robustly ordered (KDA ahead on TinyStories, StaticChannelDelta on WikiText-103, each within seed-to-seed variability).