Loopback Transformer: Towards a Unified View of Residual Looped Transformers
Loopback Transformer treats residual pathways as depth memory, using decay and attention to reuse block updates. The authors report lower loss with fewer parameters, but also identify memory and throughput trade-offs.