27 upvotes
Looped Transformers Show Depth Gains, but Truncation Bias Persists
What is Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation?
The authors reuse a single stack of transformer layers in a loop, effectively deepening the network without adding trainable parameters. They compare full back-propagation through time with truncated versions and quantify the trade-off between compute savings and gradient bias.
Who it is for
Researchers and engineers building very deep models on limited hardware should read it to decide whether truncating BPTT is worth the bias.