Transformers learn through gradual rank increase
We identify incremental learning dynamics in transformers, where the
difference between trained and initial weights progressively increases in rank.
We rigorously prove this occurs under the simplifying assumptions of diagonal
weight matrices and small initialization. Our experiments support the theory
and also show that phenomenon can occur in practice without the simplifying
assumptions.
6487ea21d66a660acdc8c90f
Enric Boix-Adsera
claimed_verified
2023-06-13T21:19:29.266Z
646e2a732f1c836fa8450786
/avatars/da8df192767bf2a2a16a3889118b5184.svg
Enric Boix
user
eboix
6487ea21d66a660acdc8c910
Etai Littwin
6487ea21d66a660acdc8c911
Emmanuel Abbe
6487ea21d66a660acdc8c912
Samy Bengio
claimed_verified
2023-06-13T21:19:17.894Z
6488d89678570cedad6f4e03
/avatars/d3c86914cab46b953c0ee7ed17e3d6bb.svg
Samy Bengio
user
bengio
6487ea21d66a660acdc8c913
Joshua Susskind
claimed_verified
2023-06-13T21:19:20.932Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
Transformers learn through gradual rank increase
We identify incremental learning dynamics in transformers, where the
difference between trained and initial weights progressively increases in rank.
We rigorously prove this occurs under the simplifying assumptions of diagonal
weight matrices and small initialization. Our experiments support the theory
and also show that phenomenon can occur in practice without the simplifying
assumptions.
The study examines incremental learning in transformers, proving that the difference in rank between trained and initial weights increases progressively.
incremental learning transformers diagonal weight matrices small initialization