title: Transformers learn through gradual rank increase
summary: We identify incremental learning dynamics in transformers, where the
  difference between trained and initial weights progressively increases in rank.
  We rigorously prove this occurs under the simplifying assumptions of diagonal
  weight matrices and small initialization. Our experiments support the theory
  and also show that phenomenon can occur in practice without the simplifying
  assumptions.
paper_authors._id: 6487ea21d66a660acdc8c90f
paper_authors.name: Enric Boix-Adsera
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T21:19:29.266Z
paper_authors._id: 646e2a732f1c836fa8450786
paper_authors.avatarUrl: /avatars/da8df192767bf2a2a16a3889118b5184.svg
paper_authors.fullname: Enric Boix
paper_authors.type: user
paper_authors.user: eboix
paper_authors._id: 6487ea21d66a660acdc8c910
paper_authors.name: Etai Littwin
paper_authors._id: 6487ea21d66a660acdc8c911
paper_authors.name: Emmanuel Abbe
paper_authors._id: 6487ea21d66a660acdc8c912
paper_authors.name: Samy Bengio
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T21:19:17.894Z
paper_authors._id: 6488d89678570cedad6f4e03
paper_authors.avatarUrl: /avatars/d3c86914cab46b953c0ee7ed17e3d6bb.svg
paper_authors.fullname: Samy Bengio
paper_authors.type: user
paper_authors.user: bengio
paper_authors._id: 6487ea21d66a660acdc8c913
paper_authors.name: Joshua Susskind
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T21:19:20.932Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_title: Transformers learn through gradual rank increase
paper_summary: We identify incremental learning dynamics in transformers, where the
  difference between trained and initial weights progressively increases in rank.
  We rigorously prove this occurs under the simplifying assumptions of diagonal
  weight matrices and small initialization. Our experiments support the theory
  and also show that phenomenon can occur in practice without the simplifying
  assumptions.
paper_ai_keywords: incremental learning dynamics transformers trained and initial weights rank diagonal weight matrices small initialization