title: Your Transformer is Secretly Linear
summary: This paper reveals a novel linear characteristic exclusive to transformer
  decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We
  analyze embedding transformations between sequential layers, uncovering a
  near-perfect linear relationship (Procrustes similarity score of 0.99).
  However, linearity decreases when the residual component is removed due to a
  consistently low output norm of the transformer layer. Our experiments show
  that removing or linearly approximating some of the most linear blocks of
  transformers does not affect significantly the loss or model performance.
  Moreover, in our pretraining experiments on smaller models we introduce a
  cosine-similarity-based regularization, aimed at reducing layer linearity. This
  regularization improves performance metrics on benchmarks like Tiny Stories and
  SuperGLUE and as well successfully decreases the linearity of the models. This
  study challenges the existing understanding of transformer architectures,
  suggesting that their operation may be more linear than previously assumed.
paper_authors._id: 664d6ab7d85a68a939cc73f4
paper_authors.name: Anton Razzhigaev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:34:57.554Z
paper_authors._id: 6172aaeec8e66e2aa84c06b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6172aaeec8e66e2aa84c06b9/ZdRZSp3P1SU6CIDbvQwkv.jpeg
paper_authors.fullname: Anton Razzhigaev
paper_authors.type: user
paper_authors.user: razzant
paper_authors._id: 664d6ab7d85a68a939cc73f5
paper_authors.name: Matvey Mikhalchuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:35:15.823Z
paper_authors._id: 64ee45a944f4b3b1bccc02d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ee45a944f4b3b1bccc02d1/SoidO9HQ4mftzbUPtuBBf.png
paper_authors.fullname: Matvey Mikhalchuk
paper_authors.type: user
paper_authors.user: matveymih
paper_authors._id: 664d6ab7d85a68a939cc73f6
paper_authors.name: Elizaveta Goncharova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:35:22.290Z
paper_authors._id: 6310ff34bc152fa3e810c186
paper_authors.avatarUrl: /avatars/bfd63bcd81548283f5e496e3693bf143.svg
paper_authors.fullname: Elizaveta Goncharova
paper_authors.type: user
paper_authors.user: Elizaveta
paper_authors._id: 664d6ab7d85a68a939cc73f7
paper_authors.name: Nikolai Gerasimenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:35:27.930Z
paper_authors._id: 63cfa7ef3b7adfa99c0eb524
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674553277288-noauth.jpeg
paper_authors.fullname: Nikolai Gerasimenko
paper_authors.type: user
paper_authors.user: nikgerasimenko
paper_authors._id: 664d6ab7d85a68a939cc73f8
paper_authors.name: Ivan Oseledets
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T07:35:33.932Z
paper_authors._id: 6169a581d05945bfd8718dfa
paper_authors.avatarUrl: /avatars/1892ab06a7ddb557232777de3cbec470.svg
paper_authors.fullname: Ivan Oseledets
paper_authors.type: user
paper_authors.user: oseledets
paper_authors._id: 664d6ab7d85a68a939cc73f9
paper_authors.name: Denis Dimitrov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T13:07:39.055Z
paper_authors._id: 6669a678465d1d802181e456
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6669a678465d1d802181e456/ZCthBBhDFQnh0bBkgUQUU.png
paper_authors.fullname: Denis Dimitrov
paper_authors.type: user
paper_authors.user: dendimitrov
paper_authors._id: 664d6ab7d85a68a939cc73fa
paper_authors.name: Andrey Kuznetsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-22T08:55:57.659Z
paper_authors._id: 643984dceb7c5616ef3f5d54
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643984dceb7c5616ef3f5d54/10JRkblrRIEVci6UJwvPz.jpeg
paper_authors.fullname: Andrey Kuznetsov
paper_authors.type: user
paper_authors.user: kuznetsoffandrey
paper_title: Your Transformer is Secretly Linear
paper_summary: This paper reveals a novel linear characteristic exclusive to transformer
  decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We
  analyze embedding transformations between sequential layers, uncovering a
  near-perfect linear relationship (Procrustes similarity score of 0.99).
  However, linearity decreases when the residual component is removed due to a
  consistently low output norm of the transformer layer. Our experiments show
  that removing or linearly approximating some of the most linear blocks of
  transformers does not affect significantly the loss or model performance.
  Moreover, in our pretraining experiments on smaller models we introduce a
  cosine-similarity-based regularization, aimed at reducing layer linearity. This
  regularization improves performance metrics on benchmarks like Tiny Stories and
  SuperGLUE and as well successfully decreases the linearity of the models. This
  study challenges the existing understanding of transformer architectures,
  suggesting that their operation may be more linear than previously assumed.
paper_ai_keywords: transformer decoders GPT LLaMA OPT BLOOM embedding transformations Procrustes similarity score residual component cosine-similarity-based regularization Tiny Stories SuperGLUE