title: You Do Not Fully Utilize Transformer's Representation Capacity
summary: In contrast to RNNs, which compress previous tokens into a single hidden
  state, Transformers can attend to all previous tokens directly. However,
  standard Transformers only use representations from the immediately preceding
  layer. In this paper, we show that this design choice causes representation
  collapse and leads to suboptimal performance. To address this issue, we
  introduce Layer-Integrated Memory (LIMe), a simple yet powerful approach that
  preserves the model's overall memory footprint while expanding its
  representational capacity by allowing access to hidden states from earlier
  layers. Through extensive experiments across various architectures and
  different lookup mechanisms, we demonstrate consistent performance improvements
  on a wide range of tasks. Moreover, our analysis of the learned representation
  dynamics and our exploration of depthwise circuits reveal how LIMe integrates
  information across layers, pointing to promising directions for future
  research.
paper_authors._id: 67b57a993d4f319f1fa9424b
paper_authors.name: Gleb Gerasimov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T10:10:30.547Z
paper_authors._id: 65db0871ab2f64915ce05e73
paper_authors.avatarUrl: /avatars/77e03f493196c5413cd2a02270e93660.svg
paper_authors.fullname: Gleb Gerasimov
paper_authors.type: user
paper_authors.user: gudleifrr
paper_authors._id: 67b57a993d4f319f1fa9424c
paper_authors.name: Yaroslav Aksenov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:00:41.123Z
paper_authors._id: 63ed5676684767daecac6f8a
paper_authors.avatarUrl: /avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
paper_authors.fullname: Yaroslav Aksenov
paper_authors.type: user
paper_authors.user: yaraksen
paper_authors._id: 67b57a993d4f319f1fa9424d
paper_authors.name: Nikita Balagansky
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:33:26.858Z
paper_authors._id: 60b364e7f88532cd79eaff7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
paper_authors.fullname: Nikita Balagansky
paper_authors.type: user
paper_authors.user: elephantmipt
paper_authors._id: 67b57a993d4f319f1fa9424e
paper_authors.name: Viacheslav Sinii
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T11:12:28.927Z
paper_authors._id: 6416272d986557e8cac64ece
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6416272d986557e8cac64ece/s3CLjNN_pGj-vJDcENFD2.jpeg
paper_authors.fullname: Viacheslav
paper_authors.type: user
paper_authors.user: ummagumm-a
paper_authors._id: 67b57a993d4f319f1fa9424f
paper_authors.name: Daniil Gavrilov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:00:43.143Z
paper_authors._id: 62a9c8edc19f92ae443ab37f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
paper_authors.fullname: Daniil Gavrilov
paper_authors.type: user
paper_authors.user: kefirski
paper_title: You Do Not Fully Utilize Transformer's Representation Capacity
paper_summary: In contrast to RNNs, which compress previous tokens into a single hidden
  state, Transformers can attend to all previous tokens directly. However,
  standard Transformers only use representations from the immediately preceding
  layer. In this paper, we show that this design choice causes representation
  collapse and leads to suboptimal performance. To address this issue, we
  introduce Layer-Integrated Memory (LIMe), a simple yet powerful approach that
  preserves the model's overall memory footprint while expanding its
  representational capacity by allowing access to hidden states from earlier
  layers. Through extensive experiments across various architectures and
  different lookup mechanisms, we demonstrate consistent performance improvements
  on a wide range of tasks. Moreover, our analysis of the learned representation
  dynamics and our exploration of depthwise circuits reveal how LIMe integrates
  information across layers, pointing to promising directions for future
  research.
paper_ai_summary: Layer-Integrated Memory (LIMe) addresses representation collapse in Transformers by allowing access to hidden states from earlier layers, leading to consistent performance improvements across tasks.
paper_ai_keywords: Transformers RNNs hidden state representation collapse Layer-Integrated Memory LIMe learned representation dynamics depthwise circuits