title: Value Residual Learning For Alleviating Attention Concentration In
    Transformers
summary: Transformers can capture long-range dependencies using self-attention,
  allowing tokens to attend to all others directly. However, stacking multiple
  attention layers leads to attention concentration. One natural way to address
  this issue is to use cross-layer attention, allowing information from earlier
  layers to be directly accessible to later layers. However, this approach is
  computationally expensive. To address this problem, we propose Transformer with
  residual value (ResFormer) which approximates cross-layer attention through
  adding a residual connection from the values of the the first layer to all
  subsequent layers. Based on this method, one variant is the Transformer with
  single layer value (SVFormer), where all layers share the same value embedding
  from first layer, reducing the KV cache by nearly 50%. Comprehensive empirical
  evidence demonstrates that ResFormer mitigates attention concentration problem
  in deeper layers and enhances representation across most layers, outperforming
  the vanilla Transformer, DenseFormer, and NeuTRENO in training error as well as
  downstream tasks. SVFormer trains significantly faster than the vanilla
  Transformer and performs better than other methods like GQA and CLA, with
  performance influenced by sequence length and cumulative learning rate.
paper_authors._id: 671b3e033fd1d03dc68d685e
paper_authors.name: Zhanchao Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:28:30.222Z
paper_authors._id: 671b3d82101c0d9f78749256
paper_authors.avatarUrl: /avatars/902c1006280c6552885b0b946cd2bb16.svg
paper_authors.fullname: Zhanchao Zhou
paper_authors.type: user
paper_authors.user: Zcchill
paper_authors._id: 671b3e033fd1d03dc68d685f
paper_authors.name: Tianyi Wu
paper_authors._id: 671b3e033fd1d03dc68d6860
paper_authors.name: Zhiyun Jiang
paper_authors._id: 671b3e033fd1d03dc68d6861
paper_authors.name: Zhenzhong Lan
paper_title: Value Residual Learning For Alleviating Attention Concentration In
    Transformers
paper_summary: Transformers can capture long-range dependencies using self-attention,
  allowing tokens to attend to all others directly. However, stacking multiple
  attention layers leads to attention concentration. One natural way to address
  this issue is to use cross-layer attention, allowing information from earlier
  layers to be directly accessible to later layers. However, this approach is
  computationally expensive. To address this problem, we propose Transformer with
  residual value (ResFormer) which approximates cross-layer attention through
  adding a residual connection from the values of the the first layer to all
  subsequent layers. Based on this method, one variant is the Transformer with
  single layer value (SVFormer), where all layers share the same value embedding
  from first layer, reducing the KV cache by nearly 50%. Comprehensive empirical
  evidence demonstrates that ResFormer mitigates attention concentration problem
  in deeper layers and enhances representation across most layers, outperforming
  the vanilla Transformer, DenseFormer, and NeuTRENO in training error as well as
  downstream tasks. SVFormer trains significantly faster than the vanilla
  Transformer and performs better than other methods like GQA and CLA, with
  performance influenced by sequence length and cumulative learning rate.
paper_ai_keywords: self-attention long-range dependencies attention concentration cross-layer attention residual value ResFormer single layer value SVFormer KV cache attention triple problem vanilla Transformer DenseFormer SVTRENO training error downstream tasks GQA CLA sequence length cumulative learning rate