title: Cure the headache of Transformers via Collinear Constrained Attention
summary: As the rapid progression of practical applications based on Large Language
  Models continues, the importance of extrapolating performance has grown
  exponentially in the research domain. In our study, we identified an anomalous
  behavior in Transformer models that had been previously overlooked, leading to
  a chaos around closest tokens which carried the most important information.
  We've coined this discovery the "headache of Transformers". To address this at
  its core, we introduced a novel self-attention structure named Collinear
  Constrained Attention (CoCA). This structure can be seamlessly integrated with
  existing extrapolation, interpolation methods, and other optimization
  strategies designed for traditional Transformer models. We have achieved
  excellent extrapolating performance even for 16 times to 24 times of sequence
  lengths during inference without any fine-tuning on our model. We have also
  enhanced CoCA's computational and spatial efficiency to ensure its
  practicality. We plan to open-source CoCA shortly. In the meantime, we've made
  our code available in the appendix for reappearing experiments.
paper_authors._id: 650903aa5ac587b3f356753f
paper_authors.name: Shiyi Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:26:35.255Z
paper_authors._id: 65099314ed23af2c5d1e8766
paper_authors.avatarUrl: /avatars/e5e61d6d937645f9ec812e23e3061fdd.svg
paper_authors.fullname: Shiyi Zhu
paper_authors.type: user
paper_authors.user: underskies
paper_authors._id: 650903aa5ac587b3f3567540
paper_authors.name: Jing Ye
paper_authors._id: 650903aa5ac587b3f3567541
paper_authors.name: Wei Jiang
paper_authors._id: 650903aa5ac587b3f3567542
paper_authors.name: Qi Zhang
paper_authors._id: 650903aa5ac587b3f3567543
paper_authors.name: Yifan Wu
paper_authors._id: 650903aa5ac587b3f3567544
paper_authors.name: Jianguo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:35:25.454Z
paper_authors._id: 640b3ac391d9f65c58a795a4
paper_authors.avatarUrl: /avatars/4c0b745b5d730cc076c798c7628f2cd2.svg
paper_authors.fullname: Jianguo Li
paper_authors.type: user
paper_authors.user: JungleLee
paper_title: Cure the headache of Transformers via Collinear Constrained Attention
paper_summary: As the rapid progression of practical applications based on Large Language
  Models continues, the importance of extrapolating performance has grown
  exponentially in the research domain. In our study, we identified an anomalous
  behavior in Transformer models that had been previously overlooked, leading to
  a chaos around closest tokens which carried the most important information.
  We've coined this discovery the "headache of Transformers". To address this at
  its core, we introduced a novel self-attention structure named Collinear
  Constrained Attention (CoCA). This structure can be seamlessly integrated with
  existing extrapolation, interpolation methods, and other optimization
  strategies designed for traditional Transformer models. We have achieved
  excellent extrapolating performance even for 16 times to 24 times of sequence
  lengths during inference without any fine-tuning on our model. We have also
  enhanced CoCA's computational and spatial efficiency to ensure its
  practicality. We plan to open-source CoCA shortly. In the meantime, we've made
  our code available in the appendix for reappearing experiments.
paper_ai_keywords: Transformer models self-attention Collinear Constrained Attention (CoCA) extrapolating performance sequence length inference