title: When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context
    Training
summary: Extending context window sizes allows large language models (LLMs) to process
  longer sequences and handle more complex tasks. Rotary Positional Embedding
  (RoPE) has become the de facto standard due to its relative positional encoding
  properties that benefit long-context training. However, we observe that using
  RoPE with BFloat16 format results in numerical issues, causing it to deviate
  from its intended relative positional encoding, especially in long-context
  scenarios. This issue arises from BFloat16's limited precision and accumulates
  as context length increases, with the first token contributing significantly to
  this problem. To address this, we develop AnchorAttention, a plug-and-play
  attention method that alleviates numerical issues caused by BFloat16, improves
  long-context capabilities, and speeds up training. AnchorAttention reduces
  unnecessary attention computations, maintains semantic coherence, and boosts
  computational efficiency by treating the first token as a shared anchor with a
  consistent position ID, making it visible to all documents within the training
  context. Experiments on three types of LLMs demonstrate that AnchorAttention
  significantly improves long-context performance and reduces training time by
  over 50\% compared to standard full attention mechanisms, while preserving the
  original LLM's capabilities on general tasks. Our code is available at
  https://github.com/haonan3/AnchorContext.
paper_authors._id: 673ea9f174d6cdda59f38d4d
paper_authors.name: Haonan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T08:09:35.543Z
paper_authors._id: 6496b06a4a9a7e1fe4253ae2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/144NlRW_ETmmOgSYUs_SM.png
paper_authors.fullname: Haonan Wang
paper_authors.type: user
paper_authors.user: haonan3
paper_authors._id: 673ea9f174d6cdda59f38d4e
paper_authors.name: Qian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T21:33:06.403Z
paper_authors._id: 612ee6a7b960e78c6d2319d4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/612ee6a7b960e78c6d2319d4/2Hu9BaAyXbyh1vt0v1Qui.jpeg
paper_authors.fullname: Qian Liu
paper_authors.type: user
paper_authors.user: SivilTaram
paper_authors._id: 673ea9f174d6cdda59f38d4f
paper_authors.name: Chao Du
paper_authors._id: 673ea9f174d6cdda59f38d50
paper_authors.name: Tongyao Zhu
paper_authors._id: 673ea9f174d6cdda59f38d51
paper_authors.name: Cunxiao Du
paper_authors._id: 673ea9f174d6cdda59f38d52
paper_authors.name: Kenji Kawaguchi
paper_authors._id: 673ea9f174d6cdda59f38d53
paper_authors.name: Tianyu Pang
paper_title: When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context
    Training
paper_summary: Extending context window sizes allows large language models (LLMs) to process
  longer sequences and handle more complex tasks. Rotary Positional Embedding
  (RoPE) has become the de facto standard due to its relative positional encoding
  properties that benefit long-context training. However, we observe that using
  RoPE with BFloat16 format results in numerical issues, causing it to deviate
  from its intended relative positional encoding, especially in long-context
  scenarios. This issue arises from BFloat16's limited precision and accumulates
  as context length increases, with the first token contributing significantly to
  this problem. To address this, we develop AnchorAttention, a plug-and-play
  attention method that alleviates numerical issues caused by BFloat16, improves
  long-context capabilities, and speeds up training. AnchorAttention reduces
  unnecessary attention computations, maintains semantic coherence, and boosts
  computational efficiency by treating the first token as a shared anchor with a
  consistent position ID, making it visible to all documents within the training
  context. Experiments on three types of LLMs demonstrate that AnchorAttention
  significantly improves long-context performance and reduces training time by
  over 50\% compared to standard full attention mechanisms, while preserving the
  original LLM's capabilities on general tasks. Our code is available at
  https://github.com/haonan3/AnchorContext.
paper_ai_keywords: Large language models (LLMs) Context window sizes Rotary Positional Embedding (RoPE) Relative positional encoding BFloat16 format Numerical issues Positional encoding AnchorAttention Attention mechanism Long-context capabilities Training time Consistent position ID Semantic coherence Computational efficiency