title: Fourier Position Embedding: Enhancing Attention's Periodic Extension for
    Length Generalization
summary: Extending the context length of Language Models (LMs) by improving Rotary
  Position Embedding (RoPE) has become a trend. While existing works mainly
  address RoPE's limitations within attention mechanism, this paper provides an
  analysis across nearly all parts of LMs, uncovering their adverse effects on
  length generalization for RoPE-based attention. Using Discrete Signal
  Processing theory, we show that RoPE enables periodic attention by implicitly
  achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is
  undermined by the spectral damage caused by: 1) linear layers and activation
  functions outside of attention; 2) insufficiently trained frequency components
  brought by time-domain truncation. Building on our observations, we propose
  Fourier Position Embedding (FoPE), which enhances attention's frequency-domain
  properties to improve both its periodic extension and length generalization.
  FoPE constructs Fourier Series and zero-outs the destructive frequency
  components, increasing model robustness against the spectrum damage.
  Experiments across various model scales show that, within varying context
  windows, FoPE can maintain a more stable perplexity and a more consistent
  accuracy in a needle-in-haystack task compared to RoPE and ALiBi. Several
  analyses and ablations bring further support to our method and theoretical
  modeling.
paper_authors._id: 676a6844bee647b8c004f469
paper_authors.name: Ermo Hua
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:34:47.877Z
paper_authors._id: 6445fa2ffc22e309d78bef3e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6445fa2ffc22e309d78bef3e/FQaINLd0PjgY9EnK_APRk.jpeg
paper_authors.fullname: Messi Hua
paper_authors.type: user
paper_authors.user: Messi-Hua
paper_authors._id: 676a6844bee647b8c004f46a
paper_authors.name: Che Jiang
paper_authors._id: 676a6844bee647b8c004f46b
paper_authors.name: Xingtai Lv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-17T08:45:51.765Z
paper_authors._id: 663f07d029be04778ba97871
paper_authors.avatarUrl: /avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
paper_authors.fullname: Xingtai Lv
paper_authors.type: user
paper_authors.user: XingtaiHF
paper_authors._id: 676a6844bee647b8c004f46c
paper_authors.name: Kaiyan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:34:45.610Z
paper_authors._id: 60bc94cd85a3ab33829b6211
paper_authors.avatarUrl: /avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
paper_authors.fullname: Kaiyan Zhang
paper_authors.type: user
paper_authors.user: iseesaw
paper_authors._id: 676a6844bee647b8c004f46d
paper_authors.name: Ning Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-31T09:50:47.792Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_authors._id: 676a6844bee647b8c004f46e
paper_authors.name: Youbang Sun
paper_authors._id: 676a6844bee647b8c004f46f
paper_authors.name: Biqing Qi
paper_authors._id: 676a6844bee647b8c004f470
paper_authors.name: Yuchen Fan
paper_authors._id: 676a6844bee647b8c004f471
paper_authors.name: Xue Kai Zhu
paper_authors._id: 676a6844bee647b8c004f472
paper_authors.name: Bowen Zhou
paper_title: Fourier Position Embedding: Enhancing Attention's Periodic Extension for
    Length Generalization
paper_summary: Extending the context length of Language Models (LMs) by improving Rotary
  Position Embedding (RoPE) has become a trend. While existing works mainly
  address RoPE's limitations within attention mechanism, this paper provides an
  analysis across nearly all parts of LMs, uncovering their adverse effects on
  length generalization for RoPE-based attention. Using Discrete Signal
  Processing theory, we show that RoPE enables periodic attention by implicitly
  achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is
  undermined by the spectral damage caused by: 1) linear layers and activation
  functions outside of attention; 2) insufficiently trained frequency components
  brought by time-domain truncation. Building on our observations, we propose
  Fourier Position Embedding (FoPE), which enhances attention's frequency-domain
  properties to improve both its periodic extension and length generalization.
  FoPE constructs Fourier Series and zero-outs the destructive frequency
  components, increasing model robustness against the spectrum damage.
  Experiments across various model scales show that, within varying context
  windows, FoPE can maintain a more stable perplexity and a more consistent
  accuracy in a needle-in-haystack task compared to RoPE and ALiBi. Several
  analyses and ablations bring further support to our method and theoretical
  modeling.
paper_ai_keywords: Rotary Position Embedding (RoPE) attention mechanism length generalization Discrete Signal Processing theory periodic attention Non-Uniform Discrete Fourier Transform spectral damage linear layers activation functions frequency components time-domain truncation Fourier Position Embedding (FoPE) frequency-domain properties periodic extension Fourier Series destructive frequency components model robustness perplexity needle-in-haystack task ALiBi