Fourier Position Embedding: Enhancing Attention's Periodic Extension for
  Length Generalization
Extending the context length of Language Models (LMs) by improving Rotary
Position Embedding (RoPE) has become a trend. While existing works mainly
address RoPE's limitations within attention mechanism, this paper provides an
analysis across nearly all parts of LMs, uncovering their adverse effects on
length generalization for RoPE-based attention. Using Discrete Signal
Processing theory, we show that RoPE enables periodic attention by implicitly
achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is
undermined by the spectral damage caused by: 1) linear layers and activation
functions outside of attention; 2) insufficiently trained frequency components
brought by time-domain truncation. Building on our observations, we propose
Fourier Position Embedding (FoPE), which enhances attention's frequency-domain
properties to improve both its periodic extension and length generalization.
FoPE constructs Fourier Series and zero-outs the destructive frequency
components, increasing model robustness against the spectrum damage.
Experiments across various model scales show that, within varying context
windows, FoPE can maintain a more stable perplexity and a more consistent
accuracy in a needle-in-haystack task compared to RoPE and ALiBi. Several
analyses and ablations bring further support to our method and theoretical
modeling.
676a6844bee647b8c004f469
Ermo Hua
claimed_verified
2024-12-30T19:34:47.877Z
6445fa2ffc22e309d78bef3e
https://cdn-avatars.huggingface.co/v1/production/uploads/6445fa2ffc22e309d78bef3e/FQaINLd0PjgY9EnK_APRk.jpeg
Messi Hua
user
Messi-Hua
676a6844bee647b8c004f46a
Che Jiang
676a6844bee647b8c004f46b
Xingtai Lv
claimed_verified
2025-03-17T08:45:51.765Z
663f07d029be04778ba97871
/avatars/fb7c9d4a2c537d918a3267e7cbc03f04.svg
Xingtai Lv
user
XingtaiHF
676a6844bee647b8c004f46c
Kaiyan Zhang
claimed_verified
2024-12-30T19:34:45.610Z
60bc94cd85a3ab33829b6211
/avatars/b57d36c7577fbbb42ea5b963eef4144a.svg
Kaiyan Zhang
user
iseesaw
676a6844bee647b8c004f46d
Ning Ding
claimed_verified
2025-01-31T09:50:47.792Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
676a6844bee647b8c004f46e
Youbang Sun
676a6844bee647b8c004f46f
Biqing Qi
676a6844bee647b8c004f470
Yuchen Fan
676a6844bee647b8c004f471
Xue Kai Zhu
676a6844bee647b8c004f472
Bowen Zhou
Fourier Position Embedding: Enhancing Attention's Periodic Extension for
  Length Generalization
Extending the context length of Language Models (LMs) by improving Rotary
Position Embedding (RoPE) has become a trend. While existing works mainly
address RoPE's limitations within attention mechanism, this paper provides an
analysis across nearly all parts of LMs, uncovering their adverse effects on
length generalization for RoPE-based attention. Using Discrete Signal
Processing theory, we show that RoPE enables periodic attention by implicitly
achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is
undermined by the spectral damage caused by: 1) linear layers and activation
functions outside of attention; 2) insufficiently trained frequency components
brought by time-domain truncation. Building on our observations, we propose
Fourier Position Embedding (FoPE), which enhances attention's frequency-domain
properties to improve both its periodic extension and length generalization.
FoPE constructs Fourier Series and zero-outs the destructive frequency
components, increasing model robustness against the spectrum damage.
Experiments across various model scales show that, within varying context
windows, FoPE can maintain a more stable perplexity and a more consistent
accuracy in a needle-in-haystack task compared to RoPE and ALiBi. Several
analyses and ablations bring further support to our method and theoretical
modeling.
New Fourier Position Embedding (FoPE) improves the frequency-domain properties of attention mechanisms, enhancing length generalization in language models compared to existing methods like Rotary Position Embedding (RoPE).
Rotary Position Embedding RoPE Discrete Signal Processing Non-Uniform Discrete Fourier Transform Fourier Position Embedding FoPE Fourier Series length generalization attention mechanism linear layers activation functions time-domain truncation perplexity accuracy needle-in-haystack task