SeqPE: Transformer with Sequential Position Encoding
Since self-attention layers in Transformers are permutation invariant by
design, positional encodings must be explicitly incorporated to enable spatial
understanding. However, fixed-size lookup tables used in traditional learnable
position embeddings (PEs) limit extrapolation capabilities beyond pre-trained
sequence lengths. Expert-designed methods such as ALiBi and RoPE, mitigate this
limitation but demand extensive modifications for adapting to new modalities,
underscoring fundamental challenges in adaptability and scalability. In this
work, we present SeqPE, a unified and fully learnable position encoding
framework that represents each n-dimensional position index as a symbolic
sequence and employs a lightweight sequential position encoder to learn their
embeddings in an end-to-end manner. To regularize SeqPE's embedding space, we
introduce two complementary objectives: a contrastive objective that aligns
embedding distances with a predefined position-distance function, and a
knowledge distillation loss that anchors out-of-distribution position
embeddings to in-distribution teacher representations, further enhancing
extrapolation performance. Experiments across language modeling, long-context
question answering, and 2D image classification demonstrate that SeqPE not only
surpasses strong baselines in perplexity, exact match (EM), and
accuracy--particularly under context length extrapolation--but also enables
seamless generalization to multi-dimensional inputs without requiring manual
architectural redesign. We release our code, data, and checkpoints at
https://github.com/ghrua/seqpe.
68516d668a68fee7f6ba4cd3
Huyang Li
68516d668a68fee7f6ba4cd4
Yahui Liu
68516d668a68fee7f6ba4cd5
Hongyu Sun
68516d668a68fee7f6ba4cd6
Deng Cai
68516d668a68fee7f6ba4cd7
Leyang Cui
68516d668a68fee7f6ba4cd8
Wei Bi
68516d668a68fee7f6ba4cd9
Peilin Zhao
68516d668a68fee7f6ba4cda
Taro Watanabe
SeqPE: Transformer with Sequential Position Encoding
Since self-attention layers in Transformers are permutation invariant by
design, positional encodings must be explicitly incorporated to enable spatial
understanding. However, fixed-size lookup tables used in traditional learnable
position embeddings (PEs) limit extrapolation capabilities beyond pre-trained
sequence lengths. Expert-designed methods such as ALiBi and RoPE, mitigate this
limitation but demand extensive modifications for adapting to new modalities,
underscoring fundamental challenges in adaptability and scalability. In this
work, we present SeqPE, a unified and fully learnable position encoding
framework that represents each n-dimensional position index as a symbolic
sequence and employs a lightweight sequential position encoder to learn their
embeddings in an end-to-end manner. To regularize SeqPE's embedding space, we
introduce two complementary objectives: a contrastive objective that aligns
embedding distances with a predefined position-distance function, and a
knowledge distillation loss that anchors out-of-distribution position
embeddings to in-distribution teacher representations, further enhancing
extrapolation performance. Experiments across language modeling, long-context
question answering, and 2D image classification demonstrate that SeqPE not only
surpasses strong baselines in perplexity, exact match (EM), and
accuracy--particularly under context length extrapolation--but also enables
seamless generalization to multi-dimensional inputs without requiring manual
architectural redesign. We release our code, data, and checkpoints at
https://github.com/ghrua/seqpe.
SeqPE, a fully learnable position encoding framework, enhances the adaptability and scalability of positional encodings in Transformers, improving performance in various tasks and seamless multi-dimensional generalization.
self-attention layers Transformers permutation invariant positional encodings learnable position embeddings ALiBi RoPE symbolic sequence sequential position encoder contrastive objective knowledge distillation loss language modeling long-context question answering 2D image classification perplexity exact match accuracy context length extrapolation