title: Rethinking Addressing in Language Models via Contexualized Equivariant
    Positional Encoding
summary: Transformers rely on both content-based and position-based addressing
  mechanisms to make predictions, but existing positional encoding techniques
  often diminish the effectiveness of position-based addressing. Many current
  methods enforce rigid patterns in attention maps, limiting the ability to model
  long-range dependencies and adapt to diverse tasks. Additionally, most
  positional encodings are learned as general biases, lacking the specialization
  required for different instances within a dataset. To address this, we propose
  conTextualized equivariAnt Position
  Embedding (TAPE), a novel framework that enhances
  positional embeddings by incorporating sequence content across layers. TAPE
  introduces dynamic, context-aware positional encodings, overcoming the
  constraints of traditional fixed patterns. By enforcing permutation and
  orthogonal equivariance, TAPE ensures the stability of positional encodings
  during updates, improving robustness and adaptability. Our method can be easily
  integrated into pre-trained transformers, offering parameter-efficient
  fine-tuning with minimal overhead. Extensive experiments shows that TAPE
  achieves superior performance in language modeling, arithmetic reasoning, and
  long-context retrieval tasks compared to existing positional embedding
  techniques.
paper_authors._id: 67778d3621a9158f25a2d23b
paper_authors.name: Jiajun Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:02:56.780Z
paper_authors._id: 65dc42ad1ed8dc6d82e3e4d8
paper_authors.avatarUrl: /avatars/d2001d58b4fff24509c9e8e952042fe5.svg
paper_authors.fullname: Jiajun Zhu
paper_authors.type: user
paper_authors.user: lanczos
paper_authors._id: 67778d3621a9158f25a2d23c
paper_authors.name: Peihao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T16:12:48.411Z
paper_authors._id: 62c8f1773e28b8ee4cddca09
paper_authors.avatarUrl: /avatars/9edbcc84e4ab2b335011daab8acfdbb8.svg
paper_authors.fullname: Peihao Wang
paper_authors.type: user
paper_authors.user: peihaowang
paper_authors._id: 67778d3621a9158f25a2d23d
paper_authors.name: Ruisi Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T16:14:43.111Z
paper_authors._id: 63c99e2ddfac8071d01b89da
paper_authors.avatarUrl: /avatars/196af8c8498aa7a930e77c0a169da678.svg
paper_authors.fullname: Ruisi Cai
paper_authors.type: user
paper_authors.user: CCCCRS
paper_authors._id: 67778d3621a9158f25a2d23e
paper_authors.name: Jason D. Lee
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-03T07:09:43.441Z
paper_authors._id: 64ad70f7f9f761a4c3910c63
paper_authors.avatarUrl: /avatars/69b6c167299ca7a174a69ddc5a95e4a2.svg
paper_authors.fullname: Jason Lee
paper_authors.type: user
paper_authors.user: Jasondlee
paper_authors._id: 67778d3621a9158f25a2d23f
paper_authors.name: Pan Li
paper_authors._id: 67778d3621a9158f25a2d240
paper_authors.name: Zhangyang Wang
paper_title: Rethinking Addressing in Language Models via Contexualized Equivariant
    Positional Encoding
paper_summary: Transformers rely on both content-based and position-based addressing
  mechanisms to make predictions, but existing positional encoding techniques
  often diminish the effectiveness of position-based addressing. Many current
  methods enforce rigid patterns in attention maps, limiting the ability to model
  long-range dependencies and adapt to diverse tasks. Additionally, most
  positional encodings are learned as general biases, lacking the specialization
  required for different instances within a dataset. To address this, we propose
  conTextualized equivariAnt Position
  Embedding (TAPE), a novel framework that enhances
  positional embeddings by incorporating sequence content across layers. TAPE
  introduces dynamic, context-aware positional encodings, overcoming the
  constraints of traditional fixed patterns. By enforcing permutation and
  orthogonal equivariance, TAPE ensures the stability of positional encodings
  during updates, improving robustness and adaptability. Our method can be easily
  integrated into pre-trained transformers, offering parameter-efficient
  fine-tuning with minimal overhead. Extensive experiments shows that TAPE
  achieves superior performance in language modeling, arithmetic reasoning, and
  long-context retrieval tasks compared to existing positional embedding
  techniques.
paper_ai_keywords: Transformers attention maps position-based addressing positional encoding techniques long-range dependencies learned positional encodings context-aware positional encodings permutation equivariance orthogonal equivariance parameter-efficient fine-tuning language modeling arithmetic reasoning long-context retrieval tasks