3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
<|reference_start|>3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding: Inspired by the Bloch Sphere representation, we propose a novel rotary position encoding on a three-dimensional sphere, named 3D Rotary Position Encoding (3D-RPE). 3D-RPE is an advanced version of the widely used 2D Rotary Position Encoding (RoPE), with two major advantages for modeling long contexts: controllable long-term decay and improved position resolution. For controllable long-term decay, 3D-RPE allows for the regulation of long-term decay within the chunk size, ensuring the modeling of relative positional information between tokens at a distant relative position. For enhanced position resolution, 3D-RPE can mitigate the degradation of position resolution caused by position interpolation on RoPE. We have conducted experiments on long-context Natural Language Understanding (NLU) and long-sequence Language Modeling (LM) tasks. From the experimental results, 3D-RPE achieved performance improvements over RoPE, especially in long-context NLU tasks.<|reference_end|>
@article{ma20243d-rpe:,
    title={3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position
  Encoding},
    author={Xindian Ma, Wenyuan Liu, Peng Zhang, Nan Xu},
    journal={arXiv preprint arXiv:2406.09897},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.09897},
    primaryClass={cs.CL}
}