1Here, the position refers to relative position rather than ab-
solute position. The relative position is m − n in RoPE, where
m and n are the absolute positions of two tokens. The positional
O.O.D refers to cases where the value of m − n during inference
is unseen, i.e., larger than the values observed during pretraining.
In this paper, we map unseen large relative positions to those ob-
served during pretraining. More details about m − n are provided
in Section 2.2.

2The tokenization result is from OpenAI’s tokenization tool.
Self-Extend LLM