2

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

T5 (Xue et al., 2020), Transformer-XL (Dai et al., 2019); a
fixed linear attention decay called Alibi (Press et al., 2021);
rotating query and key sequences based on distance such
as RoPE (Su et al., 2023), and XPos (Press et al., 2021).
The proposed method in this work is based on the Rotary
Position Embedding (RoPE) introduced in (Su et al., 2022).

2.2. RoPE
Self-Extend LLM