Long sequences in LLMs. Scaling Transformers and LLMs to long input sequences has attracted much
recent interest (Dai et al., 2019; Beltagy et al., 2020; Yu et al., 2023; Ding et al., 2023). The context lengths
supported by available models and APIs has seen a steady increase, with StarCoder being trained on 8K
token sequences ((Li et al., 2023), up from the 4K of Allal et al. (2023)), recent GPT versions supporting
16K (gpt-3.5-turbo-16k) and 32K tokens (gpt-4-32k), MPT-7b fine-tuned on 65K tokens (MosaicML, 2023),
and Claude featuring 100K context windows (Anthropic, 2023). Previous research focuses on alleviating the
O(n2) space and time complexity of self-attention (Vaswani et al., 2017) by introducing sparsity patterns,
as well as by encoding positional information in such a way that models can leverage input sizes larger
than those presented at training time (length extrapolation). In our work, we do not rely on hand-crafted
CodeLlama2