Long Text Sequence Processing. Handling long
text sequences has been a persistent challenge in
natural language processing tasks. This problem
has become even more prominent with the advent
of pre-training and LLMs, as the ﬁxed input length
during pre-training and the high costs of expanding
it during the pre-training stage limit the ability to
process longer inputs. Existing solutions primar-
ily involve replacing the Attention structure during
pre-training to reduce computational costs and ex-
panding the pre-training sequence length (Beltagy
et al., 2020; Zaheer et al., 2021; Guo et al., 2022;
Phang et al., 2022; Dong et al., 2023). Another al-
ternative approach (Press et al., 2022) uses special
positional encoding during pre-training to enable
the model to learn relative positions and handle
longer input texts during inference.
Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System