2 Related Work
Large-scale Language Models. Large-scale
Language Models (LLMs) are language models
trained on massive amounts of text data, using the

3In this study, we utilize OpenAI gpt-3.5-turbo-0301.

Transformer (Vaswani et al., 2017) architecture as
their foundation. The earliest Transformer-based
pre-trained language model was GPT-1 (Radford
et al., 2018). Subsequently, GPT-2 (Radford et al.,
2019) and GPT-3 (Brown et al., 2020b) were de-
veloped with gradually increasing parameter sizes.
GPT-3 has the largest scale, with 175B parameters,
along with emergent abilities (Wei et al., 2022b,c),
which has attracted the attention of both academia
and industry.
Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System