1

Introduction

Recently, Large-scale Language Models (LLMs)
have attracted signiﬁcant attention due to their
remarkable performance in a diverse range of
tasks (Brown et al., 2020a; Zeng et al., 2023;

*The authors contribute equally and the order is random.
†Contribution during internship at ByteDance Inc.
‡Corresponding Author
1Working in progress.
2https://github.com/wbbeyourself/SCM4LLMs

Figure 1: An example comparing ChatGPT and our
SCM system. The orange text indicates the user’s cur-
rent question, bold text denotes related historical infor-
mation, red text indicates an erroneous response from
ChatGPT, and green text denotes the correct response
from our system.
Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System