Unleashing Inﬁnite-Length Input Capacity for Large-scale Language

Models with Self-Controlled Memory System

Xinnian Liang1∗†, Bing Wang1∗†, Hui Huang3†, Shuangzhi Wu2‡

Peihao Wu2, Lu Lu2, Zejun Ma2 and Zhoujun Li1‡

1State Key Lab of Software Development Environment, Beihang University, Beijing, China

2ByteDance AI Lab, Beijing, China

3Harbin Institute of Technology, Harbin, China

{xnliang,bingwang,lizj}@buaa.edu.cn; {wufurui,mazejun}@bytedance.com;

3
2
0
2

 
r
p
A
6
2

 

 
 
]
L
C
.
s
c
[
 
 

1
v
3
4
3
3
1

.

4
0
3
2
:
v
i
X
r
a

Abstract
Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System