Alec Radford, Jeff Wu, Rewon Child, David Luan,
Dario Amodei, and Ilya Sutskever. 2019. Language
models are unsupervised multitask learners.

Colin Raffel, Noam Shazeer, Adam Roberts, Kather-
ine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. 2020. Exploring
the limits of transfer learning with a uniﬁed text-to-
text transformer. Journal of Machine Learning Re-
search, 21(140):1–67.

Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing
Huang, Yadao Wang, Weichao Wang, Pengfei Li,
Xiaoda Zhang, Alexander Podolskiy, Grigory Arshi-
nov, Andrey Bout, Irina Piontkovskaya, Jiansheng
Wei, Xin Jiang, Teng Su, Qun Liu, and Jun Yao.
2023. Pangu-Σ: Towards trillion parameter lan-
guage model with sparse heterogeneous computing.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec
Radford, and Oleg Klimov. 2017. Proximal policy
optimization algorithms.
Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System