InternLM2 Technical Report
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has
sparked discussions on the advent of Artificial General Intelligence (AGI).
However, replicating such advancements in open-source models has been
challenging. This paper introduces InternLM2, an open-source LLM that
outperforms its predecessors in comprehensive evaluations across 6 dimensions
and 30 benchmarks, long-context modeling, and open-ended subjective evaluations
through innovative pre-training and optimization techniques. The pre-training
process of InternLM2 is meticulously detailed, highlighting the preparation of
diverse data types including text, code, and long-context data. InternLM2
efficiently captures long-term dependencies, initially trained on 4k tokens
before advancing to 32k tokens in pre-training and fine-tuning stages,
exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test.
InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel
Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF)
strategy that addresses conflicting human preferences and reward hacking. By
releasing InternLM2 models in different training stages and model sizes, we
provide the community with insights into the model's evolution.
660396b0126abce2e4d76dc8
Zheng Cai
admin_assigned
2024-03-27T09:02:18.432Z
6535d28e25a58aec65c2d9e1
https://cdn-avatars.huggingface.co/v1/production/uploads/6535d28e25a58aec65c2d9e1/qtcXmY3H96rpcqNULgxQH.jpeg
Zheng Cai
user
zigzagcai
660396b0126abce2e4d76dc9
Maosong Cao
660396b0126abce2e4d76dca
Haojiong Chen
660396b0126abce2e4d76dcb
Kai Chen
660396b0126abce2e4d76dcc
Keyu Chen
660396b0126abce2e4d76dcd
Xin Chen
660396b0126abce2e4d76dce
Xun Chen
660396b0126abce2e4d76dcf
Zehui Chen
admin_assigned
2024-03-27T09:13:40.632Z
64892d31cbda0d1cdb956897
/avatars/3cdafe03a8295124636347d15a099aaf.svg
Zehui Chen
user
lovesnowbest
660396b0126abce2e4d76dd0
Zhi Chen
660396b0126abce2e4d76dd1
Pei Chu
claimed_verified
2024-12-17T08:05:33.845Z
64c9beb2904317f42de06dd8
https://cdn-avatars.huggingface.co/v1/production/uploads/64c9beb2904317f42de06dd8/he3rxfyzfwEd1vLuK6_o2.jpeg
Pei Chu
user
chupei
660396b0126abce2e4d76dd2
Xiaoyi Dong
660396b0126abce2e4d76dd3
Haodong Duan
admin_assigned
2024-03-27T09:13:09.092Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
660396b0126abce2e4d76dd4
Qi Fan
claimed_verified
2025-04-15T08:04:49.450Z
64b8a72952b7353d8c669086
https://cdn-avatars.huggingface.co/v1/production/uploads/64b8a72952b7353d8c669086/3PUTNmx9kd17gtZ9-Yviw.jpeg
Qi Fan
user
fanqiNO1
660396b0126abce2e4d76dd5
Zhaoye Fei
admin_assigned
2024-03-27T09:15:09.534Z
629ef8544313a7c1dd671130
https://cdn-avatars.huggingface.co/v1/production/uploads/629ef8544313a7c1dd671130/i5xfHIgELcuO1Ew19ebTw.png
Zhaoye Fei
user
ngc7293
660396b0126abce2e4d76dd6
Yang Gao
660396b0126abce2e4d76dd7
Jiaye Ge
660396b0126abce2e4d76dd8
Chenya Gu
660396b0126abce2e4d76dd9
Yuzhe Gu
claimed_verified
2024-07-15T09:03:29.372Z
6601196cc91ba4c08ad6e270
https://cdn-avatars.huggingface.co/v1/production/uploads/6601196cc91ba4c08ad6e270/X2YPNzUOQXBz5Gv-xR9LW.jpeg
yuzhe gu
user
vanilla1116
660396b0126abce2e4d76dda
Tao Gui
660396b0126abce2e4d76ddb
Aijia Guo
660396b0126abce2e4d76ddc
Qipeng Guo
admin_assigned
2024-03-27T09:16:02.474Z
6491cd52b1e5d3444528edb1
/avatars/a85635d886c7f157b6723dec5c01c030.svg
Qipeng Guo
user
QipengGuo
660396b0126abce2e4d76ddd
Conghui He
admin_assigned
2024-03-27T09:16:09.185Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
660396b0126abce2e4d76dde
Yingfan Hu
660396b0126abce2e4d76ddf
Ting Huang
660396b0126abce2e4d76de0
Tao Jiang
660396b0126abce2e4d76de1
Penglong Jiao
660396b0126abce2e4d76de2
Zhenjiang Jin
660396b0126abce2e4d76de3
Zhikai Lei
660396b0126abce2e4d76de4
Jiaxing Li
660396b0126abce2e4d76de5
Jingwen Li
660396b0126abce2e4d76de6
Linyang Li
claimed_verified
2024-07-18T09:09:02.780Z
6698ad87a9778d20ff373142
/avatars/5e3546d124e8c4f4636a41eab6f0cf68.svg
linyangli
user
linyangli
660396b0126abce2e4d76de7
Shuaibin Li
660396b0126abce2e4d76de8
Wei Li
660396b0126abce2e4d76de9
Yining Li
claimed_verified
2025-04-22T10:18:05.726Z
649988726677f66c2b486392
/avatars/b649a77370660e129726e29504daba34.svg
Yining Li
user
ly015
660396b0126abce2e4d76dea
Hongwei Liu
660396b0126abce2e4d76deb
Jiangning Liu
660396b0126abce2e4d76dec
Jiawei Hong
660396b0126abce2e4d76ded
Kaiwen Liu
660396b0126abce2e4d76dee
Kuikun Liu
claimed_verified
2024-08-02T07:35:12.402Z
63fd691794cc8f815d50c112
/avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
liu
user
Harold-lkk
660396b0126abce2e4d76def
Xiaoran Liu
claimed_verified
2024-08-15T10:13:03.333Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Liu Xiaoran
user
LiuXR
660396b0126abce2e4d76df0
Chengqi Lv
660396b0126abce2e4d76df1
Haijun Lv
660396b0126abce2e4d76df2
Kai Lv
660396b0126abce2e4d76df3
Li Ma
660396b0126abce2e4d76df4
Runyuan Ma
660396b0126abce2e4d76df5
Zerun Ma
claimed_verified
2024-07-22T08:43:41.444Z
634fb993685e43e02d2544a2
https://cdn-avatars.huggingface.co/v1/production/uploads/634fb993685e43e02d2544a2/L9MuXYlXMBuQgnwXd4lCU.jpeg
mzr1996
user
mzr1996
660396b0126abce2e4d76df6
Wenchang Ning
660396b0126abce2e4d76df7
Linke Ouyang
660396b0126abce2e4d76df8
Jiantao Qiu
660396b0126abce2e4d76df9
Yuan Qu
660396b0126abce2e4d76dfa
Fukai Shang
660396b0126abce2e4d76dfb
Yunfan Shao
claimed_verified
2024-07-18T09:09:05.069Z
613b08710fcccdd3b430478a
/avatars/6bb368110cd50996b26046cd09ab9bcc.svg
Yunfan Shao
user
yf
660396b0126abce2e4d76dfc
Demin Song
660396b0126abce2e4d76dfd
Zifan Song
660396b0126abce2e4d76dfe
Zhihao Sui
660396b0126abce2e4d76dff
Peng Sun
660396b0126abce2e4d76e00
Yu Sun
660396b0126abce2e4d76e01
Huanze Tang
660396b0126abce2e4d76e02
Bin Wang
claimed_verified
2024-04-03T09:17:29.527Z
63ae9ff5557befe297a76f90
https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
Bin Wang
user
wanderkid
660396b0126abce2e4d76e03
Guoteng Wang
660396b0126abce2e4d76e04
Jiaqi Wang
claimed_verified
2024-06-09T20:06:38.421Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
660396b0126abce2e4d76e05
Jiayu Wang
660396b0126abce2e4d76e06
Rui Wang
660396b0126abce2e4d76e07
Yudong Wang
claimed_verified
2025-05-09T07:24:04.427Z
63be286fb3b8c44f8cecc16f
https://cdn-avatars.huggingface.co/v1/production/uploads/63be286fb3b8c44f8cecc16f/1CIkfEKoTnBYdYDSuQ8AT.jpeg
Yudong Wang
user
BigDong
660396b0126abce2e4d76e08
Ziyi Wang
660396b0126abce2e4d76e09
Xingjian Wei
660396b0126abce2e4d76e0a
Qizhen Weng
660396b0126abce2e4d76e0b
Fan Wu
660396b0126abce2e4d76e0c
Yingtong Xiong
660396b0126abce2e4d76e0d
Chao Xu
660396b0126abce2e4d76e0e
Ruiliang Xu
660396b0126abce2e4d76e0f
Hang Yan
660396b0126abce2e4d76e10
Yirong Yan
660396b0126abce2e4d76e11
Xiaogui Yang
660396b0126abce2e4d76e12
Haochen Ye
660396b0126abce2e4d76e13
Huaiyuan Ying
660396b0126abce2e4d76e14
Jia Yu
660396b0126abce2e4d76e15
Jing Yu
660396b0126abce2e4d76e16
Yuhang Zang
claimed_verified
2024-04-23T09:09:41.778Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
660396b0126abce2e4d76e17
Chuyu Zhang
660396b0126abce2e4d76e18
Li Zhang
660396b0126abce2e4d76e19
Pan Zhang
660396b0126abce2e4d76e1a
Peng Zhang
660396b0126abce2e4d76e1b
Ruijie Zhang
660396b0126abce2e4d76e1c
Shuo Zhang
claimed_verified
2024-06-18T13:29:05.639Z
6334f2f1259c518276efa730
https://cdn-avatars.huggingface.co/v1/production/uploads/6334f2f1259c518276efa730/z_SH_OBkDyj4RCN9mqsKS.jpeg
Shuo Zhang
user
Meteonis
660396b0126abce2e4d76e1d
Songyang Zhang
admin_assigned
2024-03-27T09:26:00.336Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
660396b0126abce2e4d76e1e
Wenjian Zhang
660396b0126abce2e4d76e1f
Wenwei Zhang
admin_assigned
2024-03-27T09:25:05.529Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
660396b0126abce2e4d76e20
Xingcheng Zhang
660396b0126abce2e4d76e21
Xinyue Zhang
660396b0126abce2e4d76e22
Hui Zhao
660396b0126abce2e4d76e23
Qian Zhao
660396b0126abce2e4d76e24
Xiaomeng Zhao
claimed_verified
2024-10-07T07:32:27.904Z
668f77a3b3991ac0c308a441
https://cdn-avatars.huggingface.co/v1/production/uploads/668f77a3b3991ac0c308a441/w5-9UAxi32HnrTjW9_fGC.jpeg
xiaomeng zhao
user
myhloli
660396b0126abce2e4d76e25
Fengzhe Zhou
admin_assigned
2024-03-27T09:24:10.251Z
63f04fb94a788ed1dd89daf4
https://cdn-avatars.huggingface.co/v1/production/uploads/63f04fb94a788ed1dd89daf4/5XNPePED6fTMH0SAM2lhj.jpeg
zhoufengzhe
user
Leymore
660396b0126abce2e4d76e26
Zaida Zhou
admin_assigned
2024-03-27T09:23:58.339Z
64409d69518271b0d1c033a6
/avatars/c79eb36c4ad96286afda834e260a1c09.svg
zhouzaida
user
zhouzaida
660396b0126abce2e4d76e27
Jingming Zhuo
admin_assigned
2024-03-27T09:23:44.582Z
6530c7263976e5f4412ba737
/avatars/8b4d9d847a9e115c3da8cad629bd0a41.svg
Jingming Zhuo
user
JingmingZ
660396b0126abce2e4d76e28
Yicheng Zou
admin_assigned
2024-03-27T09:23:36.551Z
6234238485575ce6ff1f169a
/avatars/e5fff05f21cdea4e5aebc8ba426fac29.svg
Yicheng Zou
user
RowitZou
660396b0126abce2e4d76e29
Xipeng Qiu
admin_assigned
2024-03-27T09:23:28.522Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
660396b0126abce2e4d76e2a
Yu Qiao
660396b0126abce2e4d76e2b
Dahua Lin
admin_assigned
2024-03-27T09:23:14.735Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
InternLM2 Technical Report
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has
sparked discussions on the advent of Artificial General Intelligence (AGI).
However, replicating such advancements in open-source models has been
challenging. This paper introduces InternLM2, an open-source LLM that
outperforms its predecessors in comprehensive evaluations across 6 dimensions
and 30 benchmarks, long-context modeling, and open-ended subjective evaluations
through innovative pre-training and optimization techniques. The pre-training
process of InternLM2 is meticulously detailed, highlighting the preparation of
diverse data types including text, code, and long-context data. InternLM2
efficiently captures long-term dependencies, initially trained on 4k tokens
before advancing to 32k tokens in pre-training and fine-tuning stages,
exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test.
InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel
Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF)
strategy that addresses conflicting human preferences and reward hacking. By
releasing InternLM2 models in different training stages and model sizes, we
provide the community with insights into the model's evolution.
InternLM2 is an open-source LLM that outperforms predecessors through innovative pre-training and optimization techniques, including Supervised Fine-Tuning and Conditional Online Reinforcement Learning from Human Feedback.
Large Language Models LLMs ChatGPT GPT-4 Artificial General Intelligence AGI pre-training optimization techniques Supervised Fine-Tuning Conditional Online Reinforcement Learning from Human Feedback COOL RLHF long-context modeling evaluation dimensions benchmarks long-term dependencies token training stages