title: InternLM2 Technical Report
summary: The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has
  sparked discussions on the advent of Artificial General Intelligence (AGI).
  However, replicating such advancements in open-source models has been
  challenging. This paper introduces InternLM2, an open-source LLM that
  outperforms its predecessors in comprehensive evaluations across 6 dimensions
  and 30 benchmarks, long-context modeling, and open-ended subjective evaluations
  through innovative pre-training and optimization techniques. The pre-training
  process of InternLM2 is meticulously detailed, highlighting the preparation of
  diverse data types including text, code, and long-context data. InternLM2
  efficiently captures long-term dependencies, initially trained on 4k tokens
  before advancing to 32k tokens in pre-training and fine-tuning stages,
  exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test.
  InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel
  Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF)
  strategy that addresses conflicting human preferences and reward hacking. By
  releasing InternLM2 models in different training stages and model sizes, we
  provide the community with insights into the model's evolution.
paper_authors._id: 660396b0126abce2e4d76dc8
paper_authors.name: Zheng Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:02:18.432Z
paper_authors._id: 6535d28e25a58aec65c2d9e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6535d28e25a58aec65c2d9e1/qtcXmY3H96rpcqNULgxQH.jpeg
paper_authors.fullname: Zheng Cai
paper_authors.type: user
paper_authors.user: zigzagcai
paper_authors._id: 660396b0126abce2e4d76dc9
paper_authors.name: Maosong Cao
paper_authors._id: 660396b0126abce2e4d76dca
paper_authors.name: Haojiong Chen
paper_authors._id: 660396b0126abce2e4d76dcb
paper_authors.name: Kai Chen
paper_authors._id: 660396b0126abce2e4d76dcc
paper_authors.name: Keyu Chen
paper_authors._id: 660396b0126abce2e4d76dcd
paper_authors.name: Xin Chen
paper_authors._id: 660396b0126abce2e4d76dce
paper_authors.name: Xun Chen
paper_authors._id: 660396b0126abce2e4d76dcf
paper_authors.name: Zehui Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:13:40.632Z
paper_authors._id: 64892d31cbda0d1cdb956897
paper_authors.avatarUrl: /avatars/3cdafe03a8295124636347d15a099aaf.svg
paper_authors.fullname: Zehui Chen
paper_authors.type: user
paper_authors.user: lovesnowbest
paper_authors._id: 660396b0126abce2e4d76dd0
paper_authors.name: Zhi Chen
paper_authors._id: 660396b0126abce2e4d76dd1
paper_authors.name: Pei Chu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:05:33.845Z
paper_authors._id: 64c9beb2904317f42de06dd8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c9beb2904317f42de06dd8/he3rxfyzfwEd1vLuK6_o2.jpeg
paper_authors.fullname: Pei Chu
paper_authors.type: user
paper_authors.user: chupei
paper_authors._id: 660396b0126abce2e4d76dd2
paper_authors.name: Xiaoyi Dong
paper_authors._id: 660396b0126abce2e4d76dd3
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:13:09.092Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 660396b0126abce2e4d76dd4
paper_authors.name: Qi Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T08:04:49.450Z
paper_authors._id: 64b8a72952b7353d8c669086
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b8a72952b7353d8c669086/3PUTNmx9kd17gtZ9-Yviw.jpeg
paper_authors.fullname: Qi Fan
paper_authors.type: user
paper_authors.user: fanqiNO1
paper_authors._id: 660396b0126abce2e4d76dd5
paper_authors.name: Zhaoye Fei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:15:09.534Z
paper_authors._id: 629ef8544313a7c1dd671130
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/629ef8544313a7c1dd671130/i5xfHIgELcuO1Ew19ebTw.png
paper_authors.fullname: Zhaoye Fei
paper_authors.type: user
paper_authors.user: ngc7293
paper_authors._id: 660396b0126abce2e4d76dd6
paper_authors.name: Yang Gao
paper_authors._id: 660396b0126abce2e4d76dd7
paper_authors.name: Jiaye Ge
paper_authors._id: 660396b0126abce2e4d76dd8
paper_authors.name: Chenya Gu
paper_authors._id: 660396b0126abce2e4d76dd9
paper_authors.name: Yuzhe Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:03:29.372Z
paper_authors._id: 6601196cc91ba4c08ad6e270
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6601196cc91ba4c08ad6e270/X2YPNzUOQXBz5Gv-xR9LW.jpeg
paper_authors.fullname: yuzhe gu
paper_authors.type: user
paper_authors.user: vanilla1116
paper_authors._id: 660396b0126abce2e4d76dda
paper_authors.name: Tao Gui
paper_authors._id: 660396b0126abce2e4d76ddb
paper_authors.name: Aijia Guo
paper_authors._id: 660396b0126abce2e4d76ddc
paper_authors.name: Qipeng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:16:02.474Z
paper_authors._id: 6491cd52b1e5d3444528edb1
paper_authors.avatarUrl: /avatars/a85635d886c7f157b6723dec5c01c030.svg
paper_authors.fullname: Qipeng Guo
paper_authors.type: user
paper_authors.user: QipengGuo
paper_authors._id: 660396b0126abce2e4d76ddd
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:16:09.185Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 660396b0126abce2e4d76dde
paper_authors.name: Yingfan Hu
paper_authors._id: 660396b0126abce2e4d76ddf
paper_authors.name: Ting Huang
paper_authors._id: 660396b0126abce2e4d76de0
paper_authors.name: Tao Jiang
paper_authors._id: 660396b0126abce2e4d76de1
paper_authors.name: Penglong Jiao
paper_authors._id: 660396b0126abce2e4d76de2
paper_authors.name: Zhenjiang Jin
paper_authors._id: 660396b0126abce2e4d76de3
paper_authors.name: Zhikai Lei
paper_authors._id: 660396b0126abce2e4d76de4
paper_authors.name: Jiaxing Li
paper_authors._id: 660396b0126abce2e4d76de5
paper_authors.name: Jingwen Li
paper_authors._id: 660396b0126abce2e4d76de6
paper_authors.name: Linyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:09:02.780Z
paper_authors._id: 6698ad87a9778d20ff373142
paper_authors.avatarUrl: /avatars/5e3546d124e8c4f4636a41eab6f0cf68.svg
paper_authors.fullname: linyangli
paper_authors.type: user
paper_authors.user: linyangli
paper_authors._id: 660396b0126abce2e4d76de7
paper_authors.name: Shuaibin Li
paper_authors._id: 660396b0126abce2e4d76de8
paper_authors.name: Wei Li
paper_authors._id: 660396b0126abce2e4d76de9
paper_authors.name: Yining Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:18:05.726Z
paper_authors._id: 649988726677f66c2b486392
paper_authors.avatarUrl: /avatars/b649a77370660e129726e29504daba34.svg
paper_authors.fullname: Yining Li
paper_authors.type: user
paper_authors.user: ly015
paper_authors._id: 660396b0126abce2e4d76dea
paper_authors.name: Hongwei Liu
paper_authors._id: 660396b0126abce2e4d76deb
paper_authors.name: Jiangning Liu
paper_authors._id: 660396b0126abce2e4d76dec
paper_authors.name: Jiawei Hong
paper_authors._id: 660396b0126abce2e4d76ded
paper_authors.name: Kaiwen Liu
paper_authors._id: 660396b0126abce2e4d76dee
paper_authors.name: Kuikun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-02T07:35:12.402Z
paper_authors._id: 63fd691794cc8f815d50c112
paper_authors.avatarUrl: /avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: Harold-lkk
paper_authors._id: 660396b0126abce2e4d76def
paper_authors.name: Xiaoran Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-15T10:13:03.333Z
paper_authors._id: 64f033ef82c6eea604c4da8b
paper_authors.avatarUrl: /avatars/51b93fea7fd68b4274ee03701245dcca.svg
paper_authors.fullname: Liu Xiaoran
paper_authors.type: user
paper_authors.user: LiuXR
paper_authors._id: 660396b0126abce2e4d76df0
paper_authors.name: Chengqi Lv
paper_authors._id: 660396b0126abce2e4d76df1
paper_authors.name: Haijun Lv
paper_authors._id: 660396b0126abce2e4d76df2
paper_authors.name: Kai Lv
paper_authors._id: 660396b0126abce2e4d76df3
paper_authors.name: Li Ma
paper_authors._id: 660396b0126abce2e4d76df4
paper_authors.name: Runyuan Ma
paper_authors._id: 660396b0126abce2e4d76df5
paper_authors.name: Zerun Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T08:43:41.444Z
paper_authors._id: 634fb993685e43e02d2544a2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634fb993685e43e02d2544a2/L9MuXYlXMBuQgnwXd4lCU.jpeg
paper_authors.fullname: mzr1996
paper_authors.type: user
paper_authors.user: mzr1996
paper_authors._id: 660396b0126abce2e4d76df6
paper_authors.name: Wenchang Ning
paper_authors._id: 660396b0126abce2e4d76df7
paper_authors.name: Linke Ouyang
paper_authors._id: 660396b0126abce2e4d76df8
paper_authors.name: Jiantao Qiu
paper_authors._id: 660396b0126abce2e4d76df9
paper_authors.name: Yuan Qu
paper_authors._id: 660396b0126abce2e4d76dfa
paper_authors.name: Fukai Shang
paper_authors._id: 660396b0126abce2e4d76dfb
paper_authors.name: Yunfan Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:09:05.069Z
paper_authors._id: 613b08710fcccdd3b430478a
paper_authors.avatarUrl: /avatars/6bb368110cd50996b26046cd09ab9bcc.svg
paper_authors.fullname: Yunfan Shao
paper_authors.type: user
paper_authors.user: yf
paper_authors._id: 660396b0126abce2e4d76dfc
paper_authors.name: Demin Song
paper_authors._id: 660396b0126abce2e4d76dfd
paper_authors.name: Zifan Song
paper_authors._id: 660396b0126abce2e4d76dfe
paper_authors.name: Zhihao Sui
paper_authors._id: 660396b0126abce2e4d76dff
paper_authors.name: Peng Sun
paper_authors._id: 660396b0126abce2e4d76e00
paper_authors.name: Yu Sun
paper_authors._id: 660396b0126abce2e4d76e01
paper_authors.name: Huanze Tang
paper_authors._id: 660396b0126abce2e4d76e02
paper_authors.name: Bin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:17:29.527Z
paper_authors._id: 63ae9ff5557befe297a76f90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1672388558183-noauth.jpeg
paper_authors.fullname: Bin Wang
paper_authors.type: user
paper_authors.user: wanderkid
paper_authors._id: 660396b0126abce2e4d76e03
paper_authors.name: Guoteng Wang
paper_authors._id: 660396b0126abce2e4d76e04
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:06:38.421Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_authors._id: 660396b0126abce2e4d76e05
paper_authors.name: Jiayu Wang
paper_authors._id: 660396b0126abce2e4d76e06
paper_authors.name: Rui Wang
paper_authors._id: 660396b0126abce2e4d76e07
paper_authors.name: Yudong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-09T07:24:04.427Z
paper_authors._id: 63be286fb3b8c44f8cecc16f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63be286fb3b8c44f8cecc16f/1CIkfEKoTnBYdYDSuQ8AT.jpeg
paper_authors.fullname: Yudong Wang
paper_authors.type: user
paper_authors.user: BigDong
paper_authors._id: 660396b0126abce2e4d76e08
paper_authors.name: Ziyi Wang
paper_authors._id: 660396b0126abce2e4d76e09
paper_authors.name: Xingjian Wei
paper_authors._id: 660396b0126abce2e4d76e0a
paper_authors.name: Qizhen Weng
paper_authors._id: 660396b0126abce2e4d76e0b
paper_authors.name: Fan Wu
paper_authors._id: 660396b0126abce2e4d76e0c
paper_authors.name: Yingtong Xiong
paper_authors._id: 660396b0126abce2e4d76e0d
paper_authors.name: Chao Xu
paper_authors._id: 660396b0126abce2e4d76e0e
paper_authors.name: Ruiliang Xu
paper_authors._id: 660396b0126abce2e4d76e0f
paper_authors.name: Hang Yan
paper_authors._id: 660396b0126abce2e4d76e10
paper_authors.name: Yirong Yan
paper_authors._id: 660396b0126abce2e4d76e11
paper_authors.name: Xiaogui Yang
paper_authors._id: 660396b0126abce2e4d76e12
paper_authors.name: Haochen Ye
paper_authors._id: 660396b0126abce2e4d76e13
paper_authors.name: Huaiyuan Ying
paper_authors._id: 660396b0126abce2e4d76e14
paper_authors.name: Jia Yu
paper_authors._id: 660396b0126abce2e4d76e15
paper_authors.name: Jing Yu
paper_authors._id: 660396b0126abce2e4d76e16
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-23T09:09:41.778Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 660396b0126abce2e4d76e17
paper_authors.name: Chuyu Zhang
paper_authors._id: 660396b0126abce2e4d76e18
paper_authors.name: Li Zhang
paper_authors._id: 660396b0126abce2e4d76e19
paper_authors.name: Pan Zhang
paper_authors._id: 660396b0126abce2e4d76e1a
paper_authors.name: Peng Zhang
paper_authors._id: 660396b0126abce2e4d76e1b
paper_authors.name: Ruijie Zhang
paper_authors._id: 660396b0126abce2e4d76e1c
paper_authors.name: Shuo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T13:29:05.639Z
paper_authors._id: 6334f2f1259c518276efa730
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6334f2f1259c518276efa730/z_SH_OBkDyj4RCN9mqsKS.jpeg
paper_authors.fullname: Shuo Zhang
paper_authors.type: user
paper_authors.user: Meteonis
paper_authors._id: 660396b0126abce2e4d76e1d
paper_authors.name: Songyang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:26:00.336Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 660396b0126abce2e4d76e1e
paper_authors.name: Wenjian Zhang
paper_authors._id: 660396b0126abce2e4d76e1f
paper_authors.name: Wenwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:25:05.529Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 660396b0126abce2e4d76e20
paper_authors.name: Xingcheng Zhang
paper_authors._id: 660396b0126abce2e4d76e21
paper_authors.name: Xinyue Zhang
paper_authors._id: 660396b0126abce2e4d76e22
paper_authors.name: Hui Zhao
paper_authors._id: 660396b0126abce2e4d76e23
paper_authors.name: Qian Zhao
paper_authors._id: 660396b0126abce2e4d76e24
paper_authors.name: Xiaomeng Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:32:27.904Z
paper_authors._id: 668f77a3b3991ac0c308a441
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/668f77a3b3991ac0c308a441/w5-9UAxi32HnrTjW9_fGC.jpeg
paper_authors.fullname: xiaomeng zhao
paper_authors.type: user
paper_authors.user: myhloli
paper_authors._id: 660396b0126abce2e4d76e25
paper_authors.name: Fengzhe Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:24:10.251Z
paper_authors._id: 63f04fb94a788ed1dd89daf4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f04fb94a788ed1dd89daf4/5XNPePED6fTMH0SAM2lhj.jpeg
paper_authors.fullname: zhoufengzhe
paper_authors.type: user
paper_authors.user: Leymore
paper_authors._id: 660396b0126abce2e4d76e26
paper_authors.name: Zaida Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:23:58.339Z
paper_authors._id: 64409d69518271b0d1c033a6
paper_authors.avatarUrl: /avatars/c79eb36c4ad96286afda834e260a1c09.svg
paper_authors.fullname: zhouzaida
paper_authors.type: user
paper_authors.user: zhouzaida
paper_authors._id: 660396b0126abce2e4d76e27
paper_authors.name: Jingming Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:23:44.582Z
paper_authors._id: 6530c7263976e5f4412ba737
paper_authors.avatarUrl: /avatars/8b4d9d847a9e115c3da8cad629bd0a41.svg
paper_authors.fullname: Jingming Zhuo
paper_authors.type: user
paper_authors.user: JingmingZ
paper_authors._id: 660396b0126abce2e4d76e28
paper_authors.name: Yicheng Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:23:36.551Z
paper_authors._id: 6234238485575ce6ff1f169a
paper_authors.avatarUrl: /avatars/e5fff05f21cdea4e5aebc8ba426fac29.svg
paper_authors.fullname: Yicheng Zou
paper_authors.type: user
paper_authors.user: RowitZou
paper_authors._id: 660396b0126abce2e4d76e29
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:23:28.522Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_authors._id: 660396b0126abce2e4d76e2a
paper_authors.name: Yu Qiao
paper_authors._id: 660396b0126abce2e4d76e2b
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-27T09:23:14.735Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_title: InternLM2 Technical Report
paper_summary: The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has
  sparked discussions on the advent of Artificial General Intelligence (AGI).
  However, replicating such advancements in open-source models has been
  challenging. This paper introduces InternLM2, an open-source LLM that
  outperforms its predecessors in comprehensive evaluations across 6 dimensions
  and 30 benchmarks, long-context modeling, and open-ended subjective evaluations
  through innovative pre-training and optimization techniques. The pre-training
  process of InternLM2 is meticulously detailed, highlighting the preparation of
  diverse data types including text, code, and long-context data. InternLM2
  efficiently captures long-term dependencies, initially trained on 4k tokens
  before advancing to 32k tokens in pre-training and fine-tuning stages,
  exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test.
  InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel
  Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF)
  strategy that addresses conflicting human preferences and reward hacking. By
  releasing InternLM2 models in different training stages and model sizes, we
  provide the community with insights into the model's evolution.
paper_ai_summary: InternLM2 is an open-source LLM that outperforms predecessors through innovative pre-training and optimization techniques, including Supervised Fine-Tuning and Conditional Online Reinforcement Learning from Human Feedback.
paper_ai_keywords: Large Language Models LLMs ChatGPT GPT-4 Artificial General Intelligence AGI pre-training optimization techniques Supervised Fine-Tuning Conditional Online Reinforcement Learning from Human Feedback COOL RLHF long-context modeling evaluation dimensions benchmarks long-term dependencies token training stages