title: Baichuan 2: Open Large-scale Language Models
summary: Large language models (LLMs) have demonstrated remarkable performance on a
  variety of natural language tasks based on just a few examples of natural
  language instructions, reducing the need for extensive feature engineering.
  However, most powerful LLMs are closed-source or limited in their capability
  for languages other than English. In this technical report, we present Baichuan
  2, a series of large-scale multilingual language models containing 7 billion
  and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.
  Baichuan 2 matches or outperforms other open-source models of similar size on
  public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan
  2 excels in vertical domains such as medicine and law. We will release all
  pre-training model checkpoints to benefit the research community in better
  understanding the training dynamics of Baichuan 2.
paper_authors._id: 650a4a35ce81b73e63833df6
paper_authors.name: Aiyuan Yang
paper_authors._id: 650a4a35ce81b73e63833df7
paper_authors.name: Bin Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:47:08.804Z
paper_authors._id: 64f1568afcc7700c949c6154
paper_authors.avatarUrl: /avatars/f23c1922f18731b77679b61aa89b309d.svg
paper_authors.fullname: BinXiao
paper_authors.type: user
paper_authors.user: BinXiao
paper_authors._id: 650a4a35ce81b73e63833df8
paper_authors.name: Bingning Wang
paper_authors._id: 650a4a35ce81b73e63833df9
paper_authors.name: Borong Zhang
paper_authors._id: 650a4a35ce81b73e63833dfa
paper_authors.name: Chao Yin
paper_authors._id: 650a4a35ce81b73e63833dfb
paper_authors.name: Chenxu Lv
paper_authors._id: 650a4a35ce81b73e63833dfc
paper_authors.name: Da Pan
paper_authors._id: 650a4a35ce81b73e63833dfd
paper_authors.name: Dian Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:48:47.942Z
paper_authors._id: 64a383eac9b4db7cc77b55af
paper_authors.avatarUrl: /avatars/b8bb4757c1693aedb86aa763e3a66354.svg
paper_authors.fullname: Hellen Wang
paper_authors.type: user
paper_authors.user: wangdianhellen
paper_authors._id: 650a4a35ce81b73e63833dfe
paper_authors.name: Dong Yan
paper_authors._id: 650a4a35ce81b73e63833dff
paper_authors.name: Fan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T08:17:18.155Z
paper_authors._id: 641c45c921964f8f6d451d16
paper_authors.avatarUrl: /avatars/da06cc603f8f9ee46ddb7dc72aae5bec.svg
paper_authors.fullname: FanYang
paper_authors.type: user
paper_authors.user: fairyang
paper_authors._id: 650a4a35ce81b73e63833e00
paper_authors.name: Fei Deng
paper_authors._id: 650a4a35ce81b73e63833e01
paper_authors.name: Feng Wang
paper_authors._id: 650a4a35ce81b73e63833e02
paper_authors.name: Feng Liu
paper_authors._id: 650a4a35ce81b73e63833e03
paper_authors.name: Guangwei Ai
paper_authors._id: 650a4a35ce81b73e63833e04
paper_authors.name: Guosheng Dong Haizhou Zhao
paper_authors._id: 650a4a35ce81b73e63833e05
paper_authors.name: Hang Xu
paper_authors._id: 650a4a35ce81b73e63833e06
paper_authors.name: Haoze Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T15:05:54.056Z
paper_authors._id: 6436bb0dd58a5ea528c55acb
paper_authors.avatarUrl: /avatars/df17b66780e14e07bbe4625f068a94ad.svg
paper_authors.fullname: Alvin Sun
paper_authors.type: user
paper_authors.user: AlvinSunYooo
paper_authors._id: 650a4a35ce81b73e63833e07
paper_authors.name: Hongda Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:22:42.147Z
paper_authors._id: 642ae59bd651bae3c11e9371
paper_authors.avatarUrl: /avatars/898275d5680a1a1f917f53448dfeedb3.svg
paper_authors.fullname: zhang hongda
paper_authors.type: user
paper_authors.user: hongdaaaaaaaa
paper_authors._id: 650a4a35ce81b73e63833e08
paper_authors.name: Hui Liu
paper_authors._id: 650a4a35ce81b73e63833e09
paper_authors.name: Jiaming Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:24:19.245Z
paper_authors._id: 6429b13816d4d8293c97a949
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6429b13816d4d8293c97a949/LNuQ1gvW2NMmWyIuEpzCA.jpeg
paper_authors.fullname: JiJiaming
paper_authors.type: user
paper_authors.user: jijiaming
paper_authors._id: 650a4a35ce81b73e63833e0a
paper_authors.name: Jian Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:24:52.729Z
paper_authors._id: 62d65139667051e0a29bffe7
paper_authors.avatarUrl: /avatars/0252aa2bcd4cf1c8e4b87e5f164b6da5.svg
paper_authors.fullname: Jian Xie
paper_authors.type: user
paper_authors.user: hsaest
paper_authors._id: 650a4a35ce81b73e63833e0b
paper_authors.name: Juntao Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:25:01.971Z
paper_authors._id: 6426a75336ab74ff2bca0542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6426a75336ab74ff2bca0542/qL3QQofTr4yo8Cs58oYwK.png
paper_authors.fullname: Juntao Dai
paper_authors.type: user
paper_authors.user: calico-1226
paper_authors._id: 650a4a35ce81b73e63833e0c
paper_authors.name: Kun Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:25:11.794Z
paper_authors._id: 64e48059b9455cd53e17fe0c
paper_authors.avatarUrl: /avatars/37863f1c106215a15997b3090ce53ab1.svg
paper_authors.fullname: Kun Fang
paper_authors.type: user
paper_authors.user: KunFang
paper_authors._id: 650a4a35ce81b73e63833e0d
paper_authors.name: Lei Su Liang Song
paper_authors._id: 650a4a35ce81b73e63833e0e
paper_authors.name: Lifeng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:26:02.714Z
paper_authors._id: 6466db5b14e059dde8bbc0a2
paper_authors.avatarUrl: /avatars/daf6aea9ded455c0224002090be904e3.svg
paper_authors.fullname: lili
paper_authors.type: user
paper_authors.user: liulifeng
paper_authors._id: 650a4a35ce81b73e63833e0f
paper_authors.name: Liyun Ru
paper_authors._id: 650a4a35ce81b73e63833e10
paper_authors.name: Luyao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:26:21.707Z
paper_authors._id: 63f8b05ebb02399771116965
paper_authors.avatarUrl: /avatars/edd1227a179d8f0db45b49ca53aeb4e7.svg
paper_authors.fullname: Luyao Ma
paper_authors.type: user
paper_authors.user: leonma
paper_authors._id: 650a4a35ce81b73e63833e11
paper_authors.name: Mang Wang
paper_authors._id: 650a4a35ce81b73e63833e12
paper_authors.name: Mickel Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:26:32.951Z
paper_authors._id: 643f986a18afbc4d1f3e2456
paper_authors.avatarUrl: /avatars/96a6e5f18386a9864dfdded8f25ec72d.svg
paper_authors.fullname: Mickel Liu
paper_authors.type: user
paper_authors.user: mickelliu
paper_authors._id: 650a4a35ce81b73e63833e13
paper_authors.name: MingAn Lin
paper_authors._id: 650a4a35ce81b73e63833e14
paper_authors.name: Nuolan Nie
paper_authors._id: 650a4a35ce81b73e63833e15
paper_authors.name: Peidong Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:27:01.597Z
paper_authors._id: 6487e2e1eec01aee99cf4c10
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6487e2e1eec01aee99cf4c10/1U6zZ2OaUOrR1ueD5yraR.jpeg
paper_authors.fullname: Peidong Guo
paper_authors.type: user
paper_authors.user: GuoPD
paper_authors._id: 650a4a35ce81b73e63833e16
paper_authors.name: Ruiyang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:28:51.811Z
paper_authors._id: 63fca6b85d2bea4588bc9a94
paper_authors.avatarUrl: /avatars/c8ae8c02d6d02af90678e8ee4122eafc.svg
paper_authors.fullname: Ruiyang Sun
paper_authors.type: user
paper_authors.user: RuiyangSun
paper_authors._id: 650a4a35ce81b73e63833e17
paper_authors.name: Tao Zhang
paper_authors._id: 650a4a35ce81b73e63833e18
paper_authors.name: Tianpeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:31:16.763Z
paper_authors._id: 64be4c97ef8c0e42bf448469
paper_authors.avatarUrl: /avatars/a2a45b5f17d3e27c59547312bcf7bb52.svg
paper_authors.fullname: Tianpeng Li
paper_authors.type: user
paper_authors.user: TJU-Tianpengli
paper_authors._id: 650a4a35ce81b73e63833e19
paper_authors.name: Tianyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:31:25.826Z
paper_authors._id: 6350d2b7fc22005c1aafcd13
paper_authors.avatarUrl: /avatars/4401c0ed328c3b919d430f3610719952.svg
paper_authors.fullname: Tianyu Lin
paper_authors.type: user
paper_authors.user: TianyuLin
paper_authors._id: 650a4a35ce81b73e63833e1a
paper_authors.name: Wei Cheng
paper_authors._id: 650a4a35ce81b73e63833e1b
paper_authors.name: Weipeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:33:43.548Z
paper_authors._id: 60d0870f778bafd0fbcfa3fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624280312404-60d0870f778bafd0fbcfa3fa.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Weipeng
paper_authors._id: 650a4a35ce81b73e63833e1c
paper_authors.name: Xiangrong Zeng
paper_authors._id: 650a4a35ce81b73e63833e1d
paper_authors.name: Xiaochuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:34:58.345Z
paper_authors._id: 6425449ace70775b51aecc28
paper_authors.avatarUrl: /avatars/32609ac321cb0a98ace3b35e8a5e61cc.svg
paper_authors.fullname: 5555
paper_authors.type: user
paper_authors.user: wangxiaochuan
paper_authors._id: 650a4a35ce81b73e63833e1e
paper_authors.name: Xiaoxi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:35:16.087Z
paper_authors._id: 646b2092e96a751c52505f3b
paper_authors.avatarUrl: /avatars/4b3e93c9603ebe9822a2aa5b5e39d7b1.svg
paper_authors.fullname: xiaoxi
paper_authors.type: user
paper_authors.user: chenxiaoxi
paper_authors._id: 650a4a35ce81b73e63833e1f
paper_authors.name: Xin Men
paper_authors.statusLastChangedAt: 2023-09-20T12:35:30.937Z
paper_authors._id: 650a4a35ce81b73e63833e20
paper_authors.name: Xin Yu
paper_authors._id: 650a4a35ce81b73e63833e21
paper_authors.name: Xuehai Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:36:25.424Z
paper_authors._id: 64364e87fae2870051496e13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t67EsNoRvRYXKwi0G59oa.jpeg
paper_authors.fullname: Xuehai Pan
paper_authors.type: user
paper_authors.user: XuehaiPan
paper_authors._id: 650a4a35ce81b73e63833e22
paper_authors.name: Yanjun Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:36:38.322Z
paper_authors._id: 64ced7c01720c7a483d0862f
paper_authors.avatarUrl: /avatars/1fae685b7226a0a1c95b2d4993f2ab60.svg
paper_authors.fullname: shenyanjun
paper_authors.type: user
paper_authors.user: zilchshen
paper_authors._id: 650a4a35ce81b73e63833e23
paper_authors.name: Yiding Wang
paper_authors._id: 650a4a35ce81b73e63833e24
paper_authors.name: Yiyu Li
paper_authors.statusLastChangedAt: 2023-09-20T12:36:56.644Z
paper_authors._id: 650a4a35ce81b73e63833e25
paper_authors.name: Youxin Jiang
paper_authors._id: 650a4a35ce81b73e63833e26
paper_authors.name: Yuchen Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:37:14.423Z
paper_authors._id: 6437d48151c7ebfc813c96b4
paper_authors.avatarUrl: /avatars/d5ec60bb8247f1bb9b615382400bc462.svg
paper_authors.fullname: yuchen gao
paper_authors.type: user
paper_authors.user: mervyngo
paper_authors._id: 650a4a35ce81b73e63833e27
paper_authors.name: Yupeng Zhang
paper_authors._id: 650a4a35ce81b73e63833e28
paper_authors.name: Zenan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:37:53.074Z
paper_authors._id: 64020919f98fbc64bcdab85d
paper_authors.avatarUrl: /avatars/530cfe3c16b6bd19e9a7232405e1918a.svg
paper_authors.fullname: zhouzenan
paper_authors.type: user
paper_authors.user: zhouzn
paper_authors._id: 650a4a35ce81b73e63833e29
paper_authors.name: Zhiying Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T12:15:14.603Z
paper_authors._id: 646c36d1942fd7c135df8ff0
paper_authors.avatarUrl: /avatars/9c85c6975e097d7de28538cb6fa091ff.svg
paper_authors.fullname: Zhiying Wu
paper_authors.type: user
paper_authors.user: wzy199520
paper_title: Baichuan 2: Open Large-scale Language Models
paper_summary: Large language models (LLMs) have demonstrated remarkable performance on a
  variety of natural language tasks based on just a few examples of natural
  language instructions, reducing the need for extensive feature engineering.
  However, most powerful LLMs are closed-source or limited in their capability
  for languages other than English. In this technical report, we present Baichuan
  2, a series of large-scale multilingual language models containing 7 billion
  and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.
  Baichuan 2 matches or outperforms other open-source models of similar size on
  public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan
  2 excels in vertical domains such as medicine and law. We will release all
  pre-training model checkpoints to benefit the research community in better
  understanding the training dynamics of Baichuan 2.
paper_ai_summary: Baichuan 2, a series of 7 billion and 13 billion parameter multilingual LLMs, achieves competitive performance on benchmarks and excels in specialized domains, with open-source pre-training checkpoints released.
paper_ai_keywords: large language models multilingual language models pre-training MMLU CMMLU GSM8K HumanEval medicine law training dynamics