title: YAYI 2: Multilingual Open-Source Large Language Models
summary: As the latest advancements in natural language processing, large language
  models (LLMs) have achieved human-level language understanding and generation
  abilities in many real-world tasks, and even have been regarded as a potential
  path to the artificial general intelligence. To better facilitate research on
  LLMs, many open-source LLMs, such as Llama 2 and Falcon, have recently been
  proposed and gained comparable performances to proprietary models. However,
  these models are primarily designed for English scenarios and exhibit poor
  performances in Chinese contexts. In this technical report, we propose YAYI 2,
  including both base and chat models, with 30 billion parameters. YAYI 2 is
  pre-trained from scratch on a multilingual corpus which contains 2.65 trillion
  tokens filtered by our pre-training data processing pipeline. The base model is
  aligned with human values through supervised fine-tuning with millions of
  instructions and reinforcement learning from human feedback. Extensive
  experiments on multiple benchmarks, such as MMLU and CMMLU, consistently
  demonstrate that the proposed YAYI 2 outperforms other similar sized
  open-source models.
paper_authors._id: 658a4b116cf9325ae6fede3d
paper_authors.name: Yin Luo
paper_authors._id: 658a4b116cf9325ae6fede3e
paper_authors.name: Qingchao Kong
paper_authors._id: 658a4b116cf9325ae6fede3f
paper_authors.name: Nan Xu
paper_authors._id: 658a4b116cf9325ae6fede40
paper_authors.name: Jia Cao
paper_authors._id: 658a4b116cf9325ae6fede41
paper_authors.name: Bao Hao
paper_authors._id: 658a4b116cf9325ae6fede42
paper_authors.name: Baoyu Qu
paper_authors._id: 658a4b116cf9325ae6fede43
paper_authors.name: Bo Chen
paper_authors._id: 658a4b116cf9325ae6fede44
paper_authors.name: Chao Zhu
paper_authors._id: 658a4b116cf9325ae6fede45
paper_authors.name: Chenyang Zhao
paper_authors._id: 658a4b116cf9325ae6fede46
paper_authors.name: Donglei Zhang
paper_authors._id: 658a4b116cf9325ae6fede47
paper_authors.name: Fan Feng
paper_authors._id: 658a4b116cf9325ae6fede48
paper_authors.name: Feifei Zhao
paper_authors._id: 658a4b116cf9325ae6fede49
paper_authors.name: Hailong Sun
paper_authors._id: 658a4b116cf9325ae6fede4a
paper_authors.name: Hanxuan Yang
paper_authors._id: 658a4b116cf9325ae6fede4b
paper_authors.name: Haojun Pan
paper_authors._id: 658a4b116cf9325ae6fede4c
paper_authors.name: Hongyu Liu
paper_authors._id: 658a4b116cf9325ae6fede4d
paper_authors.name: Jianbin Guo
paper_authors._id: 658a4b116cf9325ae6fede4e
paper_authors.name: Jiangtao Du
paper_authors._id: 658a4b116cf9325ae6fede4f
paper_authors.name: Jingyi Wang
paper_authors._id: 658a4b116cf9325ae6fede50
paper_authors.name: Junfeng Li
paper_authors._id: 658a4b116cf9325ae6fede51
paper_authors.name: Lei Sun
paper_authors._id: 658a4b116cf9325ae6fede52
paper_authors.name: Liduo Liu
paper_authors._id: 658a4b116cf9325ae6fede53
paper_authors.name: Lifeng Dong
paper_authors._id: 658a4b116cf9325ae6fede54
paper_authors.name: Lili Liu
paper_authors._id: 658a4b116cf9325ae6fede55
paper_authors.name: Lin Wang
paper_authors._id: 658a4b116cf9325ae6fede56
paper_authors.name: Liwen Zhang
paper_authors._id: 658a4b116cf9325ae6fede57
paper_authors.name: Minzheng Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T08:17:47.653Z
paper_authors._id: 64bcc373ef8c0e42bf16acc5
paper_authors.avatarUrl: /avatars/873308203d28115ae1a9e4d0e26508f4.svg
paper_authors.fullname: mz.w
paper_authors.type: user
paper_authors.user: iiiiwis
paper_authors._id: 658a4b116cf9325ae6fede58
paper_authors.name: Pin Wang
paper_authors._id: 658a4b116cf9325ae6fede59
paper_authors.name: Ping Yu
paper_authors._id: 658a4b116cf9325ae6fede5a
paper_authors.name: Qingxiao Li
paper_authors._id: 658a4b116cf9325ae6fede5b
paper_authors.name: Rui Yan
paper_authors._id: 658a4b116cf9325ae6fede5c
paper_authors.name: Rui Zou
paper_authors._id: 658a4b116cf9325ae6fede5d
paper_authors.name: Ruiqun Li
paper_authors._id: 658a4b116cf9325ae6fede5e
paper_authors.name: Taiwen Huang
paper_authors._id: 658a4b116cf9325ae6fede5f
paper_authors.name: Xiaodong Wang
paper_authors._id: 658a4b116cf9325ae6fede60
paper_authors.name: Xiaofei Wu
paper_authors._id: 658a4b116cf9325ae6fede61
paper_authors.name: Xin Peng
paper_authors._id: 658a4b116cf9325ae6fede62
paper_authors.name: Xina Zhang
paper_authors._id: 658a4b116cf9325ae6fede63
paper_authors.name: Xing Fang
paper_authors._id: 658a4b116cf9325ae6fede64
paper_authors.name: Xinglin Xiao
paper_authors._id: 658a4b116cf9325ae6fede65
paper_authors.name: Yanni Hao
paper_authors._id: 658a4b116cf9325ae6fede66
paper_authors.name: Yao Dong
paper_authors._id: 658a4b116cf9325ae6fede67
paper_authors.name: Yigang Wang
paper_authors._id: 658a4b116cf9325ae6fede68
paper_authors.name: Ying Liu
paper_authors._id: 658a4b116cf9325ae6fede69
paper_authors.name: Yongyu Jiang
paper_authors._id: 658a4b116cf9325ae6fede6a
paper_authors.name: Yungan Wang
paper_authors._id: 658a4b116cf9325ae6fede6b
paper_authors.name: Yuqi Wang
paper_authors._id: 658a4b116cf9325ae6fede6c
paper_authors.name: Zhangsheng Wang
paper_authors._id: 658a4b116cf9325ae6fede6d
paper_authors.name: Zhaoxin Yu
paper_authors._id: 658a4b116cf9325ae6fede6e
paper_authors.name: Zhen Luo
paper_authors._id: 658a4b116cf9325ae6fede6f
paper_authors.name: Wenji Mao
paper_authors._id: 658a4b116cf9325ae6fede70
paper_authors.name: Lei Wang
paper_authors._id: 658a4b116cf9325ae6fede71
paper_authors.name: Dajun Zeng
paper_title: YAYI 2: Multilingual Open-Source Large Language Models
paper_summary: As the latest advancements in natural language processing, large language
  models (LLMs) have achieved human-level language understanding and generation
  abilities in many real-world tasks, and even have been regarded as a potential
  path to the artificial general intelligence. To better facilitate research on
  LLMs, many open-source LLMs, such as Llama 2 and Falcon, have recently been
  proposed and gained comparable performances to proprietary models. However,
  these models are primarily designed for English scenarios and exhibit poor
  performances in Chinese contexts. In this technical report, we propose YAYI 2,
  including both base and chat models, with 30 billion parameters. YAYI 2 is
  pre-trained from scratch on a multilingual corpus which contains 2.65 trillion
  tokens filtered by our pre-training data processing pipeline. The base model is
  aligned with human values through supervised fine-tuning with millions of
  instructions and reinforcement learning from human feedback. Extensive
  experiments on multiple benchmarks, such as MMLU and CMMLU, consistently
  demonstrate that the proposed YAYI 2 outperforms other similar sized
  open-source models.