YAYI 2: Multilingual Open-Source Large Language Models
As the latest advancements in natural language processing, large language
models (LLMs) have achieved human-level language understanding and generation
abilities in many real-world tasks, and even have been regarded as a potential
path to the artificial general intelligence. To better facilitate research on
LLMs, many open-source LLMs, such as Llama 2 and Falcon, have recently been
proposed and gained comparable performances to proprietary models. However,
these models are primarily designed for English scenarios and exhibit poor
performances in Chinese contexts. In this technical report, we propose YAYI 2,
including both base and chat models, with 30 billion parameters. YAYI 2 is
pre-trained from scratch on a multilingual corpus which contains 2.65 trillion
tokens filtered by our pre-training data processing pipeline. The base model is
aligned with human values through supervised fine-tuning with millions of
instructions and reinforcement learning from human feedback. Extensive
experiments on multiple benchmarks, such as MMLU and CMMLU, consistently
demonstrate that the proposed YAYI 2 outperforms other similar sized
open-source models.
658a4b116cf9325ae6fede3d
Yin Luo
658a4b116cf9325ae6fede3e
Qingchao Kong
658a4b116cf9325ae6fede3f
Nan Xu
658a4b116cf9325ae6fede40
Jia Cao
658a4b116cf9325ae6fede41
Bao Hao
658a4b116cf9325ae6fede42
Baoyu Qu
658a4b116cf9325ae6fede43
Bo Chen
658a4b116cf9325ae6fede44
Chao Zhu
658a4b116cf9325ae6fede45
Chenyang Zhao
658a4b116cf9325ae6fede46
Donglei Zhang
658a4b116cf9325ae6fede47
Fan Feng
658a4b116cf9325ae6fede48
Feifei Zhao
658a4b116cf9325ae6fede49
Hailong Sun
658a4b116cf9325ae6fede4a
Hanxuan Yang
658a4b116cf9325ae6fede4b
Haojun Pan
658a4b116cf9325ae6fede4c
Hongyu Liu
658a4b116cf9325ae6fede4d
Jianbin Guo
658a4b116cf9325ae6fede4e
Jiangtao Du
658a4b116cf9325ae6fede4f
Jingyi Wang
658a4b116cf9325ae6fede50
Junfeng Li
658a4b116cf9325ae6fede51
Lei Sun
658a4b116cf9325ae6fede52
Liduo Liu
658a4b116cf9325ae6fede53
Lifeng Dong
658a4b116cf9325ae6fede54
Lili Liu
658a4b116cf9325ae6fede55
Lin Wang
658a4b116cf9325ae6fede56
Liwen Zhang
658a4b116cf9325ae6fede57
Minzheng Wang
claimed_verified
2024-10-08T08:17:47.653Z
64bcc373ef8c0e42bf16acc5
/avatars/873308203d28115ae1a9e4d0e26508f4.svg
mz.w
user
iiiiwis
658a4b116cf9325ae6fede58
Pin Wang
658a4b116cf9325ae6fede59
Ping Yu
658a4b116cf9325ae6fede5a
Qingxiao Li
658a4b116cf9325ae6fede5b
Rui Yan
658a4b116cf9325ae6fede5c
Rui Zou
658a4b116cf9325ae6fede5d
Ruiqun Li
658a4b116cf9325ae6fede5e
Taiwen Huang
658a4b116cf9325ae6fede5f
Xiaodong Wang
658a4b116cf9325ae6fede60
Xiaofei Wu
658a4b116cf9325ae6fede61
Xin Peng
658a4b116cf9325ae6fede62
Xina Zhang
658a4b116cf9325ae6fede63
Xing Fang
658a4b116cf9325ae6fede64
Xinglin Xiao
658a4b116cf9325ae6fede65
Yanni Hao
658a4b116cf9325ae6fede66
Yao Dong
658a4b116cf9325ae6fede67
Yigang Wang
658a4b116cf9325ae6fede68
Ying Liu
658a4b116cf9325ae6fede69
Yongyu Jiang
658a4b116cf9325ae6fede6a
Yungan Wang
658a4b116cf9325ae6fede6b
Yuqi Wang
658a4b116cf9325ae6fede6c
Zhangsheng Wang
658a4b116cf9325ae6fede6d
Zhaoxin Yu
658a4b116cf9325ae6fede6e
Zhen Luo
658a4b116cf9325ae6fede6f
Wenji Mao
658a4b116cf9325ae6fede70
Lei Wang
658a4b116cf9325ae6fede71
Dajun Zeng
YAYI 2: Multilingual Open-Source Large Language Models
As the latest advancements in natural language processing, large language
models (LLMs) have achieved human-level language understanding and generation
abilities in many real-world tasks, and even have been regarded as a potential
path to the artificial general intelligence. To better facilitate research on
LLMs, many open-source LLMs, such as Llama 2 and Falcon, have recently been
proposed and gained comparable performances to proprietary models. However,
these models are primarily designed for English scenarios and exhibit poor
performances in Chinese contexts. In this technical report, we propose YAYI 2,
including both base and chat models, with 30 billion parameters. YAYI 2 is
pre-trained from scratch on a multilingual corpus which contains 2.65 trillion
tokens filtered by our pre-training data processing pipeline. The base model is
aligned with human values through supervised fine-tuning with millions of
instructions and reinforcement learning from human feedback. Extensive
experiments on multiple benchmarks, such as MMLU and CMMLU, consistently
demonstrate that the proposed YAYI 2 outperforms other similar sized
open-source models.
YAYI 2, a multilingual large language model with 30 billion parameters, achieves superior performance in Chinese contexts after pre-training and fine-tuning compared to other open-source models.
large language models LLMs pre-trained multilingual corpus supervised fine-tuning reinforcement learning from human feedback MMLU CMMLU