Yi: Open Foundation Models by 01.AI
We introduce the Yi model family, a series of language and multimodal models
that demonstrate strong multi-dimensional capabilities. The Yi model family is
based on 6B and 34B pretrained language models, then we extend them to chat
models, 200K long context models, depth-upscaled models, and vision-language
models. Our base models achieve strong performance on a wide range of
benchmarks like MMLU, and our finetuned chat models deliver strong human
preference rate on major evaluation platforms like AlpacaEval and Chatbot
Arena. Building upon our scalable super-computing infrastructure and the
classical transformer architecture, we attribute the performance of Yi models
primarily to its data quality resulting from our data-engineering efforts. For
pretraining, we construct 3.1 trillion tokens of English and Chinese corpora
using a cascaded data deduplication and quality filtering pipeline. For
finetuning, we polish a small scale (less than 10K) instruction dataset over
multiple iterations such that every single instance has been verified directly
by our machine learning engineers. For vision-language, we combine the chat
language model with a vision transformer encoder and train the model to align
visual representations to the semantic space of the language model. We further
extend the context length to 200K through lightweight continual pretraining and
demonstrate strong needle-in-a-haystack retrieval performance. We show that
extending the depth of the pretrained checkpoint through continual pretraining
further improves performance. We believe that given our current results,
continuing to scale up model parameters using thoroughly optimized data will
lead to even stronger frontier models.
65ea7bec0d2a8c5d00438dfb
01. AI
65ea7bec0d2a8c5d00438dfd
Alex Young
65ea7bec0d2a8c5d00438dfe
Bei Chen
65ea7bec0d2a8c5d00438dff
Chao Li
65ea7bec0d2a8c5d00438e00
Chengen Huang
65ea7bec0d2a8c5d00438e01
Ge Zhang
claimed_verified
2024-03-08T08:32:50.223Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
65ea7bec0d2a8c5d00438e02
Guanwei Zhang
admin_assigned
2024-03-08T09:36:15.703Z
64a65ea92a782c381a81041d
/avatars/e566a76ee66c16389890bbc8bbcbffdb.svg
Guanwei Zhang
user
zgw95
65ea7bec0d2a8c5d00438e03
Heng Li
65ea7bec0d2a8c5d00438e04
Jiangcheng Zhu
65ea7bec0d2a8c5d00438e05
Jianqun Chen
admin_assigned
2024-03-08T09:36:06.829Z
64b8f57a4d3c95077089cd1a
/avatars/a966441fd777ed88fad8dd9ef3af8bd6.svg
Chen
user
Jianqun
65ea7bec0d2a8c5d00438e06
Jing Chang
65ea7bec0d2a8c5d00438e07
Kaidong Yu
admin_assigned
2024-03-08T09:36:51.331Z
64a52249978242bb77e03ce1
/avatars/2bddd991a64ea88173db533a4b955476.svg
Kaidong Yu
user
KaidongYu
65ea7bec0d2a8c5d00438e08
Peng Liu
65ea7bec0d2a8c5d00438e09
Qiang Liu
65ea7bec0d2a8c5d00438e0a
Shawn Yue
65ea7bec0d2a8c5d00438e0b
Senbin Yang
65ea7bec0d2a8c5d00438e0c
Shiming Yang
65ea7bec0d2a8c5d00438e0d
Tao Yu
65ea7bec0d2a8c5d00438e0e
Wen Xie
admin_assigned
2024-03-08T09:37:46.855Z
60edf189075d9bdb89803e7f
/avatars/dfb635faaefa87983603a28b7d26d6bd.svg
VVVVVince
user
wenxie
65ea7bec0d2a8c5d00438e0f
Wenhao Huang
claimed_verified
2024-10-13T20:31:14.460Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
65ea7bec0d2a8c5d00438e10
Xiaohui Hu
admin_assigned
2024-03-08T09:38:06.532Z
64096da29e9f790c905b75f0
/avatars/a5e0bca66c5e698830ad640fe77e2242.svg
huxiaohui
user
xiaohuihui
65ea7bec0d2a8c5d00438e11
Xiaoyi Ren
65ea7bec0d2a8c5d00438e12
Xinyao Niu
admin_assigned
2024-03-08T09:38:18.451Z
627f4c100d2c1c0ba3da8a72
/avatars/404869a87480ef526c1d5fccb583bfa8.svg
Xinyao Niu
user
sirius-ctrl
65ea7bec0d2a8c5d00438e13
Pengcheng Nie
65ea7bec0d2a8c5d00438e14
Yuchi Xu
65ea7bec0d2a8c5d00438e15
Yudong Liu
65ea7bec0d2a8c5d00438e16
Yue Wang
65ea7bec0d2a8c5d00438e17
Yuxuan Cai
65ea7bec0d2a8c5d00438e18
Zhenyu Gu
65ea7bec0d2a8c5d00438e19
Zhiyuan Liu
65ea7bec0d2a8c5d00438e1a
Zonghong Dai
Yi: Open Foundation Models by 01.AI
We introduce the Yi model family, a series of language and multimodal models
that demonstrate strong multi-dimensional capabilities. The Yi model family is
based on 6B and 34B pretrained language models, then we extend them to chat
models, 200K long context models, depth-upscaled models, and vision-language
models. Our base models achieve strong performance on a wide range of
benchmarks like MMLU, and our finetuned chat models deliver strong human
preference rate on major evaluation platforms like AlpacaEval and Chatbot
Arena. Building upon our scalable super-computing infrastructure and the
classical transformer architecture, we attribute the performance of Yi models
primarily to its data quality resulting from our data-engineering efforts. For
pretraining, we construct 3.1 trillion tokens of English and Chinese corpora
using a cascaded data deduplication and quality filtering pipeline. For
finetuning, we polish a small scale (less than 10K) instruction dataset over
multiple iterations such that every single instance has been verified directly
by our machine learning engineers. For vision-language, we combine the chat
language model with a vision transformer encoder and train the model to align
visual representations to the semantic space of the language model. We further
extend the context length to 200K through lightweight continual pretraining and
demonstrate strong needle-in-a-haystack retrieval performance. We show that
extending the depth of the pretrained checkpoint through continual pretraining
further improves performance. We believe that given our current results,
continuing to scale up model parameters using thoroughly optimized data will
lead to even stronger frontier models.
The Yi model family, based on transformer architecture, showcases strong performance across benchmarks and modalities through optimized data and scalable infrastructure.
language models multimodal models chat models 200K long context models depth-upscaled models vision-language models MMLU AlpacaEval Chatbot Arena transformer architecture data deduplication quality filtering vision transformer continual pretraining needle-in-a-haystack retrieval