title: Yi: Open Foundation Models by 01.AI
summary: We introduce the Yi model family, a series of language and multimodal models
  that demonstrate strong multi-dimensional capabilities. The Yi model family is
  based on 6B and 34B pretrained language models, then we extend them to chat
  models, 200K long context models, depth-upscaled models, and vision-language
  models. Our base models achieve strong performance on a wide range of
  benchmarks like MMLU, and our finetuned chat models deliver strong human
  preference rate on major evaluation platforms like AlpacaEval and Chatbot
  Arena. Building upon our scalable super-computing infrastructure and the
  classical transformer architecture, we attribute the performance of Yi models
  primarily to its data quality resulting from our data-engineering efforts. For
  pretraining, we construct 3.1 trillion tokens of English and Chinese corpora
  using a cascaded data deduplication and quality filtering pipeline. For
  finetuning, we polish a small scale (less than 10K) instruction dataset over
  multiple iterations such that every single instance has been verified directly
  by our machine learning engineers. For vision-language, we combine the chat
  language model with a vision transformer encoder and train the model to align
  visual representations to the semantic space of the language model. We further
  extend the context length to 200K through lightweight continual pretraining and
  demonstrate strong needle-in-a-haystack retrieval performance. We show that
  extending the depth of the pretrained checkpoint through continual pretraining
  further improves performance. We believe that given our current results,
  continuing to scale up model parameters using thoroughly optimized data will
  lead to even stronger frontier models.
paper_authors._id: 65ea7bec0d2a8c5d00438dfb
paper_authors.name: 01. AI
paper_authors._id: 65ea7bec0d2a8c5d00438dfd
paper_authors.name: Alex Young
paper_authors._id: 65ea7bec0d2a8c5d00438dfe
paper_authors.name: Bei Chen
paper_authors._id: 65ea7bec0d2a8c5d00438dff
paper_authors.name: Chao Li
paper_authors._id: 65ea7bec0d2a8c5d00438e00
paper_authors.name: Chengen Huang
paper_authors._id: 65ea7bec0d2a8c5d00438e01
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-08T08:32:50.223Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 65ea7bec0d2a8c5d00438e02
paper_authors.name: Guanwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:36:15.703Z
paper_authors._id: 64a65ea92a782c381a81041d
paper_authors.avatarUrl: /avatars/e566a76ee66c16389890bbc8bbcbffdb.svg
paper_authors.fullname: Guanwei Zhang
paper_authors.type: user
paper_authors.user: zgw95
paper_authors._id: 65ea7bec0d2a8c5d00438e03
paper_authors.name: Heng Li
paper_authors._id: 65ea7bec0d2a8c5d00438e04
paper_authors.name: Jiangcheng Zhu
paper_authors._id: 65ea7bec0d2a8c5d00438e05
paper_authors.name: Jianqun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:36:06.829Z
paper_authors._id: 64b8f57a4d3c95077089cd1a
paper_authors.avatarUrl: /avatars/a966441fd777ed88fad8dd9ef3af8bd6.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Jianqun
paper_authors._id: 65ea7bec0d2a8c5d00438e06
paper_authors.name: Jing Chang
paper_authors._id: 65ea7bec0d2a8c5d00438e07
paper_authors.name: Kaidong Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:36:51.331Z
paper_authors._id: 64a52249978242bb77e03ce1
paper_authors.avatarUrl: /avatars/2bddd991a64ea88173db533a4b955476.svg
paper_authors.fullname: Kaidong Yu
paper_authors.type: user
paper_authors.user: KaidongYu
paper_authors._id: 65ea7bec0d2a8c5d00438e08
paper_authors.name: Peng Liu
paper_authors._id: 65ea7bec0d2a8c5d00438e09
paper_authors.name: Qiang Liu
paper_authors._id: 65ea7bec0d2a8c5d00438e0a
paper_authors.name: Shawn Yue
paper_authors._id: 65ea7bec0d2a8c5d00438e0b
paper_authors.name: Senbin Yang
paper_authors._id: 65ea7bec0d2a8c5d00438e0c
paper_authors.name: Shiming Yang
paper_authors._id: 65ea7bec0d2a8c5d00438e0d
paper_authors.name: Tao Yu
paper_authors._id: 65ea7bec0d2a8c5d00438e0e
paper_authors.name: Wen Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:37:46.855Z
paper_authors._id: 60edf189075d9bdb89803e7f
paper_authors.avatarUrl: /avatars/dfb635faaefa87983603a28b7d26d6bd.svg
paper_authors.fullname: VVVVVince
paper_authors.type: user
paper_authors.user: wenxie
paper_authors._id: 65ea7bec0d2a8c5d00438e0f
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:31:14.460Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 65ea7bec0d2a8c5d00438e10
paper_authors.name: Xiaohui Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:38:06.532Z
paper_authors._id: 64096da29e9f790c905b75f0
paper_authors.avatarUrl: /avatars/a5e0bca66c5e698830ad640fe77e2242.svg
paper_authors.fullname: huxiaohui
paper_authors.type: user
paper_authors.user: xiaohuihui
paper_authors._id: 65ea7bec0d2a8c5d00438e11
paper_authors.name: Xiaoyi Ren
paper_authors._id: 65ea7bec0d2a8c5d00438e12
paper_authors.name: Xinyao Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:38:18.451Z
paper_authors._id: 627f4c100d2c1c0ba3da8a72
paper_authors.avatarUrl: /avatars/404869a87480ef526c1d5fccb583bfa8.svg
paper_authors.fullname: Xinyao Niu
paper_authors.type: user
paper_authors.user: sirius-ctrl
paper_authors._id: 65ea7bec0d2a8c5d00438e13
paper_authors.name: Pengcheng Nie
paper_authors._id: 65ea7bec0d2a8c5d00438e14
paper_authors.name: Yuchi Xu
paper_authors._id: 65ea7bec0d2a8c5d00438e15
paper_authors.name: Yudong Liu
paper_authors._id: 65ea7bec0d2a8c5d00438e16
paper_authors.name: Yue Wang
paper_authors._id: 65ea7bec0d2a8c5d00438e17
paper_authors.name: Yuxuan Cai
paper_authors._id: 65ea7bec0d2a8c5d00438e18
paper_authors.name: Zhenyu Gu
paper_authors._id: 65ea7bec0d2a8c5d00438e19
paper_authors.name: Zhiyuan Liu
paper_authors._id: 65ea7bec0d2a8c5d00438e1a
paper_authors.name: Zonghong Dai
paper_title: Yi: Open Foundation Models by 01.AI
paper_summary: We introduce the Yi model family, a series of language and multimodal models
  that demonstrate strong multi-dimensional capabilities. The Yi model family is
  based on 6B and 34B pretrained language models, then we extend them to chat
  models, 200K long context models, depth-upscaled models, and vision-language
  models. Our base models achieve strong performance on a wide range of
  benchmarks like MMLU, and our finetuned chat models deliver strong human
  preference rate on major evaluation platforms like AlpacaEval and Chatbot
  Arena. Building upon our scalable super-computing infrastructure and the
  classical transformer architecture, we attribute the performance of Yi models
  primarily to its data quality resulting from our data-engineering efforts. For
  pretraining, we construct 3.1 trillion tokens of English and Chinese corpora
  using a cascaded data deduplication and quality filtering pipeline. For
  finetuning, we polish a small scale (less than 10K) instruction dataset over
  multiple iterations such that every single instance has been verified directly
  by our machine learning engineers. For vision-language, we combine the chat
  language model with a vision transformer encoder and train the model to align
  visual representations to the semantic space of the language model. We further
  extend the context length to 200K through lightweight continual pretraining and
  demonstrate strong needle-in-a-haystack retrieval performance. We show that
  extending the depth of the pretrained checkpoint through continual pretraining
  further improves performance. We believe that given our current results,
  continuing to scale up model parameters using thoroughly optimized data will
  lead to even stronger frontier models.
paper_ai_keywords: language and multimodal models multi-dimensional capabilities 6B and 34B pretrained language models chat models 200K long context models depth-upscaled models vision-language models MMLU AlpacaEval Chatbot Arena classical transformer architecture data deduplication quality filtering pipeline instruction dataset machine learning BERTs vision transformer encoder semantic space lightweight continual pretraining needle-in-a-haystack retrieval performance