title: OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model
    Pre-trained from Scratch
summary: Large language models (LLMs) with billions of parameters have demonstrated
  outstanding performance on various natural language processing tasks. This
  report presents OpenBA, an open-sourced 15B bilingual asymmetric seq2seq model,
  to contribute an LLM variant to the Chinese-oriented open-source model
  community. We enhance OpenBA with effective and efficient techniques as well as
  adopt a three-stage training strategy to train the model from scratch. Our
  solution can also achieve very competitive performance with only 380B tokens,
  which is better than LLaMA-70B on the BELEBELE benchmark, BLOOM-176B on the
  MMLU benchmark, GLM-130B on the C-Eval (hard) benchmark. This report provides
  the main details to pre-train an analogous model, including pre-training data
  processing, Bilingual Flan data collection, the empirical observations that
  inspire our model architecture design, training objectives of different stages,
  and other enhancement techniques. We have refactored our code to follow the
  design principles of the Huggingface Transformers Library, making it more
  convenient for developers to use, and released checkpoints of different
  training stages at https://huggingface.co/openBA. More details of our project
  are available at https://github.com/OpenNLG/openBA.git.
paper_authors._id: 650a512069739cd31040d902
paper_authors.name: Juntao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:33:18.279Z
paper_authors._id: 60d1d7f94672e8c13e7c2bce
paper_authors.avatarUrl: /avatars/be3a3990492ef1c5f3ff33cc043ae166.svg
paper_authors.fullname: Juntao Li
paper_authors.type: user
paper_authors.user: ljtsuda
paper_authors._id: 650a512069739cd31040d903
paper_authors.name: Zecheng Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:33:25.380Z
paper_authors._id: 64096ef79e9f790c905b846d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
paper_authors.fullname: Zecheng Tang
paper_authors.type: user
paper_authors.user: ZetangForward
paper_authors._id: 650a512069739cd31040d904
paper_authors.name: Yuyang Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-22T19:34:48.556Z
paper_authors._id: 626cf0f65651e31a7a2b9779
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626cf0f65651e31a7a2b9779/TAE--QIKo1vz4Rb8aearl.jpeg
paper_authors.fullname: Ding
paper_authors.type: user
paper_authors.user: dyyyyyyyy
paper_authors._id: 650a512069739cd31040d905
paper_authors.name: Pinzheng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:33:41.635Z
paper_authors._id: 64f91b9d0d678a97a87e9403
paper_authors.avatarUrl: /avatars/ff090a63f7488986641399df89f28884.svg
paper_authors.fullname: Pinzheng Wang
paper_authors.type: user
paper_authors.user: PinzhengWang
paper_authors._id: 650a512069739cd31040d906
paper_authors.name: Pei Guo
paper_authors._id: 650a512069739cd31040d907
paper_authors.name: Wangjie You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:34:35.513Z
paper_authors._id: 64e5e3b0b687eece0edd5d6b
paper_authors.avatarUrl: /avatars/48129ccecee0211fc1f78bb48740aa6c.svg
paper_authors.fullname: youwangjie
paper_authors.type: user
paper_authors.user: Moriarty0923
paper_authors._id: 650a512069739cd31040d908
paper_authors.name: Dan Qiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:34:44.785Z
paper_authors._id: 64f7dae2baa3b4ec4e3a1eab
paper_authors.avatarUrl: /avatars/6815e94648f1243a637a16ae9ac9ad61.svg
paper_authors.fullname: Dan Qiao
paper_authors.type: user
paper_authors.user: jorjordan
paper_authors._id: 650a512069739cd31040d909
paper_authors.name: Wenliang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:34:58.974Z
paper_authors._id: 63477237511cd17d2c7a1be9
paper_authors.avatarUrl: /avatars/97cb3f640ded58344bca0ab279473134.svg
paper_authors.fullname: Wenliang Chen
paper_authors.type: user
paper_authors.user: jokephp
paper_authors._id: 650a512069739cd31040d90a
paper_authors.name: Guohong Fu
paper_authors._id: 650a512069739cd31040d90b
paper_authors.name: Qiaoming Zhu
paper_authors._id: 650a512069739cd31040d90c
paper_authors.name: Guodong Zhou
paper_authors._id: 650a512069739cd31040d90d
paper_authors.name: Min Zhang
paper_title: OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model
    Pre-trained from Scratch
paper_summary: Large language models (LLMs) with billions of parameters have demonstrated
  outstanding performance on various natural language processing tasks. This
  report presents OpenBA, an open-sourced 15B bilingual asymmetric seq2seq model,
  to contribute an LLM variant to the Chinese-oriented open-source model
  community. We enhance OpenBA with effective and efficient techniques as well as
  adopt a three-stage training strategy to train the model from scratch. Our
  solution can also achieve very competitive performance with only 380B tokens,
  which is better than LLaMA-70B on the BELEBELE benchmark, BLOOM-176B on the
  MMLU benchmark, GLM-130B on the C-Eval (hard) benchmark. This report provides
  the main details to pre-train an analogous model, including pre-training data
  processing, Bilingual Flan data collection, the empirical observations that
  inspire our model architecture design, training objectives of different stages,
  and other enhancement techniques. We have refactored our code to follow the
  design principles of the Huggingface Transformers Library, making it more
  convenient for developers to use, and released checkpoints of different
  training stages at https://huggingface.co/openBA. More details of our project
  are available at https://github.com/OpenNLG/openBA.git.
paper_ai_summary: OpenBA, a 15B bilingual asymmetric seq2seq model, achieves competitive performance on natural language processing tasks using an efficient three-stage training strategy and minimal pre-training data.
paper_ai_keywords: bilingual asymmetric seq2seq model OpenBA three-stage training strategy Bilingual Flan data collection empirical observations model architecture design training objectives pre-training data processing Huggingface Transformers Library