OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model
  Pre-trained from Scratch
Large language models (LLMs) with billions of parameters have demonstrated
outstanding performance on various natural language processing tasks. This
report presents OpenBA, an open-sourced 15B bilingual asymmetric seq2seq model,
to contribute an LLM variant to the Chinese-oriented open-source model
community. We enhance OpenBA with effective and efficient techniques as well as
adopt a three-stage training strategy to train the model from scratch. Our
solution can also achieve very competitive performance with only 380B tokens,
which is better than LLaMA-70B on the BELEBELE benchmark, BLOOM-176B on the
MMLU benchmark, GLM-130B on the C-Eval (hard) benchmark. This report provides
the main details to pre-train an analogous model, including pre-training data
processing, Bilingual Flan data collection, the empirical observations that
inspire our model architecture design, training objectives of different stages,
and other enhancement techniques. We have refactored our code to follow the
design principles of the Huggingface Transformers Library, making it more
convenient for developers to use, and released checkpoints of different
training stages at https://huggingface.co/openBA. More details of our project
are available at https://github.com/OpenNLG/openBA.git.
650a512069739cd31040d902
Juntao Li
admin_assigned
2023-09-20T11:33:18.279Z
60d1d7f94672e8c13e7c2bce
/avatars/be3a3990492ef1c5f3ff33cc043ae166.svg
Juntao Li
user
ljtsuda
650a512069739cd31040d903
Zecheng Tang
admin_assigned
2023-09-20T11:33:25.380Z
64096ef79e9f790c905b846d
https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
Zecheng Tang
user
ZetangForward
650a512069739cd31040d904
Yuyang Ding
claimed_verified
2023-10-22T19:34:48.556Z
626cf0f65651e31a7a2b9779
https://cdn-avatars.huggingface.co/v1/production/uploads/626cf0f65651e31a7a2b9779/TAE--QIKo1vz4Rb8aearl.jpeg
Ding
user
dyyyyyyyy
650a512069739cd31040d905
Pinzheng Wang
admin_assigned
2023-09-20T11:33:41.635Z
64f91b9d0d678a97a87e9403
/avatars/ff090a63f7488986641399df89f28884.svg
Pinzheng Wang
user
PinzhengWang
650a512069739cd31040d906
Pei Guo
650a512069739cd31040d907
Wangjie You
admin_assigned
2023-09-20T11:34:35.513Z
64e5e3b0b687eece0edd5d6b
/avatars/48129ccecee0211fc1f78bb48740aa6c.svg
youwangjie
user
Moriarty0923
650a512069739cd31040d908
Dan Qiao
admin_assigned
2023-09-20T11:34:44.785Z
64f7dae2baa3b4ec4e3a1eab
/avatars/6815e94648f1243a637a16ae9ac9ad61.svg
Dan Qiao
user
jorjordan
650a512069739cd31040d909
Wenliang Chen
admin_assigned
2023-09-20T11:34:58.974Z
63477237511cd17d2c7a1be9
/avatars/97cb3f640ded58344bca0ab279473134.svg
Wenliang Chen
user
jokephp
650a512069739cd31040d90a
Guohong Fu
650a512069739cd31040d90b
Qiaoming Zhu
650a512069739cd31040d90c
Guodong Zhou
650a512069739cd31040d90d
Min Zhang
OpenBA: An Open-sourced 15B Bilingual Asymmetric seq2seq Model
  Pre-trained from Scratch
Large language models (LLMs) with billions of parameters have demonstrated
outstanding performance on various natural language processing tasks. This
report presents OpenBA, an open-sourced 15B bilingual asymmetric seq2seq model,
to contribute an LLM variant to the Chinese-oriented open-source model
community. We enhance OpenBA with effective and efficient techniques as well as
adopt a three-stage training strategy to train the model from scratch. Our
solution can also achieve very competitive performance with only 380B tokens,
which is better than LLaMA-70B on the BELEBELE benchmark, BLOOM-176B on the
MMLU benchmark, GLM-130B on the C-Eval (hard) benchmark. This report provides
the main details to pre-train an analogous model, including pre-training data
processing, Bilingual Flan data collection, the empirical observations that
inspire our model architecture design, training objectives of different stages,
and other enhancement techniques. We have refactored our code to follow the
design principles of the Huggingface Transformers Library, making it more
convenient for developers to use, and released checkpoints of different
training stages at https://huggingface.co/openBA. More details of our project
are available at https://github.com/OpenNLG/openBA.git.
OpenBA, a 15B bilingual asymmetric seq2seq model, achieves competitive performance on natural language processing tasks using an efficient three-stage training strategy and minimal pre-training data.
bilingual asymmetric seq2seq model OpenBA three-stage training strategy Bilingual Flan data collection empirical observations model architecture design training objectives pre-training data processing Huggingface Transformers Library