Aquila2 Technical Report
This paper introduces the Aquila2 series, which comprises a wide range of
bilingual models with parameter sizes of 7, 34, and 70 billion. These models
are trained based on an innovative framework named HeuriMentor (HM), which
offers real-time insights into model convergence and enhances the training
process and data management. The HM System, comprising the Adaptive Training
Engine (ATE), Training State Monitor (TSM), and Data Management Unit (DMU),
allows for precise monitoring of the model's training progress and enables
efficient optimization of data distribution, thereby enhancing training
effectiveness. Extensive evaluations show that the Aquila2 model series
performs comparably well on both English and Chinese benchmarks. Specifically,
Aquila2-34B demonstrates only a slight decrease in performance when quantized
to Int4. Furthermore, we have made our training code
(https://github.com/FlagOpen/FlagScale) and model weights
(https://github.com/FlagAI-Open/Aquila2) publicly available to support ongoing
research and the development of applications.
66bd86b6a474a917fcfed504
Bo-Wen Zhang
claimed_verified
2024-08-19T22:13:47.584Z
6335113375bed9932474315e
/avatars/aa1837a3a07514b89c48de888934a7b2.svg
bowenzhang
user
bowen92
66bd86b6a474a917fcfed505
Liangdong Wang
claimed_verified
2024-08-27T07:44:17.629Z
63a11ce02fabbbb899a01d58
/avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
ldwang
user
ldwang
66bd86b6a474a917fcfed506
Jijie Li
66bd86b6a474a917fcfed507
Shuhao Gu
claimed_verified
2024-10-30T09:56:05.363Z
642e72cec1b0f8e4e76af16d
/avatars/f900811d3c22a114c67283b646949f86.svg
shuhao gu
user
gsh33
66bd86b6a474a917fcfed508
Xinya Wu
66bd86b6a474a917fcfed509
Zhengduo Zhang
66bd86b6a474a917fcfed50a
Boyan Gao
66bd86b6a474a917fcfed50b
Yulong Ao
claimed_verified
2025-07-08T12:40:53.796Z
64f3eb4fd41e83d74a0173ec
/avatars/461f804bb5a7c5b31f862cfa27e52de1.svg
Yulong Ao
user
aoyulong
66bd86b6a474a917fcfed50c
Guang Liu
claimed_verified
2024-10-28T11:28:30.171Z
632c234f42c386ebd2710434
https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
Guang Liu
user
ZacLiu
Aquila2 Technical Report
This paper introduces the Aquila2 series, which comprises a wide range of
bilingual models with parameter sizes of 7, 34, and 70 billion. These models
are trained based on an innovative framework named HeuriMentor (HM), which
offers real-time insights into model convergence and enhances the training
process and data management. The HM System, comprising the Adaptive Training
Engine (ATE), Training State Monitor (TSM), and Data Management Unit (DMU),
allows for precise monitoring of the model's training progress and enables
efficient optimization of data distribution, thereby enhancing training
effectiveness. Extensive evaluations show that the Aquila2 model series
performs comparably well on both English and Chinese benchmarks. Specifically,
Aquila2-34B demonstrates only a slight decrease in performance when quantized
to Int4. Furthermore, we have made our training code
(https://github.com/FlagOpen/FlagScale) and model weights
(https://github.com/FlagAI-Open/Aquila2) publicly available to support ongoing
research and the development of applications.
The Aquila2 series of bilingual models, using the HeuriMentor framework, achieve competitive performance on English and Chinese benchmarks with efficient training and data management.
HeuriMentor HM System Adaptive Training Engine Training State Monitor Data Management Unit quantized Int4