title: Aquila2 Technical Report
summary: This paper introduces the Aquila2 series, which comprises a wide range of
  bilingual models with parameter sizes of 7, 34, and 70 billion. These models
  are trained based on an innovative framework named HeuriMentor (HM), which
  offers real-time insights into model convergence and enhances the training
  process and data management. The HM System, comprising the Adaptive Training
  Engine (ATE), Training State Monitor (TSM), and Data Management Unit (DMU),
  allows for precise monitoring of the model's training progress and enables
  efficient optimization of data distribution, thereby enhancing training
  effectiveness. Extensive evaluations show that the Aquila2 model series
  performs comparably well on both English and Chinese benchmarks. Specifically,
  Aquila2-34B demonstrates only a slight decrease in performance when quantized
  to Int4. Furthermore, we have made our training code
  (https://github.com/FlagOpen/FlagScale) and model weights
  (https://github.com/FlagAI-Open/Aquila2) publicly available to support ongoing
  research and the development of applications.
paper_authors._id: 66bd86b6a474a917fcfed504
paper_authors.name: Bo-Wen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:47.584Z
paper_authors._id: 6335113375bed9932474315e
paper_authors.avatarUrl: /avatars/aa1837a3a07514b89c48de888934a7b2.svg
paper_authors.fullname: bowenzhang
paper_authors.type: user
paper_authors.user: bowen92
paper_authors._id: 66bd86b6a474a917fcfed505
paper_authors.name: Liangdong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:44:17.629Z
paper_authors._id: 63a11ce02fabbbb899a01d58
paper_authors.avatarUrl: /avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
paper_authors.fullname: ldwang
paper_authors.type: user
paper_authors.user: ldwang
paper_authors._id: 66bd86b6a474a917fcfed506
paper_authors.name: Jijie Li
paper_authors._id: 66bd86b6a474a917fcfed507
paper_authors.name: Shuhao Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:56:05.363Z
paper_authors._id: 642e72cec1b0f8e4e76af16d
paper_authors.avatarUrl: /avatars/f900811d3c22a114c67283b646949f86.svg
paper_authors.fullname: shuhao gu
paper_authors.type: user
paper_authors.user: gsh33
paper_authors._id: 66bd86b6a474a917fcfed508
paper_authors.name: Xinya Wu
paper_authors._id: 66bd86b6a474a917fcfed509
paper_authors.name: Zhengduo Zhang
paper_authors._id: 66bd86b6a474a917fcfed50a
paper_authors.name: Boyan Gao
paper_authors._id: 66bd86b6a474a917fcfed50b
paper_authors.name: Yulong Ao
paper_authors._id: 66bd86b6a474a917fcfed50c
paper_authors.name: Guang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:28:30.171Z
paper_authors._id: 632c234f42c386ebd2710434
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
paper_authors.fullname: Guang Liu
paper_authors.type: user
paper_authors.user: ZacLiu
paper_title: Aquila2 Technical Report
paper_summary: This paper introduces the Aquila2 series, which comprises a wide range of
  bilingual models with parameter sizes of 7, 34, and 70 billion. These models
  are trained based on an innovative framework named HeuriMentor (HM), which
  offers real-time insights into model convergence and enhances the training
  process and data management. The HM System, comprising the Adaptive Training
  Engine (ATE), Training State Monitor (TSM), and Data Management Unit (DMU),
  allows for precise monitoring of the model's training progress and enables
  efficient optimization of data distribution, thereby enhancing training
  effectiveness. Extensive evaluations show that the Aquila2 model series
  performs comparably well on both English and Chinese benchmarks. Specifically,
  Aquila2-34B demonstrates only a slight decrease in performance when quantized
  to Int4. Furthermore, we have made our training code
  (https://github.com/FlagOpen/FlagScale) and model weights
  (https://github.com/FlagAI-Open/Aquila2) publicly available to support ongoing
  research and the development of applications.
paper_ai_summary: The Aquila2 series of bilingual models, using the HeuriMentor framework, achieve competitive performance on English and Chinese benchmarks with efficient training and data management.
paper_ai_keywords: HeuriMentor HM System Adaptive Training Engine Training State Monitor Data Management Unit quantized Int4