title: HARE: HumAn pRiors, a key to small language model Efficiency
summary: Human priors play a crucial role in efficiently utilizing data in deep
  learning. However, with the development of large language models (LLMs), there
  is an increasing emphasis on scaling both model size and data volume, which
  often diminishes the importance of human priors in data construction.
  Influenced by these trends, existing Small Language Models (SLMs) mainly rely
  on web-scraped large-scale training data, neglecting the proper incorporation
  of human priors. This oversight limits the training efficiency of language
  models in resource-constrained settings. In this paper, we propose a principle
  to leverage human priors for data construction. This principle emphasizes
  achieving high-performance SLMs by training on a concise dataset that
  accommodates both semantic diversity and data quality consistency, while
  avoiding benchmark data leakage. Following this principle, we train an SLM
  named HARE-1.1B. Extensive experiments on large-scale benchmark datasets
  demonstrate that HARE-1.1B performs favorably against state-of-the-art SLMs,
  validating the effectiveness of the proposed principle. Additionally, this
  provides new insights into efficient language model training in
  resource-constrained environments from the view of human priors.
paper_authors._id: 66717c101f4de3ba8b1dfbb1
paper_authors.name: Lingyun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:05.113Z
paper_authors._id: 64eef8cfa331d9798cb18015
paper_authors.avatarUrl: /avatars/4c1b1e0628d2d0f321ba5fe1a0ec8c03.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: lingyun1
paper_authors._id: 66717c101f4de3ba8b1dfbb2
paper_authors.name: Bin jin
paper_authors._id: 66717c101f4de3ba8b1dfbb3
paper_authors.name: Gaojian Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:22:57.410Z
paper_authors._id: 6662e38970c1a3233ca611d3
paper_authors.avatarUrl: /avatars/60a8a84199cb67b4276cd83bbe1991a6.svg
paper_authors.fullname: gegaojian
paper_authors.type: user
paper_authors.user: fo1ge
paper_authors._id: 66717c101f4de3ba8b1dfbb4
paper_authors.name: Lunhui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:23:04.909Z
paper_authors._id: 653218e5ef70d3555f4831f8
paper_authors.avatarUrl: /avatars/632f1cd0adededc1d495b9eb55d8adc2.svg
paper_authors.fullname: LunHui Liu
paper_authors.type: user
paper_authors.user: LLHLLH
paper_authors._id: 66717c101f4de3ba8b1dfbb5
paper_authors.name: Xuewen Shen
paper_authors._id: 66717c101f4de3ba8b1dfbb6
paper_authors.name: Mingyong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:23:24.373Z
paper_authors._id: 65eec2e0c9ef20fb409a15b9
paper_authors.avatarUrl: /avatars/a5a7c688de2f5f4fdef6b9ae91b98043.svg
paper_authors.fullname: wumingyong
paper_authors.type: user
paper_authors.user: wumingyong
paper_authors._id: 66717c101f4de3ba8b1dfbb7
paper_authors.name: Houqian Zhang
paper_authors._id: 66717c101f4de3ba8b1dfbb8
paper_authors.name: Yongneng Jiang
paper_authors._id: 66717c101f4de3ba8b1dfbb9
paper_authors.name: Shiqi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:23:48.808Z
paper_authors._id: 62bf99cae140faaa85335ab8
paper_authors.avatarUrl: /avatars/bcd2aa1823995b385096e2a68ce3e071.svg
paper_authors.fullname: Shiqi Chen
paper_authors.type: user
paper_authors.user: ShiqiChen
paper_authors._id: 66717c101f4de3ba8b1dfbba
paper_authors.name: Shi Pu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:24:19.644Z
paper_authors._id: 63968489fa66eacbf4a34812
paper_authors.avatarUrl: /avatars/8d3a9f053e99597c82d8d5ab491d913f.svg
paper_authors.fullname: Shi Pu
paper_authors.type: user
paper_authors.user: pushi
paper_title: HARE: HumAn pRiors, a key to small language model Efficiency
paper_summary: Human priors play a crucial role in efficiently utilizing data in deep
  learning. However, with the development of large language models (LLMs), there
  is an increasing emphasis on scaling both model size and data volume, which
  often diminishes the importance of human priors in data construction.
  Influenced by these trends, existing Small Language Models (SLMs) mainly rely
  on web-scraped large-scale training data, neglecting the proper incorporation
  of human priors. This oversight limits the training efficiency of language
  models in resource-constrained settings. In this paper, we propose a principle
  to leverage human priors for data construction. This principle emphasizes
  achieving high-performance SLMs by training on a concise dataset that
  accommodates both semantic diversity and data quality consistency, while
  avoiding benchmark data leakage. Following this principle, we train an SLM
  named HARE-1.1B. Extensive experiments on large-scale benchmark datasets
  demonstrate that HARE-1.1B performs favorably against state-of-the-art SLMs,
  validating the effectiveness of the proposed principle. Additionally, this
  provides new insights into efficient language model training in
  resource-constrained environments from the view of human priors.
paper_ai_summary: A principle for leveraging human priors in data construction is proposed to improve small language models, demonstrating favorable performance on large benchmarks in resource-constrained settings.
paper_ai_keywords: large language models small language models human priors data construction semantic diversity data quality consistency benchmark data leakage HARE-1.1B