title: Densing Law of LLMs
summary: Large Language Models (LLMs) have emerged as a milestone in artificial
  intelligence, and their performance can improve as the model size increases.
  However, this scaling brings great challenges to training and inference
  efficiency, particularly for deploying LLMs in resource-constrained
  environments, and the scaling trend is becoming increasingly unsustainable.
  This paper introduces the concept of ``capacity density'' as a new
  metric to evaluate the quality of the LLMs across different scales and
  describes the trend of LLMs in terms of both effectiveness and efficiency. To
  calculate the capacity density of a given target LLM, we first introduce a set
  of reference models and develop a scaling law to predict the downstream
  performance of these reference models based on their parameter sizes. We then
  define the effective parameter size of the target LLM as the parameter
  size required by a reference model to achieve equivalent performance, and
  formalize the capacity density as the ratio of the effective parameter size to
  the actual parameter size of the target LLM. Capacity density provides a
  unified framework for assessing both model effectiveness and efficiency. Our
  further analysis of recent open-source base LLMs reveals an empirical law (the
  densing law)that the capacity density of LLMs grows exponentially over time.
  More specifically, using some widely used benchmarks for evaluation, the
  capacity density of LLMs doubles approximately every three months. The law
  provides new perspectives to guide future LLM development, emphasizing the
  importance of improving capacity density to achieve optimal results with
  minimal computational overhead.
paper_authors._id: 675270943d5dc4e06f207c72
paper_authors.name: Chaojun Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:08:58.922Z
paper_authors._id: 608f6d72283d0a8d7be9d1f9
paper_authors.avatarUrl: /avatars/7f499a37019359a3c488ba6cc11751fc.svg
paper_authors.fullname: Chaojun XIAO
paper_authors.type: user
paper_authors.user: xcjthu
paper_authors._id: 675270943d5dc4e06f207c73
paper_authors.name: Jie Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:26:51.908Z
paper_authors._id: 642522c596f39ebc04185687
paper_authors.avatarUrl: /avatars/09a3ec01f6590ba385e15a14f4ac23eb.svg
paper_authors.fullname: Jie Cai
paper_authors.type: user
paper_authors.user: J-Sloane
paper_authors._id: 675270943d5dc4e06f207c74
paper_authors.name: Weilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:26:45.579Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 675270943d5dc4e06f207c75
paper_authors.name: Guoyang Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:26:37.344Z
paper_authors._id: 60cd679ef3c0385f86cb07c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624074137194-noauth.jpeg
paper_authors.fullname: Guoyang Zeng
paper_authors.type: user
paper_authors.user: losfen
paper_authors._id: 675270943d5dc4e06f207c76
paper_authors.name: Xu Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:26:30.328Z
paper_authors._id: 64397b4e3ab54fdbab803efe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lIZ3zeIzxfT2KUeAQuvjg.jpeg
paper_authors.fullname: xuhan
paper_authors.type: user
paper_authors.user: xuhan
paper_authors._id: 675270943d5dc4e06f207c77
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:26:23.337Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 675270943d5dc4e06f207c78
paper_authors.name: Maosong Sun
paper_title: Densing Law of LLMs
paper_summary: Large Language Models (LLMs) have emerged as a milestone in artificial
  intelligence, and their performance can improve as the model size increases.
  However, this scaling brings great challenges to training and inference
  efficiency, particularly for deploying LLMs in resource-constrained
  environments, and the scaling trend is becoming increasingly unsustainable.
  This paper introduces the concept of ``capacity density'' as a new
  metric to evaluate the quality of the LLMs across different scales and
  describes the trend of LLMs in terms of both effectiveness and efficiency. To
  calculate the capacity density of a given target LLM, we first introduce a set
  of reference models and develop a scaling law to predict the downstream
  performance of these reference models based on their parameter sizes. We then
  define the effective parameter size of the target LLM as the parameter
  size required by a reference model to achieve equivalent performance, and
  formalize the capacity density as the ratio of the effective parameter size to
  the actual parameter size of the target LLM. Capacity density provides a
  unified framework for assessing both model effectiveness and efficiency. Our
  further analysis of recent open-source base LLMs reveals an empirical law (the
  densing law)that the capacity density of LLMs grows exponentially over time.
  More specifically, using some widely used benchmarks for evaluation, the
  capacity density of LLMs doubles approximately every three months. The law
  provides new perspectives to guide future LLM development, emphasizing the
  importance of improving capacity density to achieve optimal results with
  minimal computational overhead.
paper_ai_keywords: Large Language Models (LLMs) capacity density scaling law effective parameter size parameter sizes downstream performance empirical law (the densing law) computational overhead