Densing Law of LLMs
Large Language Models (LLMs) have emerged as a milestone in artificial
intelligence, and their performance can improve as the model size increases.
However, this scaling brings great challenges to training and inference
efficiency, particularly for deploying LLMs in resource-constrained
environments, and the scaling trend is becoming increasingly unsustainable.
This paper introduces the concept of ``capacity density'' as a new
metric to evaluate the quality of the LLMs across different scales and
describes the trend of LLMs in terms of both effectiveness and efficiency. To
calculate the capacity density of a given target LLM, we first introduce a set
of reference models and develop a scaling law to predict the downstream
performance of these reference models based on their parameter sizes. We then
define the effective parameter size of the target LLM as the parameter
size required by a reference model to achieve equivalent performance, and
formalize the capacity density as the ratio of the effective parameter size to
the actual parameter size of the target LLM. Capacity density provides a
unified framework for assessing both model effectiveness and efficiency. Our
further analysis of recent open-source base LLMs reveals an empirical law (the
densing law)that the capacity density of LLMs grows exponentially over time.
More specifically, using some widely used benchmarks for evaluation, the
capacity density of LLMs doubles approximately every three months. The law
provides new perspectives to guide future LLM development, emphasizing the
importance of improving capacity density to achieve optimal results with
minimal computational overhead.
675270943d5dc4e06f207c72
Chaojun Xiao
claimed_verified
2024-12-06T09:08:58.922Z
608f6d72283d0a8d7be9d1f9
/avatars/7f499a37019359a3c488ba6cc11751fc.svg
Chaojun XIAO
user
xcjthu
675270943d5dc4e06f207c73
Jie Cai
admin_assigned
2024-12-06T17:26:51.908Z
642522c596f39ebc04185687
/avatars/09a3ec01f6590ba385e15a14f4ac23eb.svg
Jie Cai
user
J-Sloane
675270943d5dc4e06f207c74
Weilin Zhao
admin_assigned
2024-12-06T17:26:45.579Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
675270943d5dc4e06f207c75
Guoyang Zeng
admin_assigned
2024-12-06T17:26:37.344Z
60cd679ef3c0385f86cb07c2
https://cdn-avatars.huggingface.co/v1/production/uploads/1624074137194-noauth.jpeg
Guoyang Zeng
user
losfen
675270943d5dc4e06f207c76
Xu Han
admin_assigned
2024-12-06T17:26:30.328Z
64397b4e3ab54fdbab803efe
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lIZ3zeIzxfT2KUeAQuvjg.jpeg
xuhan
user
xuhan
675270943d5dc4e06f207c77
Zhiyuan Liu
admin_assigned
2024-12-06T17:26:23.337Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
675270943d5dc4e06f207c78
Maosong Sun
Densing Law of LLMs
Large Language Models (LLMs) have emerged as a milestone in artificial
intelligence, and their performance can improve as the model size increases.
However, this scaling brings great challenges to training and inference
efficiency, particularly for deploying LLMs in resource-constrained
environments, and the scaling trend is becoming increasingly unsustainable.
This paper introduces the concept of ``capacity density'' as a new
metric to evaluate the quality of the LLMs across different scales and
describes the trend of LLMs in terms of both effectiveness and efficiency. To
calculate the capacity density of a given target LLM, we first introduce a set
of reference models and develop a scaling law to predict the downstream
performance of these reference models based on their parameter sizes. We then
define the effective parameter size of the target LLM as the parameter
size required by a reference model to achieve equivalent performance, and
formalize the capacity density as the ratio of the effective parameter size to
the actual parameter size of the target LLM. Capacity density provides a
unified framework for assessing both model effectiveness and efficiency. Our
further analysis of recent open-source base LLMs reveals an empirical law (the
densing law)that the capacity density of LLMs grows exponentially over time.
More specifically, using some widely used benchmarks for evaluation, the
capacity density of LLMs doubles approximately every three months. The law
provides new perspectives to guide future LLM development, emphasizing the
importance of improving capacity density to achieve optimal results with
minimal computational overhead.
The paper introduces 'capacity density' as a metric to evaluate the efficiency and effectiveness of Large Language Models and reveals an empirical law showing their capacity density doubles every three months.
Large Language Models capacity density effective parameter size scaling law densing law