Towards Optimal Learning of Language Models
This work studies the general principles of improving the learning of
language models (LMs), which aims at reducing the necessary training steps for
achieving superior performance. Specifically, we present a theory for the
optimal learning of LMs. We first propose an objective that optimizes LM
learning by maximizing the data compression ratio in an
"LM-training-as-lossless-compression" view. Then, we derive a theorem, named
Learning Law, to reveal the properties of the dynamics in the optimal learning
process under our objective. The theorem is then validated by experiments on a
linear classification and a real-world language modeling task. Finally, we
empirically verify that the optimal learning of LMs essentially stems from the
improvement of the coefficients in the scaling law of LMs, indicating great
promise and significance for designing practical learning acceleration methods.
Our code can be found at https://aka.ms/LearningLaw.
65ded4d51bfb01764462efa0
Yuxian Gu
claimed_verified
2024-02-28T11:02:17.669Z
624ac662102fcdff87be51b9
https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
Yuxian Gu
user
t1101675
65ded4d51bfb01764462efa1
Li Dong
claimed_verified
2024-10-08T15:19:12.705Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
user
unilm
65ded4d51bfb01764462efa2
Yaru Hao
65ded4d51bfb01764462efa3
Qingxiu Dong
claimed_verified
2024-10-10T08:03:10.020Z
670740744341dcee459fb990
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
Qingxiu Dong
user
Rsy24
65ded4d51bfb01764462efa4
Minlie Huang
65ded4d51bfb01764462efa5
Furu Wei
admin_assigned
2024-02-28T12:22:16.232Z
6368c512fbfe97c16a40baba
/avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
Furu Wei
user
thegenerality
Towards Optimal Learning of Language Models
This work studies the general principles of improving the learning of
language models (LMs), which aims at reducing the necessary training steps for
achieving superior performance. Specifically, we present a theory for the
optimal learning of LMs. We first propose an objective that optimizes LM
learning by maximizing the data compression ratio in an
"LM-training-as-lossless-compression" view. Then, we derive a theorem, named
Learning Law, to reveal the properties of the dynamics in the optimal learning
process under our objective. The theorem is then validated by experiments on a
linear classification and a real-world language modeling task. Finally, we
empirically verify that the optimal learning of LMs essentially stems from the
improvement of the coefficients in the scaling law of LMs, indicating great
promise and significance for designing practical learning acceleration methods.
Our code can be found at https://aka.ms/LearningLaw.
A theory for optimal language model learning is proposed, revealing properties of the learning process through a data compression objective and improving coefficients in the scaling law, with experimental validation.
language models LM-training-as-lossless-compression Learning Law data compression scaling law learning acceleration methods