title: Towards Optimal Learning of Language Models
summary: This work studies the general principles of improving the learning of
  language models (LMs), which aims at reducing the necessary training steps for
  achieving superior performance. Specifically, we present a theory for the
  optimal learning of LMs. We first propose an objective that optimizes LM
  learning by maximizing the data compression ratio in an
  "LM-training-as-lossless-compression" view. Then, we derive a theorem, named
  Learning Law, to reveal the properties of the dynamics in the optimal learning
  process under our objective. The theorem is then validated by experiments on a
  linear classification and a real-world language modeling task. Finally, we
  empirically verify that the optimal learning of LMs essentially stems from the
  improvement of the coefficients in the scaling law of LMs, indicating great
  promise and significance for designing practical learning acceleration methods.
  Our code can be found at https://aka.ms/LearningLaw.
paper_authors._id: 65ded4d51bfb01764462efa0
paper_authors.name: Yuxian Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-28T11:02:17.669Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 65ded4d51bfb01764462efa1
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:19:12.705Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 65ded4d51bfb01764462efa2
paper_authors.name: Yaru Hao
paper_authors._id: 65ded4d51bfb01764462efa3
paper_authors.name: Qingxiu Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:03:10.020Z
paper_authors._id: 670740744341dcee459fb990
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/66UkZvrAk7fQr5YCylEFk.png
paper_authors.fullname: Qingxiu Dong
paper_authors.type: user
paper_authors.user: Rsy24
paper_authors._id: 65ded4d51bfb01764462efa4
paper_authors.name: Minlie Huang
paper_authors._id: 65ded4d51bfb01764462efa5
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:22:16.232Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Towards Optimal Learning of Language Models
paper_summary: This work studies the general principles of improving the learning of
  language models (LMs), which aims at reducing the necessary training steps for
  achieving superior performance. Specifically, we present a theory for the
  optimal learning of LMs. We first propose an objective that optimizes LM
  learning by maximizing the data compression ratio in an
  "LM-training-as-lossless-compression" view. Then, we derive a theorem, named
  Learning Law, to reveal the properties of the dynamics in the optimal learning
  process under our objective. The theorem is then validated by experiments on a
  linear classification and a real-world language modeling task. Finally, we
  empirically verify that the optimal learning of LMs essentially stems from the
  improvement of the coefficients in the scaling law of LMs, indicating great
  promise and significance for designing practical learning acceleration methods.
  Our code can be found at https://aka.ms/LearningLaw.
paper_ai_keywords: language models (LMs) data compression ratio LM-training-as-lossless-compression Learning Law scaling law