L^2M: Mutual Information Scaling Law for Long-Context Language
  Modeling
We rigorously establish a bipartite mutual information scaling law in natural
language that governs long-range dependencies. This scaling law, which we show
is distinct from and scales independently of the conventional two-point mutual
information, is the key to understanding long-context language modeling. Using
this scaling law, we formulate the Long-context Language Modeling (L^2M)
condition, which relates a model's capacity for effective long context length
modeling to the scaling of its latent state size for storing past information.
Our results are validated through experiments on both transformers and state
space models. This work establishes a theoretical foundation that guides the
development of large language models toward longer context lengths.
67ca9092ba1ee2b914e3fa4a
Zhuo Chen
claimed_verified
2025-03-07T09:09:31.232Z
65e0027c960938e63e4a0157
/avatars/c8ca0b082ee8e8004f47a23d9393df67.svg
Zhuo Chen
user
zhuoc3
67ca9092ba1ee2b914e3fa4b
Oriol Mayné i Comas
admin_assigned
2025-03-07T09:37:41.309Z
66e0619714d7a7711c6fc139
https://cdn-avatars.huggingface.co/v1/production/uploads/66e0619714d7a7711c6fc139/IZxEQ6Iv62DDHpCxFM-QG.jpeg
Oriol Mayné i Comas
user
oriolmayne
67ca9092ba1ee2b914e3fa4c
Zhuotao Jin
67ca9092ba1ee2b914e3fa4d
Di Luo
67ca9092ba1ee2b914e3fa4e
Marin Soljačić
L^2M: Mutual Information Scaling Law for Long-Context Language
  Modeling
We rigorously establish a bipartite mutual information scaling law in natural
language that governs long-range dependencies. This scaling law, which we show
is distinct from and scales independently of the conventional two-point mutual
information, is the key to understanding long-context language modeling. Using
this scaling law, we formulate the Long-context Language Modeling (L^2M)
condition, which relates a model's capacity for effective long context length
modeling to the scaling of its latent state size for storing past information.
Our results are validated through experiments on both transformers and state
space models. This work establishes a theoretical foundation that guides the
development of large language models toward longer context lengths.
A scaling law for bipartite mutual information in natural language helps understand long-context language modeling and guides the development of large language models with longer context lengths.
bipartite mutual information long-range dependencies two-point mutual information long-context language modeling L^2M condition latent state size transformers state space models