title: L^2M: Mutual Information Scaling Law for Long-Context Language
    Modeling
summary: We rigorously establish a bipartite mutual information scaling law in natural
  language that governs long-range dependencies. This scaling law, which we show
  is distinct from and scales independently of the conventional two-point mutual
  information, is the key to understanding long-context language modeling. Using
  this scaling law, we formulate the Long-context Language Modeling (L^2M)
  condition, which relates a model's capacity for effective long context length
  modeling to the scaling of its latent state size for storing past information.
  Our results are validated through experiments on both transformers and state
  space models. This work establishes a theoretical foundation that guides the
  development of large language models toward longer context lengths.
paper_authors._id: 67ca9092ba1ee2b914e3fa4a
paper_authors.name: Zhuo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T09:09:31.232Z
paper_authors._id: 65e0027c960938e63e4a0157
paper_authors.avatarUrl: /avatars/c8ca0b082ee8e8004f47a23d9393df67.svg
paper_authors.fullname: Zhuo Chen
paper_authors.type: user
paper_authors.user: zhuoc3
paper_authors._id: 67ca9092ba1ee2b914e3fa4b
paper_authors.name: Oriol Mayné i Comas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:37:41.309Z
paper_authors._id: 66e0619714d7a7711c6fc139
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66e0619714d7a7711c6fc139/IZxEQ6Iv62DDHpCxFM-QG.jpeg
paper_authors.fullname: Oriol Mayné i Comas
paper_authors.type: user
paper_authors.user: oriolmayne
paper_authors._id: 67ca9092ba1ee2b914e3fa4c
paper_authors.name: Zhuotao Jin
paper_authors._id: 67ca9092ba1ee2b914e3fa4d
paper_authors.name: Di Luo
paper_authors._id: 67ca9092ba1ee2b914e3fa4e
paper_authors.name: Marin Soljačić
paper_title: L^2M: Mutual Information Scaling Law for Long-Context Language
    Modeling
paper_summary: We rigorously establish a bipartite mutual information scaling law in natural
  language that governs long-range dependencies. This scaling law, which we show
  is distinct from and scales independently of the conventional two-point mutual
  information, is the key to understanding long-context language modeling. Using
  this scaling law, we formulate the Long-context Language Modeling (L^2M)
  condition, which relates a model's capacity for effective long context length
  modeling to the scaling of its latent state size for storing past information.
  Our results are validated through experiments on both transformers and state
  space models. This work establishes a theoretical foundation that guides the
  development of large language models toward longer context lengths.
paper_ai_keywords: bipartite mutual information long-range dependencies two-point mutual information Long-context Language Modeling (L$^2$M) effective long context length modeling latent state size transformers state space models theoretical foundation