title: Physics in Next-token Prediction
summary: We discovered the underlying physics in Next-token Prediction (NTP). We
  identified the law of information conservation within NTP and proposed the
  First Law of Information Capacity (IC-1), demonstrating that the essence of
  intelligence emergence in auto-regressive models is fundamentally a process of
  information transfer. We also introduced Landauer's Principle into NTP,
  formulating the Second Law of Information Capacity (IC-2), which establishes
  the relationship between auto-regressive model training and energy consumption.
  Additionally, we presented several corollaries, which hold practical
  significance for production practices. Finally, we validated the compatibility
  and complementarity of our findings with existing theories.
paper_authors._id: 6729042633e7529ed742d7b8
paper_authors.name: Hongjun An
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T17:41:45.805Z
paper_authors._id: 65bc71766db364adbb0ee817
paper_authors.avatarUrl: /avatars/4f889e82fed3aaff058bde7299b8d585.svg
paper_authors.fullname: Hongjun An
paper_authors.type: user
paper_authors.user: Coder-AN
paper_authors._id: 6729042633e7529ed742d7b9
paper_authors.name: Yiliang Song
paper_authors._id: 6729042633e7529ed742d7ba
paper_authors.name: Xuelong Li
paper_title: Physics in Next-token Prediction
paper_summary: We discovered the underlying physics in Next-token Prediction (NTP). We
  identified the law of information conservation within NTP and proposed the
  First Law of Information Capacity (IC-1), demonstrating that the essence of
  intelligence emergence in auto-regressive models is fundamentally a process of
  information transfer. We also introduced Landauer's Principle into NTP,
  formulating the Second Law of Information Capacity (IC-2), which establishes
  the relationship between auto-regressive model training and energy consumption.
  Additionally, we presented several corollaries, which hold practical
  significance for production practices. Finally, we validated the compatibility
  and complementarity of our findings with existing theories.
paper_ai_summary: The study identifies laws of information capacity in Next-token Prediction and applies Landauer's Principle to relate auto-regressive model training to energy consumption.
paper_ai_keywords: Next-token Prediction First Law of Information Capacity Second Law of Information Capacity auto-regressive models information transfer Landauer's Principle information capacity