Physics in Next-token Prediction
We discovered the underlying physics in Next-token Prediction (NTP). We
identified the law of information conservation within NTP and proposed the
First Law of Information Capacity (IC-1), demonstrating that the essence of
intelligence emergence in auto-regressive models is fundamentally a process of
information transfer. We also introduced Landauer's Principle into NTP,
formulating the Second Law of Information Capacity (IC-2), which establishes
the relationship between auto-regressive model training and energy consumption.
Additionally, we presented several corollaries, which hold practical
significance for production practices. Finally, we validated the compatibility
and complementarity of our findings with existing theories.
6729042633e7529ed742d7b8
Hongjun An
claimed_verified
2024-11-04T17:41:45.805Z
65bc71766db364adbb0ee817
/avatars/4f889e82fed3aaff058bde7299b8d585.svg
Hongjun An
user
Coder-AN
6729042633e7529ed742d7b9
Yiliang Song
6729042633e7529ed742d7ba
Xuelong Li
Physics in Next-token Prediction
We discovered the underlying physics in Next-token Prediction (NTP). We
identified the law of information conservation within NTP and proposed the
First Law of Information Capacity (IC-1), demonstrating that the essence of
intelligence emergence in auto-regressive models is fundamentally a process of
information transfer. We also introduced Landauer's Principle into NTP,
formulating the Second Law of Information Capacity (IC-2), which establishes
the relationship between auto-regressive model training and energy consumption.
Additionally, we presented several corollaries, which hold practical
significance for production practices. Finally, we validated the compatibility
and complementarity of our findings with existing theories.
The study identifies laws of information capacity in Next-token Prediction and applies Landauer's Principle to relate auto-regressive model training to energy consumption.
Next-token Prediction First Law of Information Capacity Second Law of Information Capacity auto-regressive models information transfer Landauer's Principle information capacity