Sign up to save your podcasts
Or
Share Physics in Next-token Prediction
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Physics in Next-token Prediction
🤗 Paper Upvotes: 7 | cs.LG, cs.AI
Authors:
Hongjun An, Yiliang Song, Xuelong Li
Title:
Physics in Next-token Prediction
Arxiv:
http://arxiv.org/abs/2411.00660v1
Abstract:
We discovered the underlying physics in Next-token Prediction (NTP). We identified the law of information conservation within NTP and proposed the First Law of Information Capacity (IC-1), demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer. We also introduced Landauer's Principle into NTP, formulating the Second Law of Information Capacity (IC-2), which establishes the relationship between auto-regressive model training and energy consumption. Additionally, we presented several corollaries, which hold practical significance for production practices. Finally, we validated the compatibility and complementarity of our findings with existing theories.
View all episodes
By Jingwen Liang, Gengyu Wang🤗 Paper Upvotes: 7 | cs.LG, cs.AI
Authors:
Hongjun An, Yiliang Song, Xuelong Li
Title:
Physics in Next-token Prediction
Arxiv:
http://arxiv.org/abs/2411.00660v1
Abstract:
We discovered the underlying physics in Next-token Prediction (NTP). We identified the law of information conservation within NTP and proposed the First Law of Information Capacity (IC-1), demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer. We also introduced Landauer's Principle into NTP, formulating the Second Law of Information Capacity (IC-2), which establishes the relationship between auto-regressive model training and energy consumption. Additionally, we presented several corollaries, which hold practical significance for production practices. Finally, we validated the compatibility and complementarity of our findings with existing theories.