Sign up to save your podcasts
Or
Share “Implications Of Predicting The Next Token” by jdp
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Implications Of Predicting The Next Token” by jdp
I find that a lot of people have trouble with this concept of predicting the next
token. And by trouble, I mean that they struggle to understand what it actually
means to predict the next token. It seems simpler than it is. Because when you
say "predict the next token," I think what most people think of is the Markov
chain intuition that you have a big table of statistics, and then you look at
what word is the next most likely, and then you pick that as the word.
The thing about this is that if you have ever used a Markov chain, you would know
that Markov chain text is complete gibberish. Markov chain text does not resemble
meaningful writing. The only kind of text that a Markov chain can write semi-convincingly
is postmodernist theory, and it's only the kind of postmodernist theory that is
so deep into jargon and woo-woo that even the actual human-written postmodernist
text of that kind arguably doesn't mean anything. Markov chains are not capable
of producing any kind of meaningful text. Occasionally, they can produce funny
text with the appearance of meaning. For example, someone at one point made a [...]
---
Outline:
(20:30) Next Token Prediction And g
(30:40) Self Modeling Through Outrospection
---
First published:
May 19th, 2026
Source:
https://www.lesswrong.com/posts/AzRRPDNmeEoJdSiib/implications-of-predicting-the-next-token
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrongI find that a lot of people have trouble with this concept of predicting the next
token. And by trouble, I mean that they struggle to understand what it actually
means to predict the next token. It seems simpler than it is. Because when you
say "predict the next token," I think what most people think of is the Markov
chain intuition that you have a big table of statistics, and then you look at
what word is the next most likely, and then you pick that as the word.
The thing about this is that if you have ever used a Markov chain, you would know
that Markov chain text is complete gibberish. Markov chain text does not resemble
meaningful writing. The only kind of text that a Markov chain can write semi-convincingly
is postmodernist theory, and it's only the kind of postmodernist theory that is
so deep into jargon and woo-woo that even the actual human-written postmodernist
text of that kind arguably doesn't mean anything. Markov chains are not capable
of producing any kind of meaningful text. Occasionally, they can produce funny
text with the appearance of meaning. For example, someone at one point made a [...]
---
Outline:
(20:30) Next Token Prediction And g
(30:40) Self Modeling Through Outrospection
---
First published:
May 19th, 2026
Source:
https://www.lesswrong.com/posts/AzRRPDNmeEoJdSiib/implications-of-predicting-the-next-token
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (30+ Karma)
View all
The Daily
112,330 Listeners
Astral Codex Ten Podcast
130 Listeners
Interesting Times with Ross Douthat
7,247 Listeners
Dwarkesh Podcast
563 Listeners
The Ezra Klein Show
16,328 Listeners
AI Article Readings
4 Listeners
Doom Debates!
14 Listeners
LessWrong posts by zvi
2 Listeners