Sign up to save your podcasts
Or
Share “Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer” by johnswentworth, David Lorell
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
“Why Would Belief-States Have A Fractal Structure, And Why Would That Matter For Interpretability? An Explainer” by johnswentworth, David Lorell
Yesterday Adam Shai put up a cool post which… well, take a look at the visual:
Yup, it sure looks like that fractal is very noisily embedded in the residual activations of a neural net trained on a toy problem. Linearly embedded, no less.
I (John) initially misunderstood what was going on in that post, but some back-and-forth with Adam convinced me that it really is as cool as that visual makes it look, and arguably even cooler. So David and I wrote up this post / some code, partly as an explainer for why on earth that fractal would show up, and partly as an explainer for the possibilities this work potentially opens up for interpretability.
One sentence summary: when tracking the hidden state of a hidden Markov model, a Bayesian's beliefs follow a chaos game (with the observations randomly selecting the update at each time), so [...]
---
Outline:
(01:11) Background: Fractals and Symmetry
(04:02) Background: Chaos Games
(05:58) Bayesian Belief States For A Hidden Markov Model
(09:35) Why Would That Show Up In A Neural Net?
(10:13) Why This Sort Of Thing Might Be A Pretty Big Deal
---
First published:
April 18th, 2024
Source:
https://www.lesswrong.com/posts/mBw7nc4ipdyeeEpWs/why-would-belief-states-have-a-fractal-structure-and-why
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrongYesterday Adam Shai put up a cool post which… well, take a look at the visual:
Yup, it sure looks like that fractal is very noisily embedded in the residual activations of a neural net trained on a toy problem. Linearly embedded, no less.
I (John) initially misunderstood what was going on in that post, but some back-and-forth with Adam convinced me that it really is as cool as that visual makes it look, and arguably even cooler. So David and I wrote up this post / some code, partly as an explainer for why on earth that fractal would show up, and partly as an explainer for the possibilities this work potentially opens up for interpretability.
One sentence summary: when tracking the hidden state of a hidden Markov model, a Bayesian's beliefs follow a chaos game (with the observations randomly selecting the update at each time), so [...]
---
Outline:
(01:11) Background: Fractals and Symmetry
(04:02) Background: Chaos Games
(05:58) Bayesian Belief States For A Hidden Markov Model
(09:35) Why Would That Show Up In A Neural Net?
(10:13) Why This Sort Of Thing Might Be A Pretty Big Deal
---
First published:
April 18th, 2024
Source:
https://www.lesswrong.com/posts/mBw7nc4ipdyeeEpWs/why-would-belief-states-have-a-fractal-structure-and-why
---
Narrated by TYPE III AUDIO.
More shows like LessWrong (30+ Karma)
View all
Making Sense with Sam Harris
26,446 Listeners
Conversations with Tyler
2,389 Listeners
The Peter Attia Drive
7,910 Listeners
Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas
4,136 Listeners
ManifoldOne
87 Listeners
Your Undivided Attention
1,462 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
9,095 Listeners
Machine Learning Street Talk (MLST)
87 Listeners
Dwarkesh Podcast
389 Listeners
Hard Fork
5,432 Listeners
The Ezra Klein Show
15,174 Listeners
Moonshots with Peter Diamandis
474 Listeners
No Priors: Artificial Intelligence | Technology | Startups
121 Listeners
Latent Space: The AI Engineer Podcast
75 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
459 Listeners