Yesterday Adam Shai put up a cool post which… well, take a look at the visual:
Yup, it sure looks like that fractal is very noisily embedded in the residual activations of a neural net trained on a toy problem. Linearly embedded, no less.
I (John) initially misunderstood what was going on in that post, but some back-and-forth with Adam convinced me that it really is as cool as that visual makes it look, and arguably even cooler. So David and I wrote up this post / some code, partly as an explainer for why on earth that fractal would show up, and partly as an explainer for the possibilities this work potentially opens up for interpretability.
One sentence summary: when tracking the hidden state of a hidden Markov model, a Bayesian's beliefs follow a chaos game (with the observations randomly selecting the update at each time), so [...]
---
Outline:
(01:11) Background: Fractals and Symmetry
(04:02) Background: Chaos Games
(05:58) Bayesian Belief States For A Hidden Markov Model
(09:35) Why Would That Show Up In A Neural Net?
(10:13) Why This Sort Of Thing Might Be A Pretty Big Deal
---