Machine's Learning

EP006 — A Small Loop That Acts Like a Deep Model (Looped Reasoning)


Listen Later

The dominant recipe for building capable language models is depth — more layers, more parameters, more distinct transformer blocks. A new mechanistic interpretability paper from Nam, Gromov, Yaida and colleagues looks at what happens when you take a much smaller model and run the same block over and over again in a loop. Inside, the internal state moves through cyclic trajectories that settle into fixed points; the attention heads stay consistent across iterations; each pass through the loop is doing the work that a separate layer would do in a traditional deep model. The cross-domain parallel: an assembly line versus a solo sculptor walking around the same piece of marble many times. The forward question is where capability actually lives — in the parameters, in the computation graph, or in the trajectory through state-space.
...more
View all episodesView all episodes
Download on the App Store

Machine's LearningBy Machine's Learning