Episode title: After Titans: Behrouz on Nested Learning and Hope

The followup to our Titans episode, by the same core team a year

later. Behrouz, Razaviyayn, Zhong, and Mirrokni (Google Research)

generalize the Titans bet — that long-term memory should be a

learnable module updated at test time — into a broader paradigm

they call Nested Learning, where a "deep" architecture is really

a hierarchy of nested optimization problems each compressing its

own context flow. The episode walks through their three core

contributions: (1) reframing standard optimizers like Adam and

SGD-with-Momentum as associative-memory modules that compress

gradient information, then proposing more expressive optimizers

with their own deep memory; (2) a self-modifying sequence model

whose update rule is itself learned end-to-end — the natural

generalization of the test-time-learnable memory module Titans

introduced; (3) a continuum memory system that replaces the

traditional short-term-vs-long-term dichotomy with a continuum

across multiple update rates. Combining the self-modifying

module with the continuum memory system produces Hope, a

continual-learning architecture reported promising on language

modeling, knowledge incorporation, few-shot generalization,

continual learning, and long-context reasoning. The hosts treat

Hope as the next concrete instance of the Titans → Nested

Learning research arc, stress-test the novelty against

established meta-learning and fast-weights literature, and

distinguish what the paper actually shows from what its framing

suggests.

Sources:

Nested Learning: The Illusion of Deep Learning Architectures —

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni

(Google Research, NeurIPS 2025) — arXiv 2512.24695

https://arxiv.org/pdf/2512.24695

Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin

Zhong, Vahab Mirrokni (Jan 2025) — arXiv 2501.00663

https://arxiv.org/pdf/2501.00663

AI Post Transformers — "Titans: Learning to Memorize at Test Time"

https://podcast.do-not-panic.com/episodes/2026-05-20-titans-learning-to-memorize-at-test-time-054662.mp3

Interactive Visualization: After Titans: Behrouz on Nested Learning and Hope