This episode explores the paper Test-Time Training with KV Binding Is Secretly Linear Attention and asks whether KV-binding test-time training is really doing online memorization or instead behaving like learned linear attention with sequence-specific fast weights. It explains how this approach differs from a standard transformer KV cache, situates it within earlier test-time-training work on expressive hidden states, and connects it to the broader push for long-context models that avoid quadratic softmax attention costs. The discussion highlights several findings that weaken the retrieval-style memory story: converged models show a query-key mismatch, replacing queries with keys barely changes aggregate performance, stronger inner-loop optimization does not reliably help, and even switching from descent to ascent can still work. Listeners would find it interesting because the episode reframes a flashy mechanism in simpler algebraic terms, clarifies which equivalence claims are exact versus empirical, and shows how that shift could change how researchers think about memory and efficiency in next-generation sequence models.

Sources:

1. Test-Time Training with KV Binding Is Secretly Linear Attention — Junchen Liu, Sven Elflein, Or Litany, Zan Gojcic, Ruilong Li, 2026

http://arxiv.org/abs/2602.21204

2. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret, 2020

https://arxiv.org/abs/2006.16236

3. Linear Transformers Are Secretly Fast Weight Programmers — Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber, 2021

https://arxiv.org/abs/2102.11174

4. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Xinlei Chen, Tatsunori Hashimoto, Carlos Guestrin, et al., 2024

https://arxiv.org/abs/2407.04620

5. Test-Time Training with KV Binding Is Secretly Linear Attention — Junchen Liu, Sven Elflein, Or Litany, Zan Gojcic, Ruilong Li, 2026

https://arxiv.org/abs/2602.21204

6. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024

https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time

7. End-to-End Test-Time Training for Long Context — Arnuv Tandon et al., 2025

https://scholar.google.com/scholar?q=End-to-End+Test-Time+Training+for+Long+Context

8. Understanding Factual Recall in Transformers via Associative Memories — Eshaan Nichani, Jason D. Lee, Alberto Bietti, 2024

https://arxiv.org/abs/2412.06538

9. Quantifying Logical Consistency in Transformers via Query-Key Alignment — Eduard Tulchinskii, Anastasia Voznyuk, Laida Kushnareva, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, Serguei Barannikov, 2025

https://arxiv.org/abs/2502.17017

10. Dissecting Query-Key Interaction in Vision Transformers — Xu Pan, Aaron Philip, Ziqian Xie, Odelia Schwartz, 2024

https://arxiv.org/abs/2405.14880

11. Improved Test-Time Adaptation for Domain Generalization — Liang Chen, Yong Zhang, Yibing Song, Ying Shan, Lingqiao Liu, 2023

https://arxiv.org/abs/2304.04494

12. AdaShadow: Responsive Test-time Model Adaptation in Non-stationary Mobile Environments — Cheng Fang, Sicong Liu, Zimu Zhou, Bin Guo, Jiaqi Tang, Ke Ma, Zhiwen Yu, 2024

https://arxiv.org/abs/2410.08256

13. Beyond Model Adaptation at Test Time: A Survey — Zehao Xiao, Cees G. M. Snoek, 2024

https://arxiv.org/abs/2411.03687

14. AI Post Transformers: Atlas: Test-Time Memory for Long Contexts — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-atlas-test-time-memory-for-long-contexts-1d5545.mp3

15. AI Post Transformers: Parallelizing DeltaNet Linear Transformers over Sequence Length — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-18-parallelizing-deltanet-linear-transforme-2d0377.mp3

16. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp3

17. AI Post Transformers: δ-mem and Online Memory for LLMs — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-13-d-mem-and-online-memory-for-llms-6622fa.mp3

18. AI Post Transformers: Do Transformers Need Three Projections? — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-06-11-do-transformers-need-three-projections-c227d6.mp3