This episode explores the 1997 LSTM paper as a targeted fix for why standard recurrent neural networks failed on long-range dependencies: backpropagated error signals either vanish or explode when carried through many time steps. It explains how earlier training methods like BPTT and RTRL worked, why the underlying optimization problem was mathematically ill-conditioned, and how LSTM’s memory cells, Constant Error Carousel, and gating mechanisms were designed to preserve usable gradients over long lags. The discussion argues that the paper’s real contribution was not “solving memory” in a broad sense, but introducing a specific architectural mechanism validated mainly on synthetic delayed-response tasks. Listeners would find it interesting for its clear separation of the vanishing-gradient diagnosis from the LSTM remedy, and for its more nuanced view of a model often remembered only through later hype and tutorials.

Sources:

1. Long Short-Term Memory and Vanishing Gradients

https://www.bioinf.jku.at/publications/older/2604.pdf

2. Learning Long-Term Dependencies with Gradient Descent is Difficult — Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994

https://scholar.google.com/scholar?q=Learning+Long-Term+Dependencies+with+Gradient+Descent+is+Difficult

3. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions — Sepp Hochreiter, 1998

https://scholar.google.com/scholar?q=The+Vanishing+Gradient+Problem+During+Learning+Recurrent+Neural+Nets+and+Problem+Solutions

4. On the Difficulty of Training Recurrent Neural Networks — Razvan Pascanu, Tomas Mikolov, Yoshua Bengio, 2013

https://scholar.google.com/scholar?q=On+the+Difficulty+of+Training+Recurrent+Neural+Networks

5. Understanding the Difficulty of Training Deep Feedforward Neural Networks — Xavier Glorot, Yoshua Bengio, 2010

https://scholar.google.com/scholar?q=Understanding+the+Difficulty+of+Training+Deep+Feedforward+Neural+Networks

6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997

https://scholar.google.com/scholar?q=Long+Short-Term+Memory

7. Learning to Forget: Continual Prediction with LSTM — Felix A. Gers, Jürgen Schmidhuber, Fred Cummins, 2000

https://scholar.google.com/scholar?q=Learning+to+Forget%3A+Continual+Prediction+with+LSTM

8. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio, 2014

https://scholar.google.com/scholar?q=Empirical+Evaluation+of+Gated+Recurrent+Neural+Networks+on+Sequence+Modeling

9. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2017

https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks

10. Backpropagation Through Time: What It Does and How to Do It — Paul J. Werbos, 1990

https://scholar.google.com/scholar?q=Backpropagation+Through+Time%3A+What+It+Does+and+How+to+Do+It

11. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks — Ronald J. Williams, David Zipser, 1989

https://scholar.google.com/scholar?q=A+Learning+Algorithm+for+Continually+Running+Fully+Recurrent+Neural+Networks

12. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014

https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks

13. Untersuchungen zu dynamischen neuronalen Netzen — Sepp Hochreiter, 1991

https://scholar.google.com/scholar?q=Untersuchungen+zu+dynamischen+neuronalen+Netzen

14. The Problem of Learning Long-Term Dependencies in Recurrent Networks — Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994

https://scholar.google.com/scholar?q=The+Problem+of+Learning+Long-Term+Dependencies+in+Recurrent+Networks

15. Recurrent Networks and Neural Sequence Chunking — Jürgen Schmidhuber, 1992

https://scholar.google.com/scholar?q=Recurrent+Networks+and+Neural+Sequence+Chunking

16. The Continually Running Fully Recurrent Neural Network and Learning Long-Term Dependencies — Michael C. Mozer, 1992

https://scholar.google.com/scholar?q=The+Continually+Running+Fully+Recurrent+Neural+Network+and+Learning+Long-Term+Dependencies

17. Finding Structure in Time — Jeffrey L. Elman, 1990

https://scholar.google.com/scholar?q=Finding+Structure+in+Time

18. Learning Precise Timing with LSTM Recurrent Networks — Felix A. Gers, Jürgen Schmidhuber, Fred Cummins, 2000

https://scholar.google.com/scholar?q=Learning+Precise+Timing+with+LSTM+Recurrent+Networks

19. Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling — Hasim Sak, Andrew Senior, Françoise Beaufays, 2014

https://scholar.google.com/scholar?q=Long+Short-Term+Memory+Recurrent+Neural+Network+Architectures+for+Large+Scale+Acoustic+Modeling

20. Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training — unclear from snippet, recent

https://scholar.google.com/scholar?q=Exploiting+Symmetric+Temporally+Sparse+BPTT+for+Efficient+RNN+Training

21. Fast Training of Recurrent Neural Networks with Stationary State Feedbacks — unclear from snippet, recent

https://scholar.google.com/scholar?q=Fast+Training+of+Recurrent+Neural+Networks+with+Stationary+State+Feedbacks

22. Simple Linear Attention Language Models Balance the Recall-Throughput Tradeoff — unclear from snippet, recent

https://scholar.google.com/scholar?q=Simple+Linear+Attention+Language+Models+Balance+the+Recall-Throughput+Tradeoff

23. Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory — unclear from snippet, recent

https://scholar.google.com/scholar?q=Beyond+Attention%3A+Breaking+the+Limits+of+Transformer+Context+Length+with+Recurrent+Memory

24. A Systematic Analysis of Hybrid Linear Attention — unclear from snippet, recent

https://scholar.google.com/scholar?q=A+Systematic+Analysis+of+Hybrid+Linear+Attention

25. In-Context Learning as Conditioned Associative Memory Retrieval — unclear from snippet, recent

https://scholar.google.com/scholar?q=In-Context+Learning+as+Conditioned+Associative+Memory+Retrieval

26. Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting — unclear from snippet, recent

https://scholar.google.com/scholar?q=Dual+Process+Learning%3A+Controlling+Use+of+In-Context+vs.+In-Weights+Strategies+with+Weight+Forgetting

27. Memorization in In-Context Learning — unclear from snippet, recent

https://scholar.google.com/scholar?q=Memorization+in+In-Context+Learning

28. Pretraining Without Attention — approx. modern sequence-modeling authors; exact authors unclear from snippet, recent

https://scholar.google.com/scholar?q=Pretraining+Without+Attention

29. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp3

30. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, Sun,

https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/

Interactive Visualization: Long Short-Term Memory and Vanishing Gradients