This episode explores Paul Werbos’s 1990 paper on Backpropagation Through Time and explains how ordinary backpropagation extends to systems whose state evolves over time. It walks through the core idea of unrolling a recurrent or dynamic system into a time-indexed computation graph, then applying reverse-mode differentiation to compute exact gradients across both layers and time steps. The discussion also places BPTT in historical context, connecting it to earlier work on backpropagation, automatic differentiation, and alternative recurrent learning methods like real-time recurrent learning. Listeners would find it interesting because it shows how a foundational training method for sequence models, control systems, and differentiable simulations emerged from a simple but powerful reframing of memory and time in neural computation.

Sources:

1. Backpropagation Through Time Explained

https://podcast.do-not-panic.com/uploaded-pdfs/2026-05-08T22-17-15-230Z-Backpropagation-through-time-what-it-does-and-how-to-do-it.pdf

2. Learning representations by back-propagating errors — David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986

https://scholar.google.com/scholar?q=Learning+representations+by+back-propagating+errors

3. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks — Ronald J. Williams, David Zipser, 1989

https://scholar.google.com/scholar?q=A+Learning+Algorithm+for+Continually+Running+Fully+Recurrent+Neural+Networks

4. Backpropagation Through Time: What It Does and How to Do It — Paul J. Werbos, 1990

https://scholar.google.com/scholar?q=Backpropagation+Through+Time%3A+What+It+Does+and+How+to+Do+It

5. Learning long-term dependencies with gradient descent is difficult — Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994

https://scholar.google.com/scholar?q=Learning+long-term+dependencies+with+gradient+descent+is+difficult

6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997

https://scholar.google.com/scholar?q=Long+Short-Term+Memory

7. Taylor expansion of the accumulated rounding error — Seppo Linnainmaa, 1976

https://scholar.google.com/scholar?q=Taylor+expansion+of+the+accumulated+rounding+error

8. Fast Exact Multiplication by the Hessian — Barak A. Pearlmutter, 1994

https://scholar.google.com/scholar?q=Fast+Exact+Multiplication+by+the+Hessian

9. Automatic Differentiation in Machine Learning: a Survey — Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind, 2018

https://scholar.google.com/scholar?q=Automatic+Differentiation+in+Machine+Learning%3A+a+Survey

10. A review of automatic differentiation and its efficient implementation — Charles C. Margossian, 2019

https://scholar.google.com/scholar?q=A+review+of+automatic+differentiation+and+its+efficient+implementation

11. Generalization of Back-Propagation to Recurrent Neural Networks — Fernando J. Pineda, 1987

https://scholar.google.com/scholar?q=Generalization+of+Back-Propagation+to+Recurrent+Neural+Networks

12. Finding Structure in Time — Jeffrey L. Elman, 1990

https://scholar.google.com/scholar?q=Finding+Structure+in+Time

13. BP(lambda): Online Learning via Synthetic Gradients — approx. anonymous from snippet / modern deep learning authors, recent

https://scholar.google.com/scholar?q=BP%28lambda%29%3A+Online+Learning+via+Synthetic+Gradients

14. Streaming Propagation Through Time: A New Computational Paradigm for Recurrent Neural Networks — approx. modern recurrent-learning authors, recent

https://scholar.google.com/scholar?q=Streaming+Propagation+Through+Time%3A+A+New+Computational+Paradigm+for+Recurrent+Neural+Networks

15. Combining Truncated BPTT and Truncated RTRL for LSTM Training — Jakob Stefan Weber, recent

https://scholar.google.com/scholar?q=Combining+Truncated+BPTT+and+Truncated+RTRL+for+LSTM+Training

16. Second-order forward-mode optimization of recurrent neural networks for neuroscience — approx. modern neuroscience/optimization authors, recent

https://scholar.google.com/scholar?q=Second-order+forward-mode+optimization+of+recurrent+neural+networks+for+neuroscience

17. Sample-Based Hybrid Mode Control: Asymptotically Optimal Switching of Algorithmic and Non-Differentiable Control Modes — approx. modern control authors, recent

https://scholar.google.com/scholar?q=Sample-Based+Hybrid+Mode+Control%3A+Asymptotically+Optimal+Switching+of+Algorithmic+and+Non-Differentiable+Control+Modes

18. On the differentiability of the value function of switched linear systems under arbitrary and controlled switching — approx. control theory authors, recent

https://scholar.google.com/scholar?q=On+the+differentiability+of+the+value+function+of+switched+linear+systems+under+arbitrary+and+controlled+switching

19. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp3

20. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3

Interactive Visualization: Backpropagation Through Time Explained