Sign up to save your podcasts
Or
Share Directly Trained Spiking DQNs for Atari
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Directly Trained Spiking DQNs for Atari
This episode explores a 2023 paper on Deep Spiking Q-Networks, asking whether a directly trained spiking version of DQN can compete with earlier conversion-based spiking reinforcement learning methods on Atari while retaining the energy-efficiency promise of spiking neural networks. It explains the technical foundations behind spiking networks, including leaky integrate-and-fire neurons, surrogate-gradient training, and why SNNs remain difficult to train and awkward on conventional GPU hardware despite their appeal for neuromorphic chips like TrueNorth and Loihi. The discussion also situates the paper against the legacy of the original DeepMind DQN work, arguing that the paper’s title deliberately invites scrutiny over whether it truly matches the breadth and ambition of the classic Atari benchmark. Listeners would find it interesting for its clear framing of both the hype and the hard practical questions around neuromorphic AI: not just whether spiking RL works, but where, on what hardware, and under what conditions its efficiency claims actually matter.
Sources:
1. Human-Level Control through Directly-Trained Deep Spiking Q-Networks — Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, Huajin Tang, 2021
http://arxiv.org/abs/2201.07211
2. Spiking Neural Networks for Machine Learning: An Overview — Wolfgang Maass and others; overview literature includes major contributors such as Thomas Pfeil, Emre Neftci, and Surya Ganguli across the field, Recent overview genre, especially 2023
https://scholar.google.com/scholar?q=Spiking+Neural+Networks+for+Machine+Learning:+An+Overview
3. Training Spiking Neural Networks Using Lessons From Deep Learning — Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, Wolfgang Maass, 2018
https://scholar.google.com/scholar?q=Training+Spiking+Neural+Networks+Using+Lessons+From+Deep+Learning
4. Spiking Neural Networks in the Fourth Generation of Artificial Intelligence — Zhaofei Yu, Hanle Zheng, Yujie Wu, and others, 2023
https://scholar.google.com/scholar?q=Spiking+Neural+Networks+in+the+Fourth+Generation+of+Artificial+Intelligence
5. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks — Friedemann Zenke, Tim Vogels, 2021
https://scholar.google.com/scholar?q=The+Remarkable+Robustness+of+Surrogate+Gradient+Learning+for+Instilling+Complex+Function+in+Spiking+Neural+Networks
6. Human-level control through deep reinforcement learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis, 2015
https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning
7. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy Lillicrap, David Silver, Koray Kavukcuoglu, 2016
https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning
8. Deep Reinforcement Learning: An Overview — Yuxi Li, 2017
https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning:+An+Overview
9. Reinforcement Learning: An Introduction — Richard S. Sutton, Andrew G. Barto, 1998; 2nd edition 2018
https://scholar.google.com/scholar?q=Reinforcement+Learning:+An+Introduction
10. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks — Emre O. Neftci, Hesham Mostafa, Friedemann Zenke, 2019
https://scholar.google.com/scholar?q=Surrogate+Gradient+Learning+in+Spiking+Neural+Networks:+Bringing+the+Power+of+Gradient-Based+Optimization+to+Spiking+Neural+Networks
11. Direct Training for Spiking Neural Networks: Faster, Larger, Better — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2019
https://scholar.google.com/scholar?q=Direct+Training+for+Spiking+Neural+Networks:+Faster,+Larger,+Better
12. Going Deeper With Directly-Trained Larger Spiking Neural Networks — Chaoteng Duan, Shikuang Deng, Xingting Wang, Meng Zhang, and others, 2022
https://scholar.google.com/scholar?q=Going+Deeper+With+Directly-Trained+Larger+Spiking+Neural+Networks
13. Threshold-Dependent Batch Normalization for Training Deep Spiking Neural Networks — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2021
https://scholar.google.com/scholar?q=Threshold-Dependent+Batch+Normalization+for+Training+Deep+Spiking+Neural+Networks
14. A million spiking-neuron integrated circuit with a scalable communication network and interface — Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron Flickner, William Risk, Rajit Manohar, Dharmendra Modha, 2014
https://scholar.google.com/scholar?q=A+million+spiking-neuron+integrated+circuit+with+a+scalable+communication+network+and+interface
15. Loihi: A Neuromorphic Manycore Processor with On-Chip Learning — Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al., 2018
https://scholar.google.com/scholar?q=Loihi:+A+Neuromorphic+Manycore+Processor+with+On-Chip+Learning
16. SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation — Steve B. Furber, Francesco Galluppi, Steve Temple, Luis A. Plana, 2014
https://scholar.google.com/scholar?q=SpiNNaker:+A+1-W+18-Core+System-on-Chip+for+Massively-Parallel+Neural+Network+Simulation
17. Benchmarking Neuromorphic Systems with Nengo — Terry C. Stewart, Dan Rasmussen, Xuan Choo, Aaron Voelker, and others, 2015-2017 era benchmarking work
https://scholar.google.com/scholar?q=Benchmarking+Neuromorphic+Systems+with+Nengo
18. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013
https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning
19. Deep Reinforcement Learning with Double Q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2016
https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+with+Double+Q-learning
20. Rainbow: Combining Improvements in Deep Reinforcement Learning — Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver, 2018
https://scholar.google.com/scholar?q=Rainbow:+Combining+Improvements+in+Deep+Reinforcement+Learning
21. Enabling Deep Spiking Neural Networks for Reinforcement Learning — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2020
https://scholar.google.com/scholar?q=Enabling+Deep+Spiking+Neural+Networks+for+Reinforcement+Learning
22. Going Deeper in Spiking Neural Networks: VGG and Residual Architectures — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2021
https://scholar.google.com/scholar?q=Going+Deeper+in+Spiking+Neural+Networks:+VGG+and+Residual+Architectures
23. Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks — Yuhang Fang, Zhaofei Yu, Tielin Zhang, et al., 2021
https://scholar.google.com/scholar?q=Incorporating+Learnable+Membrane+Time+Constant+to+Enhance+Learning+of+Spiking+Neural+Networks
24. Deep Residual Learning in Spiking Neural Networks — Yujie Wu, Yuhang Zhao, et al., 2021
https://scholar.google.com/scholar?q=Deep+Residual+Learning+in+Spiking+Neural+Networks
25. A Unified Optimization Framework of ANN-SNN Conversion: Towards Optimal Mapping from Activation Values to Firing Rates — approx. recent ANN-to-SNN conversion literature, 2023-2024
https://scholar.google.com/scholar?q=A+Unified+Optimization+Framework+of+ANN-SNN+Conversion:+Towards+Optimal+Mapping+from+Activation+Values+to+Firing+Rates
26. Towards High-Performance Spiking Transformers from ANN to SNN Conversion — approx. recent conversion/transformer authors, 2024
https://scholar.google.com/scholar?q=Towards+High-Performance+Spiking+Transformers+from+ANN+to+SNN+Conversion
27. Towards Training-Free and Accurate ANN-to-SNN Conversion via Activation-Aware Redistribution — approx. recent ANN-to-SNN conversion authors, 2024
https://scholar.google.com/scholar?q=Towards+Training-Free+and+Accurate+ANN-to-SNN+Conversion+via+Activation-Aware+Redistribution
28. Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks — approx. recent SNN RL authors, 2024-2025
https://scholar.google.com/scholar?q=Adaptive+Surrogate+Gradients+for+Sequential+Reinforcement+Learning+in+Spiking+Neural+Networks
29. Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks — approx. recent theoretical SNN authors, 2023-2024
https://scholar.google.com/scholar?q=Elucidating+the+Theoretical+Underpinnings+of+Surrogate+Gradient+Learning+in+Spiking+Neural+Networks
30. Spiking Reinforcement Learning Enhanced by Bioinspired Event Source of Multi-Dendrite Spiking Neuron and Dynamic Thresholds — approx. recent spiking RL authors, 2024-2025
https://scholar.google.com/scholar?q=Spiking+Reinforcement+Learning+Enhanced+by+Bioinspired+Event+Source+of+Multi-Dendrite+Spiking+Neuron+and+Dynamic+Thresholds
31. S2Act: Simple Spiking Actor — approx. recent spiking actor-critic authors, 2024-2025
https://scholar.google.com/scholar?q=S2Act:+Simple+Spiking+Actor
32. AI Post Transformers: Zero-Shot Context Generalization in Reinforcement
Learning from Few Training Contexts — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/zero-shot-context-generalization-in-reinforcement-learning-from-few-training-con/
33. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3
Interactive Visualization: Directly Trained Spiking DQNs for Atari
View all episodes
By mcgrofThis episode explores a 2023 paper on Deep Spiking Q-Networks, asking whether a directly trained spiking version of DQN can compete with earlier conversion-based spiking reinforcement learning methods on Atari while retaining the energy-efficiency promise of spiking neural networks. It explains the technical foundations behind spiking networks, including leaky integrate-and-fire neurons, surrogate-gradient training, and why SNNs remain difficult to train and awkward on conventional GPU hardware despite their appeal for neuromorphic chips like TrueNorth and Loihi. The discussion also situates the paper against the legacy of the original DeepMind DQN work, arguing that the paper’s title deliberately invites scrutiny over whether it truly matches the breadth and ambition of the classic Atari benchmark. Listeners would find it interesting for its clear framing of both the hype and the hard practical questions around neuromorphic AI: not just whether spiking RL works, but where, on what hardware, and under what conditions its efficiency claims actually matter.
Sources:
1. Human-Level Control through Directly-Trained Deep Spiking Q-Networks — Guisong Liu, Wenjie Deng, Xiurui Xie, Li Huang, Huajin Tang, 2021
http://arxiv.org/abs/2201.07211
2. Spiking Neural Networks for Machine Learning: An Overview — Wolfgang Maass and others; overview literature includes major contributors such as Thomas Pfeil, Emre Neftci, and Surya Ganguli across the field, Recent overview genre, especially 2023
https://scholar.google.com/scholar?q=Spiking+Neural+Networks+for+Machine+Learning:+An+Overview
3. Training Spiking Neural Networks Using Lessons From Deep Learning — Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, Wolfgang Maass, 2018
https://scholar.google.com/scholar?q=Training+Spiking+Neural+Networks+Using+Lessons+From+Deep+Learning
4. Spiking Neural Networks in the Fourth Generation of Artificial Intelligence — Zhaofei Yu, Hanle Zheng, Yujie Wu, and others, 2023
https://scholar.google.com/scholar?q=Spiking+Neural+Networks+in+the+Fourth+Generation+of+Artificial+Intelligence
5. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks — Friedemann Zenke, Tim Vogels, 2021
https://scholar.google.com/scholar?q=The+Remarkable+Robustness+of+Surrogate+Gradient+Learning+for+Instilling+Complex+Function+in+Spiking+Neural+Networks
6. Human-level control through deep reinforcement learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, Demis Hassabis, 2015
https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning
7. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy Lillicrap, David Silver, Koray Kavukcuoglu, 2016
https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning
8. Deep Reinforcement Learning: An Overview — Yuxi Li, 2017
https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning:+An+Overview
9. Reinforcement Learning: An Introduction — Richard S. Sutton, Andrew G. Barto, 1998; 2nd edition 2018
https://scholar.google.com/scholar?q=Reinforcement+Learning:+An+Introduction
10. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks — Emre O. Neftci, Hesham Mostafa, Friedemann Zenke, 2019
https://scholar.google.com/scholar?q=Surrogate+Gradient+Learning+in+Spiking+Neural+Networks:+Bringing+the+Power+of+Gradient-Based+Optimization+to+Spiking+Neural+Networks
11. Direct Training for Spiking Neural Networks: Faster, Larger, Better — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2019
https://scholar.google.com/scholar?q=Direct+Training+for+Spiking+Neural+Networks:+Faster,+Larger,+Better
12. Going Deeper With Directly-Trained Larger Spiking Neural Networks — Chaoteng Duan, Shikuang Deng, Xingting Wang, Meng Zhang, and others, 2022
https://scholar.google.com/scholar?q=Going+Deeper+With+Directly-Trained+Larger+Spiking+Neural+Networks
13. Threshold-Dependent Batch Normalization for Training Deep Spiking Neural Networks — Yujie Wu, Lei Deng, Guoqi Li, Jun Zhu, Luping Shi, 2021
https://scholar.google.com/scholar?q=Threshold-Dependent+Batch+Normalization+for+Training+Deep+Spiking+Neural+Networks
14. A million spiking-neuron integrated circuit with a scalable communication network and interface — Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, Bernard Brezzo, Ivan Vo, Steven Esser, Rathinakumar Appuswamy, Brian Taba, Arnon Amir, Myron Flickner, William Risk, Rajit Manohar, Dharmendra Modha, 2014
https://scholar.google.com/scholar?q=A+million+spiking-neuron+integrated+circuit+with+a+scalable+communication+network+and+interface
15. Loihi: A Neuromorphic Manycore Processor with On-Chip Learning — Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al., 2018
https://scholar.google.com/scholar?q=Loihi:+A+Neuromorphic+Manycore+Processor+with+On-Chip+Learning
16. SpiNNaker: A 1-W 18-Core System-on-Chip for Massively-Parallel Neural Network Simulation — Steve B. Furber, Francesco Galluppi, Steve Temple, Luis A. Plana, 2014
https://scholar.google.com/scholar?q=SpiNNaker:+A+1-W+18-Core+System-on-Chip+for+Massively-Parallel+Neural+Network+Simulation
17. Benchmarking Neuromorphic Systems with Nengo — Terry C. Stewart, Dan Rasmussen, Xuan Choo, Aaron Voelker, and others, 2015-2017 era benchmarking work
https://scholar.google.com/scholar?q=Benchmarking+Neuromorphic+Systems+with+Nengo
18. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013
https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning
19. Deep Reinforcement Learning with Double Q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2016
https://scholar.google.com/scholar?q=Deep+Reinforcement+Learning+with+Double+Q-learning
20. Rainbow: Combining Improvements in Deep Reinforcement Learning — Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver, 2018
https://scholar.google.com/scholar?q=Rainbow:+Combining+Improvements+in+Deep+Reinforcement+Learning
21. Enabling Deep Spiking Neural Networks for Reinforcement Learning — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2020
https://scholar.google.com/scholar?q=Enabling+Deep+Spiking+Neural+Networks+for+Reinforcement+Learning
22. Going Deeper in Spiking Neural Networks: VGG and Residual Architectures — Nitin Rathi, Gopalakrishnan Srinivasan, Priyadarshini Panda, Kaushik Roy, 2021
https://scholar.google.com/scholar?q=Going+Deeper+in+Spiking+Neural+Networks:+VGG+and+Residual+Architectures
23. Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks — Yuhang Fang, Zhaofei Yu, Tielin Zhang, et al., 2021
https://scholar.google.com/scholar?q=Incorporating+Learnable+Membrane+Time+Constant+to+Enhance+Learning+of+Spiking+Neural+Networks
24. Deep Residual Learning in Spiking Neural Networks — Yujie Wu, Yuhang Zhao, et al., 2021
https://scholar.google.com/scholar?q=Deep+Residual+Learning+in+Spiking+Neural+Networks
25. A Unified Optimization Framework of ANN-SNN Conversion: Towards Optimal Mapping from Activation Values to Firing Rates — approx. recent ANN-to-SNN conversion literature, 2023-2024
https://scholar.google.com/scholar?q=A+Unified+Optimization+Framework+of+ANN-SNN+Conversion:+Towards+Optimal+Mapping+from+Activation+Values+to+Firing+Rates
26. Towards High-Performance Spiking Transformers from ANN to SNN Conversion — approx. recent conversion/transformer authors, 2024
https://scholar.google.com/scholar?q=Towards+High-Performance+Spiking+Transformers+from+ANN+to+SNN+Conversion
27. Towards Training-Free and Accurate ANN-to-SNN Conversion via Activation-Aware Redistribution — approx. recent ANN-to-SNN conversion authors, 2024
https://scholar.google.com/scholar?q=Towards+Training-Free+and+Accurate+ANN-to-SNN+Conversion+via+Activation-Aware+Redistribution
28. Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks — approx. recent SNN RL authors, 2024-2025
https://scholar.google.com/scholar?q=Adaptive+Surrogate+Gradients+for+Sequential+Reinforcement+Learning+in+Spiking+Neural+Networks
29. Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks — approx. recent theoretical SNN authors, 2023-2024
https://scholar.google.com/scholar?q=Elucidating+the+Theoretical+Underpinnings+of+Surrogate+Gradient+Learning+in+Spiking+Neural+Networks
30. Spiking Reinforcement Learning Enhanced by Bioinspired Event Source of Multi-Dendrite Spiking Neuron and Dynamic Thresholds — approx. recent spiking RL authors, 2024-2025
https://scholar.google.com/scholar?q=Spiking+Reinforcement+Learning+Enhanced+by+Bioinspired+Event+Source+of+Multi-Dendrite+Spiking+Neuron+and+Dynamic+Thresholds
31. S2Act: Simple Spiking Actor — approx. recent spiking actor-critic authors, 2024-2025
https://scholar.google.com/scholar?q=S2Act:+Simple+Spiking+Actor
32. AI Post Transformers: Zero-Shot Context Generalization in Reinforcement
Learning from Few Training Contexts — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/zero-shot-context-generalization-in-reinforcement-learning-from-few-training-con/
33. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3
Interactive Visualization: Directly Trained Spiking DQNs for Atari