Sign up to save your podcasts
Or
Share SiLU Activations for Replay-Free Atari Reinforcement Learning
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
SiLU Activations for Replay-Free Atari Reinforcement Learning
This episode explores a 2017 paper arguing that sigmoid-weighted activation functions, specifically SiLU and dSiLU, can materially improve deep reinforcement learning when paired with replay-free Sarsa(lambda), eligibility traces, and softmax exploration. It explains why activation choice matters more in bootstrapped value learning than in ordinary supervised settings, and uses that as a lens to unpack older RL concepts like function approximation, TD(lambda), and on-policy learning for listeners coming from modern deep learning. The discussion walks through the paper’s results on SZ-Tetris, 10x10 Tetris, and Atari-style settings, highlighting that dSiLU and mixed SiLU/dSiLU networks outperformed ReLU-based alternatives in several configurations. Listeners would find it interesting because it challenges the idea that replay buffers and DQN-style machinery are the only serious path for high-dimensional RL, and shows how a seemingly small architectural choice can reshape learning dynamics.
Sources:
1. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017
http://arxiv.org/abs/1702.03118
2. Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units — Dan Hendrycks, Kevin Gimpel, 2016
https://scholar.google.com/scholar?q=Bridging+Nonlinearities+and+Stochastic+Regularizers+with+Gaussian+Error+Linear+Units
3. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017
https://scholar.google.com/scholar?q=Sigmoid-Weighted+Linear+Units+for+Neural+Network+Function+Approximation+in+Reinforcement+Learning
4. Searching for Activation Functions — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017
https://scholar.google.com/scholar?q=Searching+for+Activation+Functions
5. GLU Variants Improve Transformer — Noam Shazeer, 2020
https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer
6. Learning to Predict by the Methods of Temporal Differences — Richard S. Sutton, 1988
https://scholar.google.com/scholar?q=Learning+to+Predict+by+the+Methods+of+Temporal+Differences
7. True Online Temporal-Difference Learning — Harm van Seijen, A. Rupam Mahmood, Patrick M. Pilarski, Marlos C. Machado, Richard S. Sutton, 2015
https://scholar.google.com/scholar?q=True+Online+Temporal-Difference+Learning
8. High-Dimensional Continuous Control Using Generalized Advantage Estimation — John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel, 2015
https://scholar.google.com/scholar?q=High-Dimensional+Continuous+Control+Using+Generalized+Advantage+Estimation
9. Multi-step Reinforcement Learning: A Unifying Algorithm — Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, Richard S. Sutton, 2017
https://scholar.google.com/scholar?q=Multi-step+Reinforcement+Learning%3A+A+Unifying+Algorithm
10. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013
https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning
11. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016
https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning
12. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017
https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms
13. Training Language Models to Follow Instructions with Human Feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe, 2022
https://scholar.google.com/scholar?q=Training+Language+Models+to+Follow+Instructions+with+Human+Feedback
14. Human-level control through deep reinforcement learning — Volodymyr Mnih et al., 2015
https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning
15. Deep reinforcement learning with double q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2015
https://scholar.google.com/scholar?q=Deep+reinforcement+learning+with+double+q-learning
16. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2016
https://scholar.google.com/scholar?q=Prioritized+Experience+Replay
17. High-dimensional function approximation for knowledge-free reinforcement learning: a case study in SZ-Tetris — Wojciech Jaskowski, Maciej Szubert, Pawel Liskowski, Krzysztof Krawiec, 2015
https://scholar.google.com/scholar?q=High-dimensional+function+approximation+for+knowledge-free+reinforcement+learning%3A+a+case+study+in+SZ-Tetris
18. Approximate dynamic programming finally performs well in the game of tetris — Victor Gabillon, Mohammad Ghavamzadeh, Bruno Scherrer, 2013
https://scholar.google.com/scholar?q=Approximate+dynamic+programming+finally+performs+well+in+the+game+of+tetris
19. Replay across Experiments: A Natural Extension of Off-Policy RL — Dhruva Tirumala et al., 2023
https://arxiv.org/abs/2311.15951
20. Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning — Theo Vincent et al., 2024
https://arxiv.org/abs/2405.16195
21. A Survey of Temporal Credit Assignment in Deep Reinforcement Learning — Eduardo Pignatelli et al., 2023
https://arxiv.org/abs/2312.01072
22. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp3
23. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3
View all episodes
By mcgrofThis episode explores a 2017 paper arguing that sigmoid-weighted activation functions, specifically SiLU and dSiLU, can materially improve deep reinforcement learning when paired with replay-free Sarsa(lambda), eligibility traces, and softmax exploration. It explains why activation choice matters more in bootstrapped value learning than in ordinary supervised settings, and uses that as a lens to unpack older RL concepts like function approximation, TD(lambda), and on-policy learning for listeners coming from modern deep learning. The discussion walks through the paper’s results on SZ-Tetris, 10x10 Tetris, and Atari-style settings, highlighting that dSiLU and mixed SiLU/dSiLU networks outperformed ReLU-based alternatives in several configurations. Listeners would find it interesting because it challenges the idea that replay buffers and DQN-style machinery are the only serious path for high-dimensional RL, and shows how a seemingly small architectural choice can reshape learning dynamics.
Sources:
1. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017
http://arxiv.org/abs/1702.03118
2. Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units — Dan Hendrycks, Kevin Gimpel, 2016
https://scholar.google.com/scholar?q=Bridging+Nonlinearities+and+Stochastic+Regularizers+with+Gaussian+Error+Linear+Units
3. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning — Stefan Elfwing, Eiji Uchibe, Kenji Doya, 2017
https://scholar.google.com/scholar?q=Sigmoid-Weighted+Linear+Units+for+Neural+Network+Function+Approximation+in+Reinforcement+Learning
4. Searching for Activation Functions — Prajit Ramachandran, Barret Zoph, Quoc V. Le, 2017
https://scholar.google.com/scholar?q=Searching+for+Activation+Functions
5. GLU Variants Improve Transformer — Noam Shazeer, 2020
https://scholar.google.com/scholar?q=GLU+Variants+Improve+Transformer
6. Learning to Predict by the Methods of Temporal Differences — Richard S. Sutton, 1988
https://scholar.google.com/scholar?q=Learning+to+Predict+by+the+Methods+of+Temporal+Differences
7. True Online Temporal-Difference Learning — Harm van Seijen, A. Rupam Mahmood, Patrick M. Pilarski, Marlos C. Machado, Richard S. Sutton, 2015
https://scholar.google.com/scholar?q=True+Online+Temporal-Difference+Learning
8. High-Dimensional Continuous Control Using Generalized Advantage Estimation — John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, Pieter Abbeel, 2015
https://scholar.google.com/scholar?q=High-Dimensional+Continuous+Control+Using+Generalized+Advantage+Estimation
9. Multi-step Reinforcement Learning: A Unifying Algorithm — Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, Richard S. Sutton, 2017
https://scholar.google.com/scholar?q=Multi-step+Reinforcement+Learning%3A+A+Unifying+Algorithm
10. Playing Atari with Deep Reinforcement Learning — Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, 2013
https://scholar.google.com/scholar?q=Playing+Atari+with+Deep+Reinforcement+Learning
11. Asynchronous Methods for Deep Reinforcement Learning — Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, Koray Kavukcuoglu, 2016
https://scholar.google.com/scholar?q=Asynchronous+Methods+for+Deep+Reinforcement+Learning
12. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017
https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms
13. Training Language Models to Follow Instructions with Human Feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe, 2022
https://scholar.google.com/scholar?q=Training+Language+Models+to+Follow+Instructions+with+Human+Feedback
14. Human-level control through deep reinforcement learning — Volodymyr Mnih et al., 2015
https://scholar.google.com/scholar?q=Human-level+control+through+deep+reinforcement+learning
15. Deep reinforcement learning with double q-learning — Hado van Hasselt, Arthur Guez, David Silver, 2015
https://scholar.google.com/scholar?q=Deep+reinforcement+learning+with+double+q-learning
16. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2016
https://scholar.google.com/scholar?q=Prioritized+Experience+Replay
17. High-dimensional function approximation for knowledge-free reinforcement learning: a case study in SZ-Tetris — Wojciech Jaskowski, Maciej Szubert, Pawel Liskowski, Krzysztof Krawiec, 2015
https://scholar.google.com/scholar?q=High-dimensional+function+approximation+for+knowledge-free+reinforcement+learning%3A+a+case+study+in+SZ-Tetris
18. Approximate dynamic programming finally performs well in the game of tetris — Victor Gabillon, Mohammad Ghavamzadeh, Bruno Scherrer, 2013
https://scholar.google.com/scholar?q=Approximate+dynamic+programming+finally+performs+well+in+the+game+of+tetris
19. Replay across Experiments: A Natural Extension of Off-Policy RL — Dhruva Tirumala et al., 2023
https://arxiv.org/abs/2311.15951
20. Adaptive Q-Network: On-the-fly Target Selection for Deep Reinforcement Learning — Theo Vincent et al., 2024
https://arxiv.org/abs/2405.16195
21. A Survey of Temporal Credit Assignment in Deep Reinforcement Learning — Eduardo Pignatelli et al., 2023
https://arxiv.org/abs/2312.01072
22. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp3
23. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp3