Sign up to save your podcasts
Or
Share Discrete Representations for Continual Reinforcement Learning
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Discrete Representations for Continual Reinforcement Learning
This episode explores whether learned discrete representations actually improve reinforcement learning, world modeling, and continual adaptation compared with standard continuous latent spaces. It explains how vector-quantized codebook latents, sparse binary-style features, and older ideas like tile coding relate to modern world models, and why the real advantage may come from reduced interference rather than discreteness alone. The discussion centers on three evaluation settings: predicting future dynamics in latent space, improving downstream control in model-free RL, and helping agents adapt to shifting tasks without forgetting earlier behavior. Listeners would find it interesting because it cuts through the “discrete vs. continuous” hype and turns the paper into a sharper engineering question about which representation bottlenecks produce more stable, reusable abstractions under changing conditions.
Sources:
1. Harnessing Discrete Representations For Continual Reinforcement Learning — Edan Meyer, Adam White, Marlos C. Machado, 2023
http://arxiv.org/abs/2312.01203
2. Neural Discrete Representation Learning — Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, 2017
https://papers.nips.cc/paper/2017/hash/7a98af17e63a0ac09ce2e96d03992fbc-Abstract.html
3. Mastering Atari with Discrete World Models — Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy Ba, 2021
https://openreview.net/forum?id=0oabwyZbOu
4. Transformers are Sample-Efficient World Models — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2023
https://openreview.net/forum?id=vhFu1Acb0xb
5. Harnessing Discrete Representations for Continual Reinforcement Learning — Edan Jacob Meyer, Adam White, Marlos C. Machado, 2024
https://openreview.net/forum?id=tCXURNlAZ3
6. Mastering Diverse Domains through World Models — Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy P. Lillicrap, 2023
https://scholar.google.com/scholar?q=Mastering+Diverse+Domains+through+World+Models
7. Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online — Yangchen Pan, Kirby Banman, Martha White, 2021
https://scholar.google.com/scholar?q=Fuzzy+Tiling+Activations%3A+A+Simple+Approach+to+Learning+Sparse+Representations+Online
8. Investigating the Properties of Neural Network Representations in Reinforcement Learning — Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, Adam White, 2022
https://scholar.google.com/scholar?q=Investigating+the+Properties+of+Neural+Network+Representations+in+Reinforcement+Learning
9. Smaller World Models for Reinforcement Learning — Jan Robine, Tobias Uelwer, Stefan Harmeling, 2021
https://scholar.google.com/scholar?q=Smaller+World+Models+for+Reinforcement+Learning
10. Continual Learning as Computationally Constrained Reinforcement Learning — Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, Benjamin Van Roy, 2023
https://scholar.google.com/scholar?q=Continual+Learning+as+Computationally+Constrained+Reinforcement+Learning
11. Efficient World Models with Context-Aware Tokenization — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2024
https://scholar.google.com/scholar?q=Efficient+World+Models+with+Context-Aware+Tokenization
12. AdaWorld: Learning Adaptable World Models with Latent Actions — authors not identifiable from the snippet, recent
https://scholar.google.com/scholar?q=AdaWorld%3A+Learning+Adaptable+World+Models+with+Latent+Actions
13. A Survey of Continual Reinforcement Learning — authors not identifiable from the snippet, recent
https://scholar.google.com/scholar?q=A+Survey+of+Continual+Reinforcement+Learning
14. Stable Continual Reinforcement Learning via Diffusion-Based Trajectory Replay — authors not identifiable from the snippet, recent
https://scholar.google.com/scholar?q=Stable+Continual+Reinforcement+Learning+via+Diffusion-Based+Trajectory+Replay
15. AI Post Transformers: DreamerV3 World Models Across 150 Tasks — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-20-dreamerv3-world-models-across-150-tasks-af5edb.mp3
16. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp3
17. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3
View all episodes
By mcgrofThis episode explores whether learned discrete representations actually improve reinforcement learning, world modeling, and continual adaptation compared with standard continuous latent spaces. It explains how vector-quantized codebook latents, sparse binary-style features, and older ideas like tile coding relate to modern world models, and why the real advantage may come from reduced interference rather than discreteness alone. The discussion centers on three evaluation settings: predicting future dynamics in latent space, improving downstream control in model-free RL, and helping agents adapt to shifting tasks without forgetting earlier behavior. Listeners would find it interesting because it cuts through the “discrete vs. continuous” hype and turns the paper into a sharper engineering question about which representation bottlenecks produce more stable, reusable abstractions under changing conditions.
Sources:
1. Harnessing Discrete Representations For Continual Reinforcement Learning — Edan Meyer, Adam White, Marlos C. Machado, 2023
http://arxiv.org/abs/2312.01203
2. Neural Discrete Representation Learning — Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, 2017
https://papers.nips.cc/paper/2017/hash/7a98af17e63a0ac09ce2e96d03992fbc-Abstract.html
3. Mastering Atari with Discrete World Models — Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy Ba, 2021
https://openreview.net/forum?id=0oabwyZbOu
4. Transformers are Sample-Efficient World Models — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2023
https://openreview.net/forum?id=vhFu1Acb0xb
5. Harnessing Discrete Representations for Continual Reinforcement Learning — Edan Jacob Meyer, Adam White, Marlos C. Machado, 2024
https://openreview.net/forum?id=tCXURNlAZ3
6. Mastering Diverse Domains through World Models — Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy P. Lillicrap, 2023
https://scholar.google.com/scholar?q=Mastering+Diverse+Domains+through+World+Models
7. Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online — Yangchen Pan, Kirby Banman, Martha White, 2021
https://scholar.google.com/scholar?q=Fuzzy+Tiling+Activations%3A+A+Simple+Approach+to+Learning+Sparse+Representations+Online
8. Investigating the Properties of Neural Network Representations in Reinforcement Learning — Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, Adam White, 2022
https://scholar.google.com/scholar?q=Investigating+the+Properties+of+Neural+Network+Representations+in+Reinforcement+Learning
9. Smaller World Models for Reinforcement Learning — Jan Robine, Tobias Uelwer, Stefan Harmeling, 2021
https://scholar.google.com/scholar?q=Smaller+World+Models+for+Reinforcement+Learning
10. Continual Learning as Computationally Constrained Reinforcement Learning — Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, Benjamin Van Roy, 2023
https://scholar.google.com/scholar?q=Continual+Learning+as+Computationally+Constrained+Reinforcement+Learning
11. Efficient World Models with Context-Aware Tokenization — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2024
https://scholar.google.com/scholar?q=Efficient+World+Models+with+Context-Aware+Tokenization
12. AdaWorld: Learning Adaptable World Models with Latent Actions — authors not identifiable from the snippet, recent
https://scholar.google.com/scholar?q=AdaWorld%3A+Learning+Adaptable+World+Models+with+Latent+Actions
13. A Survey of Continual Reinforcement Learning — authors not identifiable from the snippet, recent
https://scholar.google.com/scholar?q=A+Survey+of+Continual+Reinforcement+Learning
14. Stable Continual Reinforcement Learning via Diffusion-Based Trajectory Replay — authors not identifiable from the snippet, recent
https://scholar.google.com/scholar?q=Stable+Continual+Reinforcement+Learning+via+Diffusion-Based+Trajectory+Replay
15. AI Post Transformers: DreamerV3 World Models Across 150 Tasks — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-20-dreamerv3-world-models-across-150-tasks-af5edb.mp3
16. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp3
17. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026
https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3