This episode explores a systems paper on making reinforcement-learning post-training for large language models practical over ordinary Ethernet and even WAN links, rather than requiring expensive RDMA clusters. It explains why trainer-actor RL creates a synchronization bottleneck, how full policy refreshes can dominate runtime on 1 to 10 gigabit networks, and why that turns bandwidth into a hidden limiter of who can run serious RL workloads. The discussion centers on the paper’s proposed solution: lossless sparse delta checkpoints that send only changed parameters, along with carefully encoded indices, streamed in parallel with rollout generation so actors can reconstruct the exact updated model without quantization or approximation. Listeners would find it interesting because it connects low-level systems design to the economics and accessibility of modern LLM training, asking whether better synchronization methods could open RL post-training to labs and startups outside elite infrastructure environments.

Sources:

1. RL over Commodity Networks: Overcoming the Bandwidth Barrier with Lossless Sparse Deltas — Chaoyi Ruan, Geng Luo, Xinyi Wan, Long Zhao, Qinghe Wang, Jiaan Zhu, Duling Xu, Guanbin Xu, Dehui Wei, Xiang Liu, Cheng Li, Haifeng Sun, Congcong Miao, Jialin Li, 2026

http://arxiv.org/abs/2602.11456

2. Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL — Erfan Miahi, Eugene Belilovsky, 2026

https://scholar.google.com/scholar?q=Understanding+and+Exploiting+Weight+Update+Sparsity+for+Communication-Efficient+Distributed+RL

3. StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation — Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, Daxin Jiang, 2025

https://scholar.google.com/scholar?q=StreamRL%3A+Scalable%2C+Heterogeneous%2C+and+Elastic+RL+for+LLMs+with+Disaggregated+Stream+Generation

4. HybridFlow: A Flexible and Efficient RLHF Framework — Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu, 2024

https://scholar.google.com/scholar?q=HybridFlow%3A+A+Flexible+and+Efficient+RLHF+Framework

5. OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework — Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, Yiming Liu, 2024

https://scholar.google.com/scholar?q=OpenRLHF%3A+An+Easy-to-use%2C+Scalable+and+High-performance+RLHF+Framework

6. How Can We Train Deep Learning Models Across Clouds and Continents? An Experimental Study — Alexander Erben, Ruben Mayer, Hans-Arno Jacobsen, 2023

https://scholar.google.com/scholar?q=How+Can+We+Train+Deep+Learning+Models+Across+Clouds+and+Continents%3F+An+Experimental+Study

7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023

https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention

8. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models — authors not confirmed from snippet, recent, unconfirmed from snippet

https://scholar.google.com/scholar?q=Asynchronous+RLHF%3A+Faster+and+More+Efficient+Off-Policy+RL+for+Language+Models

9. Faster, More Efficient RLHF through Off-Policy Asynchronous Learning — authors not confirmed from snippet, recent, unconfirmed from snippet

https://scholar.google.com/scholar?q=Faster%2C+More+Efficient+RLHF+through+Off-Policy+Asynchronous+Learning

10. Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs — authors not confirmed from snippet, recent, unconfirmed from snippet

https://scholar.google.com/scholar?q=Stable+Asynchrony%3A+Variance-Controlled+Off-Policy+RL+for+LLMs

11. Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance — authors not confirmed from snippet, recent, unconfirmed from snippet

https://scholar.google.com/scholar?q=Efficient+Online+RFT+with+Plug-and-Play+LLM+Judges%3A+Unlocking+State-of-the-Art+Performance

12. Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding — authors not confirmed from snippet, recent, unconfirmed from snippet

https://scholar.google.com/scholar?q=Accelerating+RL+Post-Training+Rollouts+via+System-Integrated+Speculative+Decoding

13. Beat the Long Tail: Distribution-Aware Speculative Decoding for RL Training — authors not confirmed from snippet, recent, unconfirmed from snippet

https://scholar.google.com/scholar?q=Beat+the+Long+Tail%3A+Distribution-Aware+Speculative+Decoding+for+RL+Training

14. AI Post Transformers: HALoS: Hierarchical Asynchronous LLM Training over Slow Networks — Hal Turing & Dr. Ada Shannon, 2025

https://podcast.do-not-panic.com/episodes/halos-hierarchical-asynchronous-llm-training-over-slow-networks/

15. AI Post Transformers: TensorFlow for Distributed Machine Learning Systems — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-05-06-tensorflow-for-distributed-machine-learn-b7fa52.mp3

16. AI Post Transformers: AgenticQwen and Small Industrial Tool Agents — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-04-27-agenticqwen-and-small-industrial-tool-ag-dc676d.mp3

Interactive Visualization: Lossless Sparse Deltas for RL Networks