AI Post Transformers

This episode explores the xLLM Technical Report from JD.com, which describes a production system for running large language model inference at enterprise scale. The core technical challenge is co-locating online chatbot workloads with offline batch jobs on the same GPU cluster to maximize hardware utilization during traffic valleys, while maintaining strict latency guarantees during peak hours. The discussion covers key architectural decisions including dynamic prefill-decode disaggregation, which adaptively reallocates compute resources between the prompt processing phase and token generation phase based on real-time workload characteristics, and the management of unpredictable KV cache memory growth that makes LLM co-location harder than traditional cloud workload mixing. Listeners interested in production ML systems engineering, GPU cluster optimization, and the practical challenges of deploying transformer models at scale will find concrete insights into how a major tech company handles the resource scheduling problems that academic papers often overlook.

Sources:

1. xLLM Technical Report — Tongxuan Liu, Tao Peng, Peijun Yang, Xiaoyang Zhao, Xiusheng Lu, Weizhe Huang, Zirui Liu, Xiaoyu Chen, Zhiwei Liang, Jun Xiong, Donghe Jin, Minchao Zhang, Jinrong Guo, Yingxu Deng, Xu Zhang, Xianzhe Dong, Siqi Wang, Siyu Wu, Yu Wu, Zihan Tang, Yuting Zeng, Yanshu Wang, Jinguang Liu, Meng Kang, Menxin Li, Yunlong Wang, Yiming Liu, Xiaolong Ma, Yifan Wang, Yichen Zhang, Jinrun Yin, Keyang Zheng, Jiawei Yin, Jun Zhang, Ziyue Wang, Xiaobo Lin, Liangyu Liu, Liwei Lan, Yang Liu, Chunhua Peng, Han Liu, Songcheng Ren, Xuezhu Wang, Yunheng Shen, Yi Wang, Guyue Liu, Yitao Hu, Hui Chen, Tong Yang, Hailong Yang, Jing Li, Guiguang Ding, Ke Zhang, 2025

http://arxiv.org/abs/2510.14686v2

2. Borg, Omega, and Kubernetes: Lessons learned from three container-management systems over a decade — Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, John Wilkes, 2016

https://scholar.google.com/scholar?q=Borg%2C+Omega%2C+and+Kubernetes%3A+Lessons+learned+from+three+container-management+systems+over+a+decade

3. SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Ramachandran Ramjee, 2023

https://scholar.google.com/scholar?q=SARATHI%3A+Efficient+LLM+Inference+by+Piggybacking+Decodes+with+Chunked+Prefills

4. Paella: Low-latency Model Serving with Virtualized GPU Scheduling — Yiying Zhang, et al., 2024

https://scholar.google.com/scholar?q=Paella%3A+Low-latency+Model+Serving+with+Virtualized+GPU+Scheduling

5. Synergy: A QoS-aware Multi-DNN Inference Framework for Heterogeneous Systems-on-Chip — Jie Ren, et al., 2022

https://scholar.google.com/scholar?q=Synergy%3A+A+QoS-aware+Multi-DNN+Inference+Framework+for+Heterogeneous+Systems-on-Chip

6. TensorFlow: A System for Large-Scale Machine Learning — Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, et al., 2016

https://scholar.google.com/scholar?q=TensorFlow%3A+A+System+for+Large-Scale+Machine+Learning

7. XLA: Optimizing Compiler for Machine Learning — The TensorFlow team (Google), 2017

https://scholar.google.com/scholar?q=XLA%3A+Optimizing+Compiler+for+Machine+Learning

8. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning — Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, et al., 2018

https://scholar.google.com/scholar?q=TVM%3A+An+Automated+End-to-End+Optimizing+Compiler+for+Deep+Learning

9. torch.compile: A Compiler for PyTorch Models (PyTorch 2.0 Technical Report) — Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, et al. (Meta), 2023

https://scholar.google.com/scholar?q=torch.compile%3A+A+Compiler+for+PyTorch+Models+%28PyTorch+2.0+Technical+Report%29

10. Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al., 2022

https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models

11. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Sheng et al., 2023

https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU

12. DeepSpeed-MII: Instant Speedup on Inference with Advanced Model Implementations and System Techniques — Microsoft, 2023

https://scholar.google.com/scholar?q=DeepSpeed-MII%3A+Instant+Speedup+on+Inference+with+Advanced+Model+Implementations+and+System+Techniques

13. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving — Li et al., 2023

https://scholar.google.com/scholar?q=AlpaServe%3A+Statistical+Multiplexing+with+Model+Parallelism+for+Deep+Learning+Serving

14. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Cai et al., 2024

https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads

15. POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference — approximate 2025, 2025

https://scholar.google.com/scholar?q=POD-Attention%3A+Unlocking+Full+Prefill-Decode+Overlap+for+Faster+LLM+Inference

16. FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving — approximate 2025, 2025

https://scholar.google.com/scholar?q=FlowPrefill%3A+Decoupling+Preemption+from+Prefill+Scheduling+Granularity+to+Mitigate+Head-of-Line+Blocking+in+LLM+Serving

17. KvLink: Accelerating Large Language Models via Efficient KV Cache Reuse — approximate 2024-2025, 2024

https://scholar.google.com/scholar?q=KvLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse

18. Decoding Speculative Decoding — approximate 2024, 2024

https://scholar.google.com/scholar?q=Decoding+Speculative+Decoding

19. A Comprehensive Survey of Mixture-of-Experts — approximate 2024-2025, 2024

https://scholar.google.com/scholar?q=A+Comprehensive+Survey+of+Mixture-of-Experts

20. DualPath Breaks Storage Bandwidth Bottleneck in Agentic Inference

https://podcast.do-not-panic.com/episodes/2026-03-07-dualpath-breaks-storage-bandwidth-bottle-bc9a82.mp3

21. FlashAttention-4 Conquers Asymmetric GPU Hardware Scaling

https://podcast.do-not-panic.com/episodes/2026-03-06-flashattention-4-conquers-asymmetric-gpu-78839b.mp3

22. 50x KV Cache Compression in Seconds via Attention Matching

https://podcast.do-not-panic.com/episodes/2026-03-09-50x-kv-cache-compression-in-seconds-via-9402c1.mp3

23. Systematic Characterization of LLM Inference on GPUs

https://podcast.do-not-panic.com/episodes/2026-03-07-systematic-characterization-of-llm-infer-f7b9a8.mp3

Interactive Visualization: xLLM: Co-Locating Online and Offline LLM Inference

This episode examines CacheSlide from USENIX FAST26, a system that enables LLMs to reuse cached key-value pairs across shifting prompt positions in agentic workflows. The paper introduces chunked contextual position encoding and priority-based eviction to solve the position mismatch problem that prevents KV cache reuse when prompt segments shift in multi-turn agent conversations.

Interactive Visualization: CacheSlide: Position-Aware KV Cache Reuse for Agent LLMs

This episode examines "Attention Residuals," a March 2026 paper from Moonshot AI's Kimi team that challenges a foundational element of transformer architecture. The paper proposes replacing the fixed, uniform residual connections inherited from ResNet with learned attention mechanisms for depth-wise information aggregation. While attention replaced recurrent neural networks for sequence modeling over a decade ago, the authors argue that depth-wise aggregation remains stuck with the same fixed summation from 2015, creating an architectural asymmetry where learned selection should be used instead. The episode traces the dual role of residual connections — serving both as gradient highways for backpropagation and as information aggregation mechanisms — and explains why the latter has become a performance bottleneck in deep transformers.

The discussion centers on the PreNorm dilution problem, where unnormalized residual accumulation causes hidden-state magnitudes to grow linearly with network depth, progressively burying individual layer contributions under an ever-growing pile of summed vectors. In PreNorm architectures, despite their superior gradient stability during training, deep layer outputs contribute only one percent of the total magnitude at layer 100. This dilution effect helps explain why layer pruning experiments often show minimal performance loss when removing significant fractions of trained layers. The Kimi team's solution replaces fixed unit-weight summation with softmax attention over previous layer outputs, where each layer learns a single query vector to compute content-dependent weights that sum to one, maintaining bounded magnitude while enabling selective information aggregation.

The episode examines experimental validation across multiple model scales, from 460 million to 7 billion parameters, trained on datasets up to 100 billion tokens. Results show consistent improvements in perplexity and downstream task performance, with particularly strong gains in deeper models where dilution effects are most severe. The architecture introduces minimal computational overhead — approximately 3 percent — by sharing key-value projections with the self-attention sublayer and caching attention weights across the depth dimension. Design variants including bidirectional attention and explicit current-layer queries are explored, with causal attention and implicit queries recommended as the optimal configuration for both performance and efficiency in production deployments.

Interactive Visualization: Rethinking Residual Connections: Learned Attention Across Transformer Depth

This episode examines the MATT (Model-Aware Tokenizer Transfer) paper from AGH University of Krakow, which proposes a fundamentally different approach to extending language models to underserved languages. Using Georgian as the central case study, the episode explains tokenizer fertility — how tokenizers optimized for high-resource languages fragment Georgian words into six to eight subword pieces, consuming context budget and degrading both accuracy and inference speed.

The episode traces the lineage of tokenizer transfer methods from WECHSEL through FOCUS and ZETT, each of which initializes new embeddings by finding semantically similar source tokens via bilingual dictionaries or FastText projections. MATT's contribution — Attention-Informed Mapping (AIM) — reframes the problem: rather than asking which source tokens are semantically closest, it asks which embeddings are most compatible with what the model's attention layers already know how to route. This is grounded in mechanistic interpretability research showing that factual knowledge resides in FFN layers, not embeddings, making tokenizer swap feasible in principle.

The episode includes a detailed comparison with the Cartridges approach, which tackles a closely related problem from a different architectural angle. Four parallel threads are developed: the Structured Continual Initialization parallel, the key-as-router insight, the separation of FFN knowledge from attention routing, and the FFN token-ID binding risk that MATT's evaluation never directly probes. The discussion argues this last point represents the sharpest untested assumption in the paper — whether feed-forward layers develop token-specific associations that break silently when vocabulary changes.

Interactive Visualization: Model-Aware Tokenizer Transfer for Multilingual LLMs

We ran out of ElevenLabs credits. This episode introduces our new open-source voices powered by Kokoro, an 82-million parameter text-to-speech model built on StyleTTS 2. We explain the Docker container saga of running Python 3.12 dependencies on a 3.13 host, rave about CPU-only inference speed, tease a future deep-dive on the papers behind lightweight neural TTS, demo Spanish multilingual support, and test whether our new voices can laugh. Plus: we are massively backlogged with topics including FAST 2026 conference coverage.

Interactive Visualization: New Voices, Same Nerds: The Kokoro TTS Episode

This episode examines a Meta-led paper that develops the first systematic scaling laws for reinforcement learning in large language models, based on over 400,000 GPU-hours of experiments. The researchers propose a sigmoid framework to predict RL performance at large compute budgets from early training runs, addressing a critical gap in the field—while pre-training has well-established power-law relationships like Chinchilla scaling, RL has lacked predictive models due to shifting data distributions and bounded reward functions. The work focuses on mathematical reasoning tasks using AIME problems and introduces ScaleRL, a best-practice training recipe that successfully extrapolates performance from 50,000 to 100,000 GPU-hours. However, the hosts raise important questions about generalization beyond math to domains like code and dialogue, and whether the smooth sigmoid curves capture potential phase transitions or emergent capabilities that might appear at higher compute scales.

Sources:

1. The Art of Scaling Reinforcement Learning Compute for LLMs — Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, Rishabh Agarwal, 2025

http://arxiv.org/abs/2510.13786v1

2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al., 2020

https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models

3. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al., 2022

https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29

4. Scaling Laws for Reward Model Overoptimization — Leo Gao, John Schulman, Jacob Hilton, 2023

https://scholar.google.com/scholar?q=Scaling+Laws+for+Reward+Model+Overoptimization

5. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI Team (Guo et al.), 2025

https://scholar.google.com/scholar?q=DeepSeek-R1%3A+Incentivizing+Reasoning+Capability+in+LLMs+via+Reinforcement+Learning

6. Proximal Policy Optimization Algorithms — John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov, 2017

https://scholar.google.com/scholar?q=Proximal+Policy+Optimization+Algorithms

7. Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, 2023

https://scholar.google.com/scholar?q=Direct+Preference+Optimization%3A+Your+Language+Model+is+Secretly+a+Reward+Model

8. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, 2022

https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models

9. STaR: Self-Taught Reasoner Bootstrapping Reasoning With Reasoning — Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman, 2022

https://scholar.google.com/scholar?q=STaR%3A+Self-Taught+Reasoner+Bootstrapping+Reasoning+With+Reasoning

10. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2023

https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step

11. Reward Model Ensembles Help Mitigate Overoptimization — Thomas Coste, Usman Anwar, Robert Kirk, David Krueger, 2023

https://scholar.google.com/scholar?q=Reward+Model+Ensembles+Help+Mitigate+Overoptimization

12. DAPO: Data-Adaptive Policy Optimization — Yu et al., 2025

https://scholar.google.com/scholar?q=DAPO%3A+Data-Adaptive+Policy+Optimization

13. Chinchilla: Training Compute-Optimal Large Language Models — Hoffmann et al., 2022

https://scholar.google.com/scholar?q=Chinchilla%3A+Training+Compute-Optimal+Large+Language+Models

14. OpenAI o1 System Card — OpenAI, 2024

https://scholar.google.com/scholar?q=OpenAI+o1+System+Card

15. Direct Preference Optimization — Rafailov et al., 2023

https://scholar.google.com/scholar?q=Direct+Preference+Optimization

16. STaR: Self-Taught Reasoner — Zelikman et al., 2022

https://scholar.google.com/scholar?q=STaR%3A+Self-Taught+Reasoner

17. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization — approximate authors unknown from snippet, recent (post-2024 based on context)

https://scholar.google.com/scholar?q=REINFORCE%2B%2B%3A+Stabilizing+Critic-Free+Policy+Optimization+with+Global+Advantage+Normalization

18. Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models — approximate authors unknown from snippet, recent (likely 2024-2025)

https://scholar.google.com/scholar?q=Asynchronous+RLHF%3A+Faster+and+More+Efficient+Off-Policy+RL+for+Language+Models

19. Tapered Off-Policy REINFORCE: Stable and Efficient Reinforcement Learning for LLMs — approximate authors unknown from snippet, recent (likely 2025)

https://scholar.google.com/scholar?q=Tapered+Off-Policy+REINFORCE%3A+Stable+and+Efficient+Reinforcement+Learning+for+LLMs

20. A Survey of Post-Training Scaling in Large Language Models — approximate authors unknown from snippet, recent

https://scholar.google.com/scholar?q=A+Survey+of+Post-Training+Scaling+in+Large+Language+Models

21. Are Emergent Abilities in Large Language Models Just In-Context Learning? — approximate authors unknown from snippet, recent

https://scholar.google.com/scholar?q=Are+Emergent+Abilities+in+Large+Language+Models+Just+In-Context+Learning%3F

22. AI Post Transformers: Gradient Descent at Inference Time for LLM Reasoning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-10-gradient-descent-at-inference-time-for-l-20617d.mp3

23. AI Post Transformers: Emergent Cooperation in Self-Interested Multi-Agent AI — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-13-emergent-cooperation-in-self-interested-9c0b4c.mp3

Interactive Visualization: The Art of Scaling Reinforcement Learning Compute for LLMs

This episode of AI Post Transformers examines "Agentic Code Reasoning" by Shubham Ugare and Satish Chandra from Meta, which introduces semi-formal reasoning certificates as an inference-time scaffold for LLM agents analyzing code without executing it. Rather than letting a model produce free-form chain-of-thought verdicts, the certificate framework requires the agent to state explicit premises, trace execution paths through real repository code, and produce a structured, auditable reasoning record for every claim it makes about code behavior. The Django bug django-13670 — involving two-digit year formatting for years before 1000 CE — anchors the discussion: two patches both claim to fix the same issue, but unstated assumptions about name resolution cause an unstructured model to misidentify which one is correct. The certificate format forces the agent to chase the actual import chain across modules rather than guess based on a function name, turning premise verification into a natural driver of interprocedural analysis. Hosts Hal Turing and Dr. Ada Shannon situate the paper against the spectrum from fully formal proof assistants like Lean and Coq — which are provably correct but completely impractical for arbitrary repository code — down to unstructured LLM judges like CodeJudge and SWE-RM, which let the model skip edge cases and produce confident wrong answers. The certificate sits between those extremes, imposing enough structure to make implicit assumptions visible without requiring formalized language semantics.

The episode traces how the agentic setup amplifies the value of the certificate structure. Using a minimal SWE-agent configuration with bash tool access but no code execution, the agent can navigate the file system, run grep queries, and follow import chains — exploration scope without runtime confirmation. That constraint is precisely where interprocedural tracing becomes load-bearing: the agent cannot run the code to confirm a hypothesis, so it must read the actual call chain to know what a function does rather than infer from its name. The certificate makes that tracing explicit and auditable, which opens a secondary use case beyond RL reward signal generation: automated code review where a human auditor can inspect the agent's reasoning chain rather than accept a black-box verdict. Hal and Ada discuss RL training pipelines as the paper's stated primary motivation — execution-free reward signals could meaningfully reduce the cost of running sandboxed test suites at scale — but are careful to position that as a downstream consequence of the certificate's properties rather than its defining contribution.

The episode closes on three open problems the paper leaves unresolved. First, the inference cost gap: the certificate framework adds computation at inference time, but the paper reports no latency measurements, no tokens-per-certificate data, and no comparison against unstructured baselines on cost — making it impossible to assess whether the accuracy gains justify the overhead in production. Second, certificate reuse as a concrete future direction: common interprocedural patterns across a codebase — frequently called utilities, stable library interfaces — could in principle be cached and reused across multiple verification queries, amortizing the inference cost that the paper never measures. Third, verification independence: the paper's circular verification problem remains open, since the same model that generates a certificate is also the model best positioned to judge whether the premises in that certificate are sound. Separating generation from verification — whether through a distinct model, a symbolic checker, or a human auditor — is the structural fix the framework points toward but does not yet provide.

Sources:

1. Agentic Code Reasoning — Shubham Ugare, Satish Chandra, 2026

http://arxiv.org/abs/2603.01896

2. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez et al., 2024

https://scholar.google.com/scholar?q=SWE-bench%3A+Can+Language+Models+Resolve+Real-World+GitHub+Issues%3F

3. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al., 2022

https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models

4. Program Equivalence — Godlin and Strichman, 2008

https://scholar.google.com/scholar?q=Program+Equivalence

5. LLM-based agents for automated software engineering: A survey — Multiple authors, 2024-2025

https://scholar.google.com/scholar?q=LLM-based+agents+for+automated+software+engineering%3A+A+survey

6. CodeJudge: Evaluating Code Generation with Large Language Models — Tong and Zhang, 2024

https://scholar.google.com/scholar?q=CodeJudge%3A+Evaluating+Code+Generation+with+Large+Language+Models

7. On designing effective RL reward at training time for LLM reasoning — approximate, 2024-2025, 2024-2025

https://scholar.google.com/scholar?q=On+designing+effective+RL+reward+at+training+time+for+LLM+reasoning

8. Large language model critics for execution-free evaluation of code changes — approximate, 2024-2025, 2024-2025

https://scholar.google.com/scholar?q=Large+language+model+critics+for+execution-free+evaluation+of+code+changes

9. AgentFL: Scaling LLM-based fault localization to project-level context — approximate, 2024-2025, 2024-2025

https://scholar.google.com/scholar?q=AgentFL%3A+Scaling+LLM-based+fault+localization+to+project-level+context

10. SoapFL: A Standard Operating Procedure for LLM-based Method-Level Fault Localization — approximate, 2024-2025, 2024-2025

https://scholar.google.com/scholar?q=SoapFL%3A+A+Standard+Operating+Procedure+for+LLM-based+Method-Level+Fault+Localization

11. Structured chain-of-thought prompting for code generation — approximate, 2022-2024, 2022-2024

https://scholar.google.com/scholar?q=Structured+chain-of-thought+prompting+for+code+generation

12. Deductive verification of chain-of-thought reasoning — approximate, 2023-2024, 2023-2024

https://scholar.google.com/scholar?q=Deductive+verification+of+chain-of-thought+reasoning

13. AI Post Transformers: Reasoning About Code Without Running It — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-08-reasoning-about-code-without-running-it-a9d01a.mp3

14. AI Post Transformers: Gradient Descent at Inference Time for LLM Reasoning — Hal Turing & Dr. Ada Shannon, 2026

https://podcast.do-not-panic.com/episodes/2026-03-10-gradient-descent-at-inference-time-for-l-20617d.mp3

Interactive Visualization: LLM Agents Reason About Code Without Running It

This episode explores the challenge of getting self-interested AI agents to cooperate without hardcoding cooperative behavior, examining a 2026 Google paper on multi-agent cooperation through in-context co-player inference. The hosts build up the technical foundations carefully, explaining why standard reinforcement learning breaks down in multi-agent settings due to non-stationarity, and how social dilemmas like the Prisoner's Dilemma cause agents to reliably converge on mutual defection even when cooperation would benefit everyone. The discussion traces the lineage of learning-aware agents, particularly LOLA, which achieved cooperation by differentiating through an opponent's gradient updates — a clever but architecturally demanding approach. The paper under review argues that training a transformer on a diverse pool of co-players lets in-context learning produce emergent cooperation without any of that machinery. Listeners interested in the intersection of game theory, multi-agent RL, and modern sequence modeling will find the episode's careful unpacking of why prior approaches fell short — and what the new framing claims to replace — genuinely illuminating.

Sources:

1. Multi-agent cooperation through in-context co-player inference — Marissa A. Weis, Maciej Wołczyk, Rajai Nasser, Rif A. Saurous, Blaise Agüera y Arcas, João Sacramento, Alexander Meulemans, 2026

http://arxiv.org/abs/2602.16301

2. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments — Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch, 2017

https://scholar.google.com/scholar?q=Multi-Agent+Actor-Critic+for+Mixed+Cooperative-Competitive+Environments

3. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning — Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Waard, Georgios Papoudakis, Jakob Foerster, Shimon Whiteson, 2018

https://scholar.google.com/scholar?q=QMIX%3A+Monotonic+Value+Function+Factorisation+for+Deep+Multi-Agent+Reinforcement+Learning

4. A Survey and Critique of Multiagent Deep Reinforcement Learning — Pablo Hernandez-Leal, Bilal Kartal, Matthew E. Taylor, 2019

https://scholar.google.com/scholar?q=A+Survey+and+Critique+of+Multiagent+Deep+Reinforcement+Learning

5. Multi-Agent Transformer: Scalable Cooperative Multi-Agent Reinforcement Learning — Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, Yaodong Yang, 2022

https://scholar.google.com/scholar?q=Multi-Agent+Transformer%3A+Scalable+Cooperative+Multi-Agent+Reinforcement+Learning

6. Language Models are Few-Shot Learners — Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al., 2020

https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners

7. An Explanation of In-Context Learning as Implicit Bayesian Inference — Sang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu Ma, 2022

https://scholar.google.com/scholar?q=An+Explanation+of+In-Context+Learning+as+Implicit+Bayesian+Inference

8. Transformers Learn In-Context by Gradient Descent — Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, Joao Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov, 2023

https://scholar.google.com/scholar?q=Transformers+Learn+In-Context+by+Gradient+Descent

9. Algorithm Distillation in Reinforcement Learning — Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Morrill, Abhinav Gupta, Pieter Abbeel, Oriol Vinyals, 2023

https://scholar.google.com/scholar?q=Algorithm+Distillation+in+Reinforcement+Learning

10. Emergent Complexity via Multi-Agent Competition — Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch, 2018

https://scholar.google.com/scholar?q=Emergent+Complexity+via+Multi-Agent+Competition

11. Emergent Tool Use from Multi-Agent Interaction — Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, Igor Mordatch, 2019

https://scholar.google.com/scholar?q=Emergent+Tool+Use+from+Multi-Agent+Interaction

12. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning — Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z. Leibo, Nando de Freitas, 2019

https://scholar.google.com/scholar?q=Social+Influence+as+Intrinsic+Motivation+for+Multi-Agent+Deep+Reinforcement+Learning

13. Cooperative Multi-Agent Learning: The State of the Art — Liviu Panait, Sean Luke, 2005

https://scholar.google.com/scholar?q=Cooperative+Multi-Agent+Learning%3A+The+State+of+the+Art

14. The Evolution of Cooperation — Robert Axelrod, William D. Hamilton, 1981

https://scholar.google.com/scholar?q=The+Evolution+of+Cooperation

15. Iterated Prisoner's Dilemma Contains Strategies that Dominate Any Evolutionary Opponent — William H. Press, Freeman J. Dyson, 2012

https://scholar.google.com/scholar?q=Iterated+Prisoner%27s+Dilemma+Contains+Strategies+that+Dominate+Any+Evolutionary+Opponent

16. Evolutionary Instability of Zero-Determinant Strategies Demonstrates That Winning Is Not Everything — Christoph Adami, Arend Hintze, 2013

https://scholar.google.com/scholar?q=Evolutionary+Instability+of+Zero-Determinant+Strategies+Demonstrates+That+Winning+Is+Not+Everything

17. Learning with Opponent-Learning Awareness (LOLA) — Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, Igor Mordatch, 2018

https://scholar.google.com/scholar?q=Learning+with+Opponent-Learning+Awareness+%28LOLA%29

18. Learning with Opponent-Learning Awareness — Jakob Foerster, Richard Y. Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, Igor Mordatch, 2018

https://scholar.google.com/scholar?q=Learning+with+Opponent-Learning+Awareness

19. Model-Free Opponent Shaping — Chris Lu, Timon Willi, Christian Schroeder de Waard, Jakob Foerster, 2022

https://scholar.google.com/scholar?q=Model-Free+Opponent+Shaping

20. In-context reinforcement learning with algorithm distillation — Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., Gazeau, M., Sahni, H., Singh, S., Mnih, V., 2023

https://scholar.google.com/scholar?q=In-context+reinforcement+learning+with+algorithm+distillation

21. RL^2: Fast reinforcement learning via slow reinforcement learning — Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., Abbeel, P., 2016

https://scholar.google.com/scholar?q=RL%5E2%3A+Fast+reinforcement+learning+via+slow+reinforcement+learning

22. Cooperating with unknown teammates in complex domains by acting carefully with information — Aghajohari, M., Duque, J., Cooijmans, T., Courville, A., 2024

https://scholar.google.com/scholar?q=Cooperating+with+unknown+teammates+in+complex+domains+by+acting+carefully+with+information

23. From naive to learning-aware: Emergence of cooperative behaviors in multi-agent systems — Meulemans, A., et al., 2025

https://scholar.google.com/scholar?q=From+naive+to+learning-aware%3A+Emergence+of+cooperative+behaviors+in+multi-agent+systems

24. Generative agents: Interactive simulacra of human behavior — Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., Bernstein, M. S., 2023

https://scholar.google.com/scholar?q=Generative+agents%3A+Interactive+simulacra+of+human+behavior

25. Do Pre-trained Transformers Really Learn In-context by Gradient Descent? — Shen et al. (approximate), 2023-2024

https://scholar.google.com/scholar?q=Do+Pre-trained+Transformers+Really+Learn+In-context+by+Gradient+Descent%3F

26. When is diversity rewarded in cooperative multi-agent learning? — approximate, exact authors not in snippet, recent

https://scholar.google.com/scholar?q=When+is+diversity+rewarded+in+cooperative+multi-agent+learning%3F

27. The evolution of zero-determinant strategies in public goods game — approximate, exact authors not in snippet, recent

https://scholar.google.com/scholar?q=The+evolution+of+zero-determinant+strategies+in+public+goods+game

28. Uncoupled learning of differential Stackelberg equilibria with commitments — approximate, exact authors not in snippet, recent

https://scholar.google.com/scholar?q=Uncoupled+learning+of+differential+Stackelberg+equilibria+with+commitments

29. Non-coercive extortion in game theory — approximate, exact authors not in snippet, recent

https://scholar.google.com/scholar?q=Non-coercive+extortion+in+game+theory

30. Reciprocal reward influence encourages cooperation from self-interested agents — approximate, exact authors not in snippet, recent

https://scholar.google.com/scholar?q=Reciprocal+reward+influence+encourages+cooperation+from+self-interested+agents

31. AI Post Transformers: In-Context Learning as Implicit Learning Algorithms — Hal Turing & Dr. Ada Shannon, Wed,

https://podcasters.spotify.com/pod/show/12146088098/episodes/In-Context-Learning-as-Implicit-Learning-Algorithms-e39sjmn

32. AI Post Transformers: Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts — Hal Turing & Dr. Ada Shannon, Tue,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Zero-Shot-Context-Generalization-in-Reinforcement--Learning-from-Few-Training-Contexts-e3fi0t5

33. AI Post Transformers: Experiential Reinforcement Learning: Internalizing Reflection for Better Policy Training — Hal Turing & Dr. Ada Shannon, Fri,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Experiential-Reinforcement-Learning-Internalizing-Reflection-for-Better-Policy-Training-e3fbel0

Interactive Visualization: Emergent Cooperation in Self-Interested Multi-Agent AI

This episode examines a 2026 MIT paper claiming a 50x KV cache memory reduction that runs in seconds rather than the GPU-hours required by prior latent-space compaction methods. It grounds the claim in a detailed technical primer on KV cache mechanics — explaining why memory consumption scales multiplicatively across layers, heads, and context length, reaching 8–16 GB per request at 64K-token contexts. The discussion traces the compaction landscape from token eviction approaches like H2O and SnapKV, through token merging, to the latent-space paradigm introduced by Cartridges, establishing why earlier methods collapse at extreme compression ratios. The central question is whether "Fast KV Compaction via Attention Matching" genuinely pushes the quality-versus-speed Pareto frontier — making per-request inference-time compaction practical rather than a research pipeline operation. Listeners interested in long-context inference infrastructure, memory-efficient transformers, or the engineering constraints shaping modern LLM deployment will find the technical depth and comparative framing useful.

Sources:

1. Fast KV Compaction via Attention Matching — Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim, 2026

http://arxiv.org/abs/2602.16284

2. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

3. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianhao Guo, Patrick Lewis, Deming Chen, 2024

https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation

4. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks

5. KVMerger: KV Cache Merging for Memory-Efficient LLMs Inference — Cangqing Wang, Yuhang Yang, Liangzhen Li, Lanqing Hong, Shuo Jiang, Hui Xu, Wei Tao, 2024

https://scholar.google.com/scholar?q=KVMerger%3A+KV+Cache+Merging+for+Memory-Efficient+LLMs+Inference

6. Cartridges: Lightweight, Pluggable Contexts for Language Models — Sabri Eyuboglu, Avanika Narayan, Tao Long, Andrew Liang, Kush Bhatia, Michael Zhang, Neel Guha, James Zou, Christopher Re, Atri Rudra, 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight%2C+Pluggable+Contexts+for+Language+Models

7. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021

https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation

8. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023

https://scholar.google.com/scholar?q=Learning+to+Compress+Prompts+with+Gist+Tokens

9. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI (Zhipeng Liu, Chengqi Deng, et al.), 2024

https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model

10. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022

https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers

11. SparseGPT: Massive Language Models Can be Accurately Pruned in One Shot — Elias Frantar, Dan Alistarh, 2023

https://scholar.google.com/scholar?q=SparseGPT%3A+Massive+Language+Models+Can+be+Accurately+Pruned+in+One+Shot

12. Optimal Brain Surgeon and General Network Pruning — Babak Hassibi, David G. Stork, 1993

https://scholar.google.com/scholar?q=Optimal+Brain+Surgeon+and+General+Network+Pruning

13. LASER: The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction — Pratyusha Sharma, Jordan T. Ash, Dipendra Misra, 2023

https://scholar.google.com/scholar?q=LASER%3A+The+Truth+is+in+There%3A+Improving+Reasoning+in+Language+Models+with+Layer-Selective+Rank+Reduction

14. Cartridges: Learned KV Cache Compression for Long-Context Language Model Inference — Eyuboglu et al., 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Learned+KV+Cache+Compression+for+Long-Context+Language+Model+Inference

15. The Power of Scale for Parameter-Efficient Prompt Tuning — Lester et al., 2021

https://scholar.google.com/scholar?q=The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning

16. MagicPIG: LSH Sampling for Efficient LLM Generation — Chen et al., 2024

https://scholar.google.com/scholar?q=MagicPIG%3A+LSH+Sampling+for+Efficient+LLM+Generation

17. KV-Distill: Nearly Lossless Learnable Context Compression for LLMs — approximate (multiple authors), 2024-2025

https://scholar.google.com/scholar?q=KV-Distill%3A+Nearly+Lossless+Learnable+Context+Compression+for+LLMs

18. Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection — approximate (multiple authors), 2024-2025

https://scholar.google.com/scholar?q=Thin+Keys%2C+Full+Values%3A+Reducing+KV+Cache+via+Low-Dimensional+Attention+Selection

19. A Preliminary Study on the Promises and Challenges of Native Top-Sparse Attention — approximate (multiple authors), 2024-2025

https://scholar.google.com/scholar?q=A+Preliminary+Study+on+the+Promises+and+Challenges+of+Native+Top-Sparse+Attention

20. Beyond KV Caching: Shared Attention for Efficient LLMs — approximate (multiple authors), 2024-2025

https://scholar.google.com/scholar?q=Beyond+KV+Caching%3A+Shared+Attention+for+Efficient+LLMs

21. Compressing Many-Shots in In-Context Learning — approximate (multiple authors), 2024-2025

https://scholar.google.com/scholar?q=Compressing+Many-Shots+in+In-Context+Learning

22. AI Post Transformers: Hyper-Scaling LLM Inference with KV Cache Compression — Hal Turing & Dr. Ada Shannon, Fri,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Hyper-Scaling-LLM-Inference-with-KV-Cache-Compression-e3aalcq

23. AI Post Transformers: ShadowKV: High-Throughput Long-Context LLM Inference — Hal Turing & Dr. Ada Shannon, Wed,

https://podcasters.spotify.com/pod/show/12146088098/episodes/ShadowKV-High-Throughput-Long-Context-LLM-Inference-e38bn17

24. AI Post Transformers: Quest: Query-Aware Sparsity for Efficient LLM Inference — Hal Turing & Dr. Ada Shannon, Fri,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Quest-Query-Aware-Sparsity-for-Efficient-LLM-Inference-e3aat91

25. AI Post Transformers: Long context: Dichotomy of Findings & Status of Research — Hal Turing & Dr. Ada Shannon, Wed,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Long-context-Dichotomy-of-Findings--Status-of-Research-e3eat7c

26. AI Post Transformers: NVIDIA: TTT-E2E: Unlocking Long-Context Learning via End-to-End Test-Time Training — Hal Turing & Dr. Ada Shannon, Sat,

https://podcasters.spotify.com/pod/show/12146088098/episodes/NVIDIA-TTT-E2E-Unlocking-Long-Context-Learning-via-End-to-End-Test-Time-Training-e3dq389

Interactive Visualization: 50x KV Cache Compression in Seconds via Attention Matching

This episode examines the nabla-Reasoner paper (ICLR 2026), which proposes running gradient descent on token logits during inference — a first-order approach to test-time compute scaling that stands apart from every existing method in the field. The hosts contextualize the work against the established zeroth-order inference-time scaling landscape: Chain-of-Thought, Self-Consistency, Tree of Thoughts, and MCTS-based methods, all of which probe the reward landscape by sampling without directional information. The core argument is that zeroth-order methods hit a hard ceiling on long-horizon reasoning tasks because the search space grows exponentially while reward signals remain sparse, making random sampling increasingly futile. nabla-Reasoner sidesteps this by treating token logit vectors — normally ephemeral intermediate computations — as continuous optimization variables, computing reward gradients with respect to them and nudging the distribution toward higher-reward outputs before committing to each token. Listeners interested in the mechanics of inference-time scaling and the theoretical limits of sampling-based reasoning will find this a technically dense, well-grounded discussion of a genuinely novel approach.

Sources:

1. $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space — Peihao Wang, Ruisi Cai, Zhen Wang, Hongyuan Mei, Qiang Liu, Pan Li, Zhangyang Wang, 2026

http://arxiv.org/abs/2603.04948v1

2. ∇-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space — Peihao Wang, Ruisi Cai, Zhen Wang, Hongyuan Mei, Qiang Liu, Pan Li, Zhangyang Wang, 2026

https://scholar.google.com/scholar?q=%E2%88%87-Reasoner%3A+LLM+Reasoning+via+Test-Time+Gradient+Descent+in+Latent+Space

3. Diffusion-LM Improves Controllable Text Generation — Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori Hashimoto, 2022

https://scholar.google.com/scholar?q=Diffusion-LM+Improves+Controllable+Text+Generation

4. GFlowNet-Guided LLM Decoding: Towards Diverse and Accurate Reasoning — Jianing Li et al., 2024

https://scholar.google.com/scholar?q=GFlowNet-Guided+LLM+Decoding%3A+Towards+Diverse+and+Accurate+Reasoning

5. Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs — Ahmadianshalchi et al., 2024

https://scholar.google.com/scholar?q=Back+to+Basics%3A+Revisiting+REINFORCE-Style+Optimization+for+Learning+from+Human+Feedback+in+LLMs

6. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou, 2022

https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models

7. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan, 2023

https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models

8. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2023

https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step

9. Scaling LLM Test-Time Compute Optimally — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024

https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally

10. ARGS: Alignment as Reward-Guided Search — Maxim Khanov, Jirayu Burapacheep, Yixuan Li, 2024

https://scholar.google.com/scholar?q=ARGS%3A+Alignment+as+Reward-Guided+Search

11. Controlled Decoding from Language Models — Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanpin Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, Ahmad Beirami, 2024

https://scholar.google.com/scholar?q=Controlled+Decoding+from+Language+Models

12. AlphaCode 2 Technical Report — Google DeepMind AlphaCode Team, 2023

https://scholar.google.com/scholar?q=AlphaCode+2+Technical+Report

13. Training language models to follow instructions with human feedback — Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe, 2022

https://scholar.google.com/scholar?q=Training+language+models+to+follow+instructions+with+human+feedback

14. Direct Preference Optimization: Your Language Model is Secretly a Reward Model — Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, 2023

https://scholar.google.com/scholar?q=Direct+Preference+Optimization%3A+Your+Language+Model+is+Secretly+a+Reward+Model

15. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI (Daya Guo, Dejian Yang, Haowei Zhang, et al.), 2025

https://scholar.google.com/scholar?q=DeepSeek-R1%3A+Incentivizing+Reasoning+Capability+in+LLMs+via+Reinforcement+Learning

16. Learning to summarize from human feedback — Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano, 2020

https://scholar.google.com/scholar?q=Learning+to+summarize+from+human+feedback

17. Plug and Play Language Models: A Simple Approach to Controlled Text Generation — Dathathri et al., 2020

https://scholar.google.com/scholar?q=Plug+and+Play+Language+Models%3A+A+Simple+Approach+to+Controlled+Text+Generation

18. FUDGE: Controlled Text Generation with Future Discriminators — Yang and Klein, 2021

https://scholar.google.com/scholar?q=FUDGE%3A+Controlled+Text+Generation+with+Future+Discriminators

19. Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters — Snell et al., 2024

https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+Can+be+More+Effective+than+Scaling+Model+Parameters

20. Alignment as Reward-Guided Search — Khanov et al., 2024

https://scholar.google.com/scholar?q=Alignment+as+Reward-Guided+Search

21. Soft Prompts: The Power of Scale for Parameter-Efficient Prompt Tuning — Lester et al., 2021

https://scholar.google.com/scholar?q=Soft+Prompts%3A+The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning

22. The Generalization Gap in Offline Reinforcement Learning — Levine et al., 2020

https://scholar.google.com/scholar?q=The+Generalization+Gap+in+Offline+Reinforcement+Learning

23. Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization — Unknown (2025), 2025

https://scholar.google.com/scholar?q=Thinking+on+the+Fly%3A+Test-Time+Reasoning+Enhancement+via+Latent+Thought+Policy+Optimization

24. Logit arithmetic elicits long reasoning capabilities without training — Unknown (2025), 2025

https://scholar.google.com/scholar?q=Logit+arithmetic+elicits+long+reasoning+capabilities+without+training

25. Reinforcement Learning in Inference Time: A Perspective from Successive Policy Iterations — Unknown (2025), 2025

https://scholar.google.com/scholar?q=Reinforcement+Learning+in+Inference+Time%3A+A+Perspective+from+Successive+Policy+Iterations

26. GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning — Yang et al. (2024/2025), 2025

https://scholar.google.com/scholar?q=GenPRM%3A+Scaling+Test-Time+Compute+of+Process+Reward+Models+via+Generative+Reasoning

27. Process Reward Models That Think — Unknown (2025), 2025

https://scholar.google.com/scholar?q=Process+Reward+Models+That+Think

28. Efficient Adaptive Rejection Sampling for Accelerating Speculative Decoding in Large Language Models — Unknown (2025), 2025

https://scholar.google.com/scholar?q=Efficient+Adaptive+Rejection+Sampling+for+Accelerating+Speculative+Decoding+in+Large+Language+Models

29. Inference-time alignment control for diffusion models with reinforcement learning guidance — Unknown (2025), 2025

https://scholar.google.com/scholar?q=Inference-time+alignment+control+for+diffusion+models+with+reinforcement+learning+guidance

30. AI Post Transformers: Test-Time Reinforcement Learning for LLMs — Hal Turing & Dr. Ada Shannon, Wed,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Test-Time-Reinforcement-Learning-for-LLMs-e398hsk

31. AI Post Transformers: MetaScale: Test-Time Scaling with Evolving Meta-Thoughts — Hal Turing & Dr. Ada Shannon, Fri,

https://podcasters.spotify.com/pod/show/12146088098/episodes/MetaScale-Test-Time-Scaling-with-Evolving-Meta-Thoughts-e36kgn7

32. AI Post Transformers: Process Reward Learning for LLM Reasoning Optimization — Hal Turing & Dr. Ada Shannon, Mon,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Process-Reward-Learning-for-LLM-Reasoning-Optimization-e3dsuav

33. AI Post Transformers: Tree-based Group Policy Optimization for LLM Agents — Hal Turing & Dr. Ada Shannon, Fri,

https://podcasters.spotify.com/pod/show/12146088098/episodes/Tree-based-Group-Policy-Optimization-for-LLM-Agents-e38obfb

34. AI Post Transformers: MASA: Meta-Awareness via Self-Alignment Reinforcement Learning — Hal Turing & Dr. Ada Shannon, Sun,

https://podcasters.spotify.com/pod/show/12146088098/episodes/MASA-Meta-Awareness-via-Self-Alignment-Reinforcement-Learning-e3a2of7

Interactive Visualization: Gradient Descent at Inference Time for LLM Reasoning