AI Post Transformers

This episode examines APEX, a persistent-memory learned index from researchers at CUHK, MIT, Microsoft Research, and Simon Fraser University, presented at VLDB 2022. It traces the collision of two research threads — Intel's Optane persistent memory, which sits on the DDR bus but requires manual cache-line flushing and fencing to guarantee crash durability, and "learned indexes," which replace B-trees with lightweight regression models trained to predict a key's position. The discussion covers why naive approaches fail: running the learned index ALEX directly on persistent memory offers speed but zero crash safety, while wrapping it in standard transactional logging (PMDK) restores consistency at the cost of saturating scarce write bandwidth. It builds toward APEX's core innovation, "probe-and-stash," a technique designed to avoid the record-shifting that makes both alternatives fragile, enabling claimed 15x faster inserts and roughly 42-millisecond crash recovery. Listeners interested in database internals, memory hierarchies, or the practical gap between promising ML systems research and production-ready engineering will find the trade-offs — and the hosts' friendly disagreement over why learned indexes haven't seen wider industry adoption — a compelling entry point into the topic.

Sources:

1. APEX: A High-Performance Learned Index on Persistent Memory — Baotong Lu, Jialin Ding, Eric Lo, Umar Farooq Minhas, Tianzheng Wang, 2021

http://arxiv.org/abs/2105.00683

2. The Case for Learned Index Structures — Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis, 2018

https://scholar.google.com/scholar?q=The+Case+for+Learned+Index+Structures

3. ALEX: An Updatable Adaptive Learned Index — Jialin Ding, Umar Farooq Minhas, Jia Yu, Chi Zhang, Yinan Li, Chi Wang, Badrish Chandramouli, Johannes Gehrke, Donald Kossmann, David Lomet, 2020

https://scholar.google.com/scholar?q=ALEX%3A+An+Updatable+Adaptive+Learned+Index

4. Let's Talk About Storage & Recovery Methods for Non-Volatile Memory Database Systems — Joy Arulraj, Andrew Pavlo, Subramanya R. Dulloor, 2015

https://scholar.google.com/scholar?q=Let%27s+Talk+About+Storage+%26+Recovery+Methods+for+Non-Volatile+Memory+Database+Systems

5. An Empirical Guide to the Behavior and Use of Scalable Persistent Memory — Jian Yang, Juno Kim, Morteza Hoseinzadeh, Joseph Izraelevitz, Steve Swanson, 2020

https://scholar.google.com/scholar?q=An+Empirical+Guide+to+the+Behavior+and+Use+of+Scalable+Persistent+Memory

6. How Does Updatable Learned Index Perform on Non-Volatile Main Memory? — Leying Chen, Shimin Chen, 2021

https://scholar.google.com/scholar?q=How+Does+Updatable+Learned+Index+Perform+on+Non-Volatile+Main+Memory%3F

7. Dash: Scalable Hashing on Persistent Memory — Baotong Lu, Xiangpeng Hao, Tianzheng Wang, Eric Lo, 2020

https://scholar.google.com/scholar?q=Dash%3A+Scalable+Hashing+on+Persistent+Memory

8. FPTree: A Hybrid SCM-DRAM Persistent and Concurrent B-Tree for Storage Class Memory — Ismail Oukid, Johan Lasperas, Anisoara Nica, Thomas Willhalm, Wolfgang Lehner, 2016

https://scholar.google.com/scholar?q=FPTree%3A+A+Hybrid+SCM-DRAM+Persistent+and+Concurrent+B-Tree+for+Storage+Class+Memory

9. RECIPE: Converting Concurrent DRAM Indexes to Persistent-Memory Indexes — Se Kwon Lee, Jayashree Mohan, Sanidhya Kashyap, Taesoo Kim, Vijay Chidambaram, 2019

https://scholar.google.com/scholar?q=RECIPE%3A+Converting+Concurrent+DRAM+Indexes+to+Persistent-Memory+Indexes

Interactive Visualization: APEX: A High-Performance Learned Index on Persistent Memory

This episode unpacks ALEX, a 2020 paper from MIT, Microsoft Research, Arizona State, and Georgia Tech that reworks the "learned index" idea for real-world databases. It traces the lineage from Kraska et al.'s 2018 Learned Index, which used a hierarchy of regression models to predict a key's position in a sorted array but only worked on static, read-only data since any insert would break the model's predictions. The discussion explains how ALEX solves this with a Gapped Array that leaves deliberate empty slots for near-free inserts, plus a tree structure whose nodes can grow, shrink, split, or retrain their local models on the fly, letting it support inserts, updates, and deletes alongside lookups. The headline numbers — up to 4.1x the throughput of a B+Tree with an index up to 2000 times smaller — anchor a broader explanation of why B+Trees have dominated databases since the 1970s and what it takes for a learned alternative to finally handle OLTP-style workloads. Listeners interested in database internals or machine learning applied to systems problems will get a clear, building-block explanation of both the classic B+Tree and the model-based alternative trying to replace it.

Sources:

1. ALEX: An Updatable Adaptive Learned Index — Jialin Ding, Umar Farooq Minhas, Jia Yu, Chi Wang, Jaeyoung Do, Yinan Li, Hantian Zhang, Badrish Chandramouli, Johannes Gehrke, Donald Kossmann, David Lomet, Tim Kraska, 2019

http://arxiv.org/abs/1905.08898

2. The Case for Learned Index Structures — Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis, 2018

https://scholar.google.com/scholar?q=The+Case+for+Learned+Index+Structures

3. FITing-Tree: A Data-aware Index Structure — Alex Galakatos, Michael Markovitch, Carsten Binnig, Rodrigo Fonseca, Tim Kraska, 2019

https://scholar.google.com/scholar?q=FITing-Tree%3A+A+Data-aware+Index+Structure

4. The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds — Paolo Ferragina, Giorgio Vinciguerra, 2020

https://scholar.google.com/scholar?q=The+PGM-index%3A+a+fully-dynamic+compressed+learned+index+with+provable+worst-case+bounds

5. Benchmarking Learned Indexes — Ryan Marcus, Andreas Kipf, Alexander van Renen, Mihail Stoian, Sanchit Misra, Alfons Kemper, Thomas Neumann, Tim Kraska, 2020

https://scholar.google.com/scholar?q=Benchmarking+Learned+Indexes

6. A Sparse Table Implementation of Priority Queues — Alon Itai, Alan G. Konheim, Michael Rodeh, 1981

https://scholar.google.com/scholar?q=A+Sparse+Table+Implementation+of+Priority+Queues

7. Cache-Oblivious Streaming B-trees — Michael A. Bender, Martin Farach-Colton, Jeremy T. Fineman, Yonatan R. Fogel, Bradley C. Kuszmaul, Jelani Nelson, 2007

https://scholar.google.com/scholar?q=Cache-Oblivious+Streaming+B-trees

8. Hekaton: SQL Server's Memory-Optimized OLTP Engine — Cristian Diaconu, Craig Freedman, Erik Ismert, Per-Åke Larson, Pravin Mittal, Ryan Stonecipher, Nitin Verma, Mike Zwilling, 2013

https://scholar.google.com/scholar?q=Hekaton%3A+SQL+Server%27s+Memory-Optimized+OLTP+Engine

9. Cache Craftiness for Fast Multicore Key-Value Storage (Masstree) — Yandong Mao, Eddie Kohler, Robert Morris, 2012

https://scholar.google.com/scholar?q=Cache+Craftiness+for+Fast+Multicore+Key-Value+Storage+%28Masstree%29

10. The Adaptive Radix Tree: ARTful Indexing for Main-Memory Databases — Viktor Leis, Alfons Kemper, Thomas Neumann, 2013

https://scholar.google.com/scholar?q=The+Adaptive+Radix+Tree%3A+ARTful+Indexing+for+Main-Memory+Databases

11. Speedy Transactions in Multicore In-Memory Databases (Silo) — Stephen Tu, Wenting Zheng, Eddie Kohler, Barbara Liskov, Samuel Madden, 2013

https://scholar.google.com/scholar?q=Speedy+Transactions+in+Multicore+In-Memory+Databases+%28Silo%29

12. An Adaptive Packed-Memory Array — Michael A. Bender, Haodong Hu, 2007

https://scholar.google.com/scholar?q=An+Adaptive+Packed-Memory+Array

13. SageDB: A Learned Database System — Tim Kraska, Mohammad Alizadeh, Alex Beutel, et al., 2019

https://scholar.google.com/scholar?q=SageDB%3A+A+Learned+Database+System

Interactive Visualization: ALEX: Building an Updatable Learned Index for Databases

This episode examines RewardingDoubt, a reinforcement-learning method for training large language models to express calibrated confidence rather than defaulting to near-maximal certainty regardless of correctness. The discussion covers why miscalibrated confidence is dangerous in high-stakes domains like medicine, legal consultation, and customer service, where deferring to a human reviewer only works if the model's stated confidence is trustworthy. It walks through the technical core of the approach: a proper scoring rule (specifically the logarithmic scoring rule) that mathematically rewards models for reporting their true beliefs rather than gaming their confidence scores, framed intuitively as a betting game where lying about certainty costs real "money." The conversation contrasts this RL-based approach with prior black-box (output-only) and white-box (internals-probing) calibration methods, including Kadavath et al.'s self-evaluation technique and Lin, Hilton, and Evans' supervised fine-tuning approach, arguing RL sidesteps the quality ceiling imposed by fixed ground-truth labels. It also details the paper's MDP formulation, where the model's answer is frozen before a separate PPO-trained pass generates the confidence sequence, and explains the two evaluation metrics—Expected Calibration Error and AUROC—used to judge whether the method actually works.

Sources:

1. Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models — David Bani-Harouni, Chantal Pellegrini, Paul Stangel, Ege Özsoy, Kamilia Zaripova, Nassir Navab, Matthias Keicher, 2025

http://arxiv.org/abs/2503.02623v6

2. Language Models (Mostly) Know What They Know — Kadavath, Conerly, Askell, Henighan, Drain, Perez, Schiefer, Hatfield-Dodds, DasSarma, Tran-Johnson, et al. (Anthropic), 2022

https://scholar.google.com/scholar?q=Language+Models+%28Mostly%29+Know+What+They+Know

3. Teaching Models to Express Their Uncertainty in Words — Stephanie Lin, Jacob Hilton, Owain Evans, 2022

https://scholar.google.com/scholar?q=Teaching+Models+to+Express+Their+Uncertainty+in+Words

4. LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Models — Elias Stengel-Eskin, Peter Hase, Mohit Bansal, 2024

https://scholar.google.com/scholar?q=LACIE%3A+Listener-Aware+Finetuning+for+Confidence+Calibration+in+Large+Language+Models

5. Taming Overconfidence in LLMs: Reward Calibration in RLHF — Jixuan Leng, Chengsong Huang, Banghua Zhu, Jiaxin Huang, 2024

https://scholar.google.com/scholar?q=Taming+Overconfidence+in+LLMs%3A+Reward+Calibration+in+RLHF

6. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs — Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, Bryan Hooi, 2024

https://scholar.google.com/scholar?q=Can+LLMs+Express+Their+Uncertainty%3F+An+Empirical+Evaluation+of+Confidence+Elicitation+in+LLMs

7. Calibration-Tuning: Teaching Large Language Models to Know What They Don't Know — Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, Andrew Wilson, 2024

https://scholar.google.com/scholar?q=Calibration-Tuning%3A+Teaching+Large+Language+Models+to+Know+What+They+Don%27t+Know

Interactive Visualization: Teaching Language Models to Bet Honestly on Their Own Answers

This episode explores Simula, a system from EPFL and Google DeepMind researchers that generates entire specialized training datasets from scratch, with no seed examples required. It breaks down the reasoning-driven, multi-role agentic pipeline — where separate model roles propose, critique, sample, generate, and filter data — contrasting it with older approaches like Self-Instruct's hand-written templates and Promptbreeder's opaque evolutionary search. The discussion covers the QDC framework (quality, diversity, complexity) used to define "good" synthetic data, and how Simula builds an explicit taxonomy tree that serves as both the sampling mechanism and an auditable record of why each data point exists, echoing the transparency goals of Datasheets for Datasets. The hosts also flag a key tension worth scrutinizing: the system relies on the model to grade its own taxonomy quality, critique its own outputs, and judge complexity — assumptions that deserve real pushback rather than blind trust. Listeners interested in synthetic data generation, dataset auditability, or the mechanics of multi-stage agentic pipelines will find the walkthrough of Simula's three-stage architecture a concrete look at how reasoning-first systems aim to replace scarce human annotation.

Sources:

1. Reasoning-Driven Synthetic Data Generation and Evaluation — Tim R. Davidson, Benoit Seguin, Enrico Bacis, Cesar Ilharco, Hamza Harkous, 2026

http://arxiv.org/abs/2603.29791v1

2. Surveying the effects of quality, diversity, and complexity in synthetic data from large language models — Havrilla, Dai, O'Mahony, Oostermeijer, Zisler, Albalak, Milo, Raparthy, Gandhi, Abbasi, et al., 2024

https://scholar.google.com/scholar?q=Surveying+the+effects+of+quality%2C+diversity%2C+and+complexity+in+synthetic+data+from+large+language+models

3. LLM evaluators recognize and favor their own generations — Panickssery, Bowman, Feng, 2024

https://scholar.google.com/scholar?q=LLM+evaluators+recognize+and+favor+their+own+generations

4. Position: will we run out of data? Limits of LLM scaling based on human-generated data — Villalobos, Ho, Sevilla, Besiroglu, Heim, Hobbhahn, 2024

https://scholar.google.com/scholar?q=Position%3A+will+we+run+out+of+data%3F+Limits+of+LLM+scaling+based+on+human-generated+data

5. Scaling data-constrained language models — Muennighoff, Rush, Barak, Le Scao, Tazi, Piktus, Pyysalo, Wolf, Raffel, 2023

https://scholar.google.com/scholar?q=Scaling+data-constrained+language+models

6. Lora without regret — Schulman, Thinking Machines Lab, 2025

https://scholar.google.com/scholar?q=Lora+without+regret

7. Self-recognition in language models — Davidson, Surkov, Veselovsky, Russo, West, Gulcehre, 2024

https://scholar.google.com/scholar?q=Self-recognition+in+language+models

Interactive Visualization: Zero-Seed Data Generation: Simula's Reasoning-Driven Pipeline

This episode breaks down DeepSeek-V4.1-Flash, a 552-billion-parameter Mixture-of-Experts model whose KV cache footprint has shrunk roughly 437-fold since DeepSeek's original V1, including a 4x drop in this single generational jump from V4-Flash. The discussion explains why long-context, tool-using agents make the KV cache the real serving bottleneck, and how the model's architecture attacks it from multiple angles: a Causal Encoder-Decoder design that nearly halves prefill compute by having upper layers project keys and values from a midpoint boundary rather than computing their own, Sliding Window Attention for bounded recent-context memory, and CSA2 (Compressed Sparse Attention 2), which layers entry-size compression, sequence compression, and cross-layer sharing on top of FP4 quantization to reach roughly 890 bytes of cache per token. It also flags the gap between DeepSeek's architectural and precision claims and actual measured deployment gains, previewing a closer look at what these compression tricks mean once put into real serving conditions. Listeners interested in efficient LLM inference, MoE architectures, or the mechanics of attention and caching will find concrete, mechanism-level explanations rather than surface-level hype.

Sources:

1. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression — DeepSeek-AI, :, Anyi Xu, B. Li, Bangcai Lin, Bing Xue, BingCheng Xian, Bingzheng Xu, Bochao Wu, Bowei Zhang, Boyi Deng, C. C. Yu, Chao Jin, Chaofan Lin, Chen Dong, Chenbing Wang, Chenfan Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chengyuan Zhang, Chenhao Xu, Chenqi Zhao, Chenze Shao, Chuhao Wang, Chuqi Zhang, Damai Dai, Dejian Yang, Deli Chen, Di Huang, Di Wu, Donghao Li, Erhang Li, Eric Fu, F. Zhou, Fangwei Zhou, Fangyun Lin, Fangzhou Yuan, Feiyu Xia, Fucong Dai, Guangbo Hao, Guanglin Li, Guanting Chen, Guoai Cao, Guofan Fan, Guolai Meng, Guowei Li, Haichuan Zhang, Haiyang Ma, Haiyang Shen, Han Li, Han Yu, Han Zhang, Hangyuan Deng, Hanwei Xu, Hanxiang Xu, Hanxun Zhong, Hao Guo, Hao Jiang, Hao Li, Hao Qin, Haodong Wen, Haofen Liang, Haofeng Huang, Haohua Liu, Haoling Zhang, Haoming Luo, Haoran Yang, Haotian Xu, Haotian Yuan, Haoting Huang, Haowen Luo, Haoyang Cai, Haoyu Chen, Haozhe Ji, Hengran Zhang, Hengrui Wang, Hengxu Wu, Honghui Ding, Hongxuan Tang, Huadong Wang, Huanqi Cao, Huazuo Gao, Hui Qu, Hui Zeng, J. Yang, J. H. Jin, J. H. Zhang, J. X. Zou, Jia Yu, Jiahui Zhou, Jiajun Chen, Jialiang Huang, Jialin Zhao, Jiamin Tang, Jian Zhou, Jianan Tong, Jianwen Li, Jiaqi Zhu, Jiarui Wang, Jiasheng Ye, Jiashi Li, Jiaxin Xu, Jiaying Ding, Jibai Lu, Jiewen Hu, Jin Yan, Jincheng Zhai, Jingchang Chen, Jingcheng Hu, Jingli Zhou, Jingsheng Xu, Jingting Xiang, Jingyan Yun, Jingyang Yuan, Jingyuan Cheng, Jinhua Zhu, Jinpeng Wang, Jinyi Chen, Jinyi Hu, Jiping Yu, Jueliang Guo, Junbo Pei, Junbo Sun, Junguang Jiang, Junjie Qiu, Junkang Zhou, Junqi Liu, Junren Li, Junxian Li, Junxiao Song, Junyi Guo, Kai Dong, Kaifeng Chen, Kaige Gao, Kang Guan, Kangdong Yuan, Ke Hong, Ke Xu, Kefan Zhao, Kexin Ji, Kexin Zhang, Kexing Zhou, Kuai Yu, Lan Zhang, Lean Wang, Lecong Zhang, Lei Wang, Letian Gao, Liang Zhao, Liansheng Xu, Lihua Guo, Lingxiao Luo, Lingyue Fu, Litao Deng, Litong Wang, Liyue Zhang, Longhao Chen, Lu Chen, Luotian Huang, Luyao Ma, Luyao Wang, M. S. Di, Max Mei, Menghao Ye, Miao Cui, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingjing Zhang, Mingqi Wei, Mingshu Chen, Mingxing Liu, Mingxu Zhou, Mingyu Xu, Mingyu Yang, Mingze Wang, Muyang Chen, Ni Shentu, Ning Wang, Niufang Ning, Panpan Huang, Peixin Cong, Peiyi Wang, Peiyuan Xin, Pengfei Ren, Pengfei Yan, Pengle Zhang, Qi Kang, Qi Tang, Qiancheng Wang, Qiang Li, Qihao Zhu, Qingyang Li, Qinyu Chen, Qiushi Du, Qizhou Guo, Rongxian Xu, Rui Ding, Rui Hu, Rui Tian, Rui Yu, Ruidong Zhu, Ruifan Xu, Ruihan Yang, Ruihang Xia, Ruijie Lu, Ruilin Geng, Ruipeng Hong, Ruiqi Ge, Ruisong Zhang, Ruize Sun, Ruizhe Pan, Runji Wang, Runqian Chen, Runxin Xu, Ruohong Tian, Ruomeng Shen, Ruoyu Zhang, Ryan X., S. H. Liu, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaofei Cai, Shaoheng Nie, Shaoyuan Chen, Shengding Hu, Shengkai Lin, Shengwen Ran, Shengyu Liu, Shengyuan Jia, Shi Bai, Shi Feng, Shicheng Xu, Shichun Liu, Shiqiang Hu, Shirong Ma, Shiyu Wang, Shiyuan Feng, Shufan Gong, Shuhan Lin, Shuiping Yu, Shunfeng Zhou, Shuo Yang, Shuomeng Wang, Shuting Guo, Shuting Pan, Shuying Yu, Sinuo Cao, Siyi Lin, Sizhe Chen, Songyang Chen, Songyang Zhou, Tao Ni, Tao Yun, Tian Jin, Tian Pei, Tian Ye, Tianle Lin, Tianran Ji, Tianyi Cui, Tianyuan Yue, Tingting Yu, Tongrui Xiong, Wangding Zeng, Wei Liu, Wei Zhang, Weibin Xu, Weihao Zeng, Weilin Zhao, Wen Liu, Wenfeng Liang, Wenjie Pang, Wenjing Luo, Wenjing Yao, Wenjun Gao, Wenkai Shao, Wenkai Yang, Wenli Zhang, Wenlu Wang, Wenlve Huang, Wenqian Yan, Wentao Zhang, Xi Gao, Xiang He, Xiang Li, Xiangli Li, Xiangwen Wang, Xiangying Zhang, Xiankui Wei, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojian Qu, Xiaokang Chen, Xiaokang Zhang, Xiaotao Nie, Xiaoyao Zou, Xiaoyuan Li, Xicheng Guo, Xieting Chu, Xin Cheng, Xin Liu, Xin Xie, Xinbo Xu, Xingchao Liu, Xingchen Liu, Xingkai Yu, Xingyou Li, Xintong Yao, Xinyang Chen, Xinyong Jiang, Xinyu Yang, Xinyu Yang, Xu Chen, Xuanyu Wang, Xubei Zhong, Xuecheng Su, Xuejie Liu, Xuheng Lin, Xujie Fan, Xuncheng Zhao, Xuwei Fu, Y. C. Yan, Y. H. Jiang, Y. T. Wu, Y. W. M., Y. Z. Wang, Yafei Gao, Yang Yang, Yang Zhang, Yanru Ma, Yanwen Huang, Yao Li, Yao Li, Yao Meng, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yaoyang Ye, Yehang Yin, Yexinrui Wu, Yi Qian, Yi Tao, Yi Yu, Yichao Zhang, Yichen Jiang, Yicheng Wang, Yifan Ding, Yifan Shi, Yifeng Peng, Yifeng Zhai, Yijia Wu, Yiliang Xiong, Yilun Wang, Ying He, Ying Zhou, Yingjia Luo, Yinmin Zhong, Yiping Wang, Yisong Wang, Yixiang Zhang, Yixiao Chen, Yixuan Tan, Yixuan Wei, Yiyang Ma, Yiyao Yang, Yiyuan Liu, Yizai Cai, Yizhen Wei, Yizhi Wang, Yonglun Yang, Yongqi Zhuo, Yongqiang Guo, Yongtong Wu, Yu Wu, Yu Zhang, Yuan Bian, Yuan Cheng, Yuan Ou, Yuan Sun, Yuanfan Xu, Yuanhang Sun, Yuanhao Li, Yuchen Liu, Yuchen Yao, Yudong Han, Yuduan Wang, Yuhan Wu, Yuhao Meng, Yuheng Zou, YuKun Li, Yunchuan Wang, Yunfan Xiao, Yunfan Xiong, Yupeng Chen, Yuqian Cao, Yuqian Wang, Yuqing Chen, Yushun Zhang, Yutong Lin, Yuwei Xiao, Yuxian Gu, Yuxiang Chen, Yuxiang Huang, Yuxiang Luo, Yuxiang You, Yuxin Chen, Yuxin Xiang, Yuxuan Liu, Yuxuan Zhou, Yuyang Zhou, Yuzhe Guo, Yuzhen Huang, Yuzhuo Bai, Z. Y. Z., Zanlin Ni, Zehao Wang, Zehua Zhao, Zehui Ren, Zejun Zhao, Zhangli Sha, Zhanying Wang, Zhaochen Zhang, Zhaoshuai Du, Zhe Fu, Zhean Xu, Zhenda Xie, Zheng Liu, Zhengyan Zhang, Zhenhua Dong, Zhewen Hao, Zhibang Wang, Zhibin Gou, Zhicheng Ma, Zhihao Li, Zhihong Shao, Zhihuan Huang, Zhijie Li, Zhirui Lu, Zhixian Huang, Zhixuan Chen, Zhixuan Chen, Zhixuan Pan, Zhiyu Wu, Zhizhou Ren, Zhu He, Zhuoshu Li, Zhuping Zhang, Zian Xu, Zihao Wang, Zihui Gu, Zijia Zhu, Zili Zhang, Zilin Li, Zilong Hou, Zilong Lyu, Ziqiao Wang, Ziwei Xie, Ziya Zhang, Ziyi Gao, Zizheng Pan, Zonglin Li, Zongqing Yao, Zui Chen, Zuofan Wu, Chenchen Ling, Chengyu Hou, Chong Chen, D. Li, Di Qi, Dongjie Ji, Fang Wei, Fanyi Xia, Fei Xie, Feiyi Tan, Hailong Guo, Haiyan Zhai, Hui Zhou, Huihui Tan, Huijie Li, Jia Luo, Jia Song, Jialu Cai, Jian Liang, Jiangting Zhou, Jiaqi Gao, Jiayi Shao, Jie Chen, Jieyu Yang, Jin Chen, Jingde Zhang, Jingzi Zhou, Jinqian Wang, Jinyang Liu, JinZhao Sun, Junhua Ling, Junmin Zheng, Kaicheng Yang, Ke Xu, Le Su, Leyi Xia, Liangfeng Ding, Lin Zhuo, Linwang Ma, Linyan Zhu, Liyu Cai, Luqi Yao, M. K. Zhang, Meng Li, Miao Lin, Miaojun Wang, Min Zhang, Mingming Li, Mingming Wang, Mingze Yin, Minmin Han, Nan Cao, Ning Wang, Ningxin Ma, Panpan Wang, Peihan Lin, Peng Sun, Peng Zhang, Qian Ying, Qiang Xiang, Qiao Wang, Qingmiao Mao, Qiwei Jiang, Rongli Jin, Ruyi Chen, Sha Tao, Shangmian Sun, Shaoqing Wu, Shichao Zou, Si Lei, Tianyang Zhang, Tianyu Sun, Tingting Yin, W. L. Xiao, Wei An, Wei Li, Wei Wang, Weiwei Lin, Wenqing Hou, X. Lin, Xiangfei Meng, Xianzhu Huang, Xiao Peng, Xiaoqian Li, Xiaoting Zhang, Xiaowen Sun, Xiaoxiang Wang, Xiaoyu Ye, Xinrou Zhang, Xinyu Zhang, Xue Cao, Xueyin Chen, Yanan Zhou, Yanhong Xu, Yao Xia, Yao Xu, Yi Shao, Yihong Zhang, Yiling Ma, Ying Tang, Yining Lou, Yiru Chen, Yishi Piao, Yixuan Chen, Yong Xiong, Yuchen Xuan, Yuehan Yang, Yuer Xu, Yukun Zha, Yunxian Ma, Yuping Lin, Yuting Yan, Yutong Xie, Yuwen Sheng, Yuxuan Zhu, Zekai Zhang, Zhe Ju, Zhenzhen Lin, Zheren Gao, Zheyang Sun, Zhigang Yan, Zhongyu Wu, Zi Wang, Zihua Qu, Ziling Yan, Ziyi Wan, 2026

http://arxiv.org/abs/2609.19969

2. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, et al. (DeepSeek-AI), 2025

https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention

3. MoBA: Mixture of Block Attention for Long-Context LLMs — Enzhe Lu et al. (Moonshot AI), 2025

https://scholar.google.com/scholar?q=MoBA%3A+Mixture+of+Block+Attention+for+Long-Context+LLMs

4. Longformer: The Long-Document Transformer — Iz Beltagy, Matthew E. Peters, Arman Cohan, 2020

https://scholar.google.com/scholar?q=Longformer%3A+The+Long-Document+Transformer

5. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, et al., 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

6. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, et al., 2017

https://scholar.google.com/scholar?q=Attention+Is+All+You+Need

7. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) — Colin Raffel, Noam Shazeer, Adam Roberts, et al. (Google), 2020

https://scholar.google.com/scholar?q=Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text-to-Text+Transformer+%28T5%29

8. You Only Cache Once: Decoder-Decoder Architectures for Language Models (YOCO) — Yutao Sun, Li Dong, Yi Zhu, et al. (Microsoft Research), 2024

https://scholar.google.com/scholar?q=You+Only+Cache+Once%3A+Decoder-Decoder+Architectures+for+Language+Models+%28YOCO%29

9. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, et al., 2024

https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving

10. You Only Cache Once: Decoder-Decoder Architectures for Language Models — Y. Sun, L. Dong, Y. Zhu, S. Huang, W. Wang, S. Ma, Q. Zhang, J. Wang, F. Wei, 2024

https://scholar.google.com/scholar?q=You+Only+Cache+Once%3A+Decoder-Decoder+Architectures+for+Language+Models

11. IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse — Y. Bai, Q. Dong, T. Jiang, X. Lv, Z. Du, A. Zeng, J. Tang, J. Li, 2026

https://scholar.google.com/scholar?q=IndexCache%3A+Accelerating+Sparse+Attention+via+Cross-Layer+Index+Reuse

12. You Only Index Once: Cross-layer sparse attention with shared routing — Y. Sun, Y. Zhang, L. Dong, J. Wang, F. Wei, 2026

https://scholar.google.com/scholar?q=You+Only+Index+Once%3A+Cross-layer+sparse+attention+with+shared+routing

13. HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing — Y. Gao, J. Wei, Q. Zhang, Y. Cheng, S. Chen, et al., 2026

https://scholar.google.com/scholar?q=HySparse%3A+A+Hybrid+Sparse+Attention+Architecture+with+Oracle+Token+Selection+and+KV+Cache+Sharing

14. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram) — X. Cheng, W. Zeng, D. Dai, Q. Chen, B. Wang, et al., 2026

https://scholar.google.com/scholar?q=Conditional+Memory+via+Scalable+Lookup%3A+A+New+Axis+of+Sparsity+for+Large+Language+Models+%28Engram%29

15. Muon is Scalable for LLM Training — J. Liu, J. Su, X. Yao, Z. Jiang, et al., 2025

https://scholar.google.com/scholar?q=Muon+is+Scalable+for+LLM+Training

Interactive Visualization: DeepSeek-V4.1-Flash: Shrinking KV Cache at 552B Scale

This episode examines "Skip a Layer or Loop It? Learning Program-of-Layers in LLMs" (ICML 2026), which asks whether a frozen pretrained LLM can answer better if each input gets its own sequence of skipped and repeated layers, with no retraining of the base model. The hosts place it against prior work: layer-dropping and early-exit methods built to save compute, recurrent-depth models trained to loop, and studies showing that middle layers in frozen models tolerate skipping, repeating and reordering. They then explain how programs are searched with Monte Carlo Tree Search. That search uses skip or repeat actions on blocks of up to four layers, a binary correct-answer reward, and a penalty on program length. A learned predictor, POLAR, is meant to replace that search at inference. The hosts stress that a "valid program exists" result is judged against the ground-truth label, so it is an oracle coverage number and not an accuracy you can get at inference. They test that reading on four models and DART-Math difficulty levels, asking whether the gains are real or just search luck.

Sources:

1. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs — Ziyue Li, Yang Li, Tianyi Zhou, 2026

http://arxiv.org/abs/2606.06574

2. Transformer Layers as Painters — Qi Sun, Marc Pickett, Aakash Kumar Nain, Llion Jones, 2024

https://scholar.google.com/scholar?q=Transformer+Layers+as+Painters

3. The Remarkable Robustness of LLMs: Stages of Inference? — Vedang Lad, Wes Gurnee, Max Tegmark, 2024

https://scholar.google.com/scholar?q=The+Remarkable+Robustness+of+LLMs%3A+Stages+of+Inference%3F

4. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, et al., 2025

https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach

5. Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence — Sean McLeish et al., 2025

https://scholar.google.com/scholar?q=Teaching+Pretrained+Language+Models+to+Think+Deeper+with+Retrofitted+Recurrence

6. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, Sashank Reddi, 2025

https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts%3A+On+the+Power+of+Looped+Transformers

7. Scaling Latent Reasoning via Looped Language Models (Ouro) — Rui-Jie Zhu et al., 2025

https://scholar.google.com/scholar?q=Scaling+Latent+Reasoning+via+Looped+Language+Models+%28Ouro%29

8. Do Language Models Use Their Depth Efficiently? — Róbert Csordás, Christopher D. Manning, Christopher Potts, 2025

https://scholar.google.com/scholar?q=Do+Language+Models+Use+Their+Depth+Efficiently%3F

9. The Unreasonable Ineffectiveness of the Deeper Layers — Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts, 2024

https://scholar.google.com/scholar?q=The+Unreasonable+Ineffectiveness+of+the+Deeper+Layers

10. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, et al., 2024

https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling

11. Not All Layers of LLMs Are Necessary During Inference — Siqi Fan, Xin Jiang, Xuying Meng, et al., 2024

https://scholar.google.com/scholar?q=Not+All+Layers+of+LLMs+Are+Necessary+During+Inference

12. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Jun Zhang, Jue Wang, Huan Li, et al., 2023

https://scholar.google.com/scholar?q=Draft+%26+Verify%3A+Lossless+Large+Language+Model+Acceleration+via+Self-Speculative+Decoding

13. Adaptive Computation Time for Recurrent Neural Networks — Alex Graves, 2016

https://scholar.google.com/scholar?q=Adaptive+Computation+Time+for+Recurrent+Neural+Networks

14. Training Large Language Models to Reason in a Continuous Latent Space (Coconut) — Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, et al., 2024

https://scholar.google.com/scholar?q=Training+Large+Language+Models+to+Reason+in+a+Continuous+Latent+Space+%28Coconut%29

Interactive Visualization: Skip or Loop Layers: Learning Programs-of-Layers in Frozen LLMs

This episode examines "Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs," which proposes Chain-of-Layers (CoLa). CoLa takes a frozen pretrained model and, for each input, skips some layers, repeats others, and reorders them, with no finetuning. The hosts explain why skipping works, citing residual connections, ResNets behaving like ensembles of shallow paths, and static pruning results like ShortGPT and "The Unreasonable Ineffectiveness of the Deeper Layers". They also cover dynamic early-exit methods and looped-depth models such as Universal Transformers. They disagree about whether pruning results mean deeper layers are dead weight, since those results come mostly from multiple-choice tasks and multi-step reasoning degrades faster. The paper's claim is that many correctly answered samples still work with a shorter layer chain, and many wrong ones can be fixed by some other chain. The episode walks through the Monte Carlo Tree Search that finds these paths, including its skip and repeat edits, its scoring rule with an exploration bonus and a length penalty, and the resulting Pareto set of short, accurate paths. It also notes that the paper omits closely related prior work on frozen-model layer manipulation, which affects how novel the result is.

Sources:

1. Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs — Ziyue Li, Yang Li, Tianyi Zhou, 2025

http://arxiv.org/abs/2507.07996

2. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser, 2018 (ICLR 2019)

https://scholar.google.com/scholar?q=Universal+Transformers

3. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein, 2025

https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach

4. Mixture-of-Depths: Dynamically Allocating Compute in Transformer-Based Language Models — David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, Adam Santoro, 2024

https://scholar.google.com/scholar?q=Mixture-of-Depths%3A+Dynamically+Allocating+Compute+in+Transformer-Based+Language+Models

5. Transformer Layers as Painters — Qi Sun, Marc Pickett, Aakash Kumar Nain, Llion Jones, 2024

https://scholar.google.com/scholar?q=Transformer+Layers+as+Painters

6. The Unreasonable Ineffectiveness of the Deeper Layers — Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts, 2024

https://scholar.google.com/scholar?q=The+Unreasonable+Ineffectiveness+of+the+Deeper+Layers

7. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen, 2024

https://scholar.google.com/scholar?q=ShortGPT%3A+Layers+in+Large+Language+Models+are+More+Redundant+Than+You+Expect

8. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A. Aly, Beidi Chen, Carole-Jean Wu, 2024 (ACL 2024)

https://scholar.google.com/scholar?q=LayerSkip%3A+Enabling+Early+Exit+Inference+and+Self-Speculative+Decoding

9. Confident Adaptive Language Modeling — Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, Donald Metzler, 2022 (NeurIPS 2022)

https://scholar.google.com/scholar?q=Confident+Adaptive+Language+Modeling

10. Dr.LLM: Dynamic Layer Routing in LLMs — Ahmed Heakl et al., 2025

https://scholar.google.com/scholar?q=Dr.LLM%3A+Dynamic+Layer+Routing+in+LLMs

11. Confident Adaptive Language Modeling (CALM) — Tal Schuster et al., 2022

https://scholar.google.com/scholar?q=Confident+Adaptive+Language+Modeling+%28CALM%29

12. Router-Tuning / FlexiDepth: dynamic layer skipping in pretrained LLMs — Shwai He et al. (Router-Tuning); Xuan Luo et al. (FlexiDepth), 2024-2025

https://scholar.google.com/scholar?q=Router-Tuning+%2F+FlexiDepth%3A+dynamic+layer+skipping+in+pretrained+LLMs

13. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — Sangmin Bae et al., 2025

https://scholar.google.com/scholar?q=Mixture-of-Recursions%3A+Learning+Dynamic+Recursive+Depths+for+Adaptive+Token-Level+Computation

14. Do Language Models Use Their Depth Efficiently? — Róbert Csordás, Christopher D. Manning, Christopher Potts, 2025

https://scholar.google.com/scholar?q=Do+Language+Models+Use+Their+Depth+Efficiently%3F

15. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown et al., 2024

https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling

16. Residual Networks Behave Like Ensembles of Relatively Shallow Networks — Andreas Veit, Michael Wilber, Serge Belongie, 2016

https://scholar.google.com/scholar?q=Residual+Networks+Behave+Like+Ensembles+of+Relatively+Shallow+Networks

17. The Remarkable Robustness of LLMs: Stages of Inference? — Vedang Lad, Wes Gurnee, Max Tegmark, 2024

https://scholar.google.com/scholar?q=The+Remarkable+Robustness+of+LLMs%3A+Stages+of+Inference%3F

18. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding — Jun Zhang et al., 2023

https://scholar.google.com/scholar?q=Draft+%26+Verify%3A+Lossless+Large+Language+Model+Acceleration+via+Self-Speculative+Decoding

19. Interpreting GPT: the logit lens / Eliciting Latent Predictions from Transformers with the Tuned Lens — nostalgebraist (2020); Nora Belrose et al. (2023), 2020/2023

https://scholar.google.com/scholar?q=Interpreting+GPT%3A+the+logit+lens+%2F+Eliciting+Latent+Predictions+from+Transformers+with+the+Tuned+Lens

Interactive Visualization: Chain-of-Layers: Skipping and Looping Frozen LLM Layers Per Input

This episode explores Structural Language Models of Code, a 2020 paper from Technion, Tel Aviv University and Facebook AI Research that tackles "any-code completion": predicting a missing piece of a program with no restriction on vocabulary or structure. It traces the history from 1969-era automatic programming through domain-specific systems like FlashFill and DeepCoder, and prior general-language work that limited APIs, types or syntax. Then it contrasts a subtoken sequence-to-sequence baseline with the paper's approach of modeling code as an abstract syntax tree. That approach applies the language-model chain rule over a depth-first tree traversal, using partial AST paths that end at the node being generated, which extends the authors' earlier code2vec and code2seq work from reading code to writing it. The discussion covers the headline exact-match gains, Java accuracy@1 of 18.04 versus 16.93 and C# 37.61 versus 26.42. It also raises the question of whether a one-point win is meaningful, given that exact match undercounts logically equivalent code. Listeners interested in how syntax-aware models can generate code with unseen identifiers will find the mechanics and the skeptical read of the metrics useful.

Sources:

1. Structural Language Models of Code: Any-Code Completion with Trees

https://arxiv.org/pdf/1910.00577

2. Structured Generative Models of Natural Source Code — Chris J. Maddison, Daniel Tarlow, 2014

https://scholar.google.com/scholar?q=Structured+Generative+Models+of+Natural+Source+Code

3. On the Naturalness of Software — Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, Premkumar Devanbu, 2012

https://scholar.google.com/scholar?q=On+the+Naturalness+of+Software

4. A Syntactic Neural Model for General-Purpose Code Generation — Pengcheng Yin, Graham Neubig, 2017

https://scholar.google.com/scholar?q=A+Syntactic+Neural+Model+for+General-Purpose+Code+Generation

5. Generative Code Modeling with Graphs — Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, Oleksandr Polozov, 2019

https://scholar.google.com/scholar?q=Generative+Code+Modeling+with+Graphs

6. Code Completion with Statistical Language Models — Veselin Raychev, Martin Vechev, Eran Yahav, 2014

https://scholar.google.com/scholar?q=Code+Completion+with+Statistical+Language+Models

7. IntelliCode Compose: Code Generation Using Transformer — Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, Neel Sundaresan, 2020

https://scholar.google.com/scholar?q=IntelliCode+Compose%3A+Code+Generation+Using+Transformer

8. Efficient Training of Language Models to Fill in the Middle — Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen, 2022

https://scholar.google.com/scholar?q=Efficient+Training+of+Language+Models+to+Fill+in+the+Middle

9. A General Path-Based Representation for Predicting Program Properties — Uri Alon, Meital Zilberstein, Omer Levy, Eran Yahav, 2018

https://scholar.google.com/scholar?q=A+General+Path-Based+Representation+for+Predicting+Program+Properties

10. code2vec: Learning Distributed Representations of Code — Uri Alon, Meital Zilberstein, Omer Levy, Eran Yahav, 2019

https://scholar.google.com/scholar?q=code2vec%3A+Learning+Distributed+Representations+of+Code

11. code2seq: Generating Sequences from Structured Representations of Code — Uri Alon, Shaked Brody, Omer Levy, Eran Yahav, 2019

https://scholar.google.com/scholar?q=code2seq%3A+Generating+Sequences+from+Structured+Representations+of+Code

12. Learning to Represent Programs with Graphs — Miltiadis Allamanis, Marc Brockschmidt, Mahmoud Khademi, 2018

https://scholar.google.com/scholar?q=Learning+to+Represent+Programs+with+Graphs

13. Abstract Syntax Networks for Code Generation and Semantic Parsing — Maxim Rabinovich, Mitchell Stern, Dan Klein, 2017

https://scholar.google.com/scholar?q=Abstract+Syntax+Networks+for+Code+Generation+and+Semantic+Parsing

14. PHOG: Probabilistic Model for Code — Pavol Bielik, Veselin Raychev, Martin Vechev, 2016

https://scholar.google.com/scholar?q=PHOG%3A+Probabilistic+Model+for+Code

15. Incorporating Copying Mechanism in Sequence-to-Sequence Learning — Jiatao Gu, Zhengdong Lu, Hang Li, Victor O.K. Li, 2016

https://scholar.google.com/scholar?q=Incorporating+Copying+Mechanism+in+Sequence-to-Sequence+Learning

16. On the Bottleneck of Graph Neural Networks and its Practical Implications — Uri Alon, Eran Yahav, 2020

https://scholar.google.com/scholar?q=On+the+Bottleneck+of+Graph+Neural+Networks+and+its+Practical+Implications

17. InCoder: A Generative Model for Code Infilling and Synthesis — Daniel Fried et al., 2022

https://scholar.google.com/scholar?q=InCoder%3A+A+Generative+Model+for+Code+Infilling+and+Synthesis

18. Evaluating Large Language Models Trained on Code (Codex / HumanEval) — Mark Chen et al., 2021

https://scholar.google.com/scholar?q=Evaluating+Large+Language+Models+Trained+on+Code+%28Codex+%2F+HumanEval%29

19. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation — Fengji Zhang et al., 2023

https://scholar.google.com/scholar?q=RepoCoder%3A+Repository-Level+Code+Completion+Through+Iterative+Retrieval+and+Generation

20. UniXcoder: Unified Cross-Modal Pre-training for Code Representation — Daya Guo et al., 2022

https://scholar.google.com/scholar?q=UniXcoder%3A+Unified+Cross-Modal+Pre-training+for+Code+Representation

Interactive Visualization: Structural Language Models of Code: Any-Code Completion with Trees

This episode examines "The Unreasonable Ineffectiveness of the Deeper Layers," a 2025 study showing that up to half the layers of a 70-billion-parameter model can be removed with almost no drop in standard QA benchmark performance. The discussion covers the residual stream architecture that makes such pruning possible, why later transformer layers often contribute diminishing changes to accumulated representations, and how researchers identify which layer blocks are safe to cut by comparing input and output similarity. It also unpacks prior work on knowledge localization, including causal tracing of factual associations and feed-forward layers acting as key-value memories, to explain why deep layers might be redundant rather than essential. Finally, it details how QLoRA enables lightweight healing of the pruned model's mismatched seams using minimal compute, making the entire process feasible on a single GPU rather than a training cluster. Listeners interested in model efficiency, interpretability, or the surprising redundancy inside trusted large language models will find the core result — and the mechanics behind it — genuinely counterintuitive.

Sources:

1. The Unreasonable Ineffectiveness of the Deeper Layers — Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts, 2024

http://arxiv.org/abs/2403.17887

2. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen, 2024

https://scholar.google.com/scholar?q=ShortGPT%3A+Layers+in+Large+Language+Models+are+More+Redundant+Than+You+Expect

3. Locating and Editing Factual Associations in GPT (ROME) — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022

https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT+%28ROME%29

4. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021

https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories

5. Compact Language Models via Pruning and Knowledge Distillation (Minitron) — Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov (NVIDIA), 2024

https://scholar.google.com/scholar?q=Compact+Language+Models+via+Pruning+and+Knowledge+Distillation+%28Minitron%29

6. The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (LASER) — Pratyusha Sharma, Jordan T. Ash, Dipendra Misra, 2023

https://scholar.google.com/scholar?q=The+Truth+is+in+There%3A+Improving+Reasoning+in+Language+Models+with+Layer-Selective+Rank+Reduction+%28LASER%29

7. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, et al., 2024

https://scholar.google.com/scholar?q=LayerSkip%3A+Enabling+Early+Exit+Inference+and+Self-Speculative+Decoding

8. Are Emergent Abilities of Large Language Models a Mirage? — Rylan Schaeffer, Brando Miranda, Sanmi Koyejo, 2023

https://scholar.google.com/scholar?q=Are+Emergent+Abilities+of+Large+Language+Models+a+Mirage%3F

9. SliceGPT: Compress Large Language Models by Deleting Rows and Columns — Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, James Hensman, 2024

https://scholar.google.com/scholar?q=SliceGPT%3A+Compress+Large+Language+Models+by+Deleting+Rows+and+Columns

10. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks — Greg Yang, Dingli Yu, Chen Zhu, Soufiane Hayou, 2023

https://scholar.google.com/scholar?q=Tensor+Programs+VI%3A+Feature+Learning+in+Infinite-Depth+Neural+Networks

Interactive Visualization: The Unreasonable Ineffectiveness of Deep Transformer Layers

This episode examines ShortGPT, a layer-pruning method that ranks the layers of a pre-norm LLM by Block Influence (one minus the average cosine similarity between a layer's input and output hidden states) and deletes the lowest-scoring ones with no gradients or retraining. It sets the method against unstructured pruning and the structured baselines LLM-Pruner, SliceGPT, and LaCo. It also covers the pre-norm residual-stream argument for why deep layers may be redundant, along with the limits of that argument. The central tension is the headline result: removing nine layers from a 32-layer model barely dents MMLU (45.4 to 44.0) but collapses XSum summarization (19.40 to 0.67). That gap prompts the question of whether "roughly 90% of performance retained" reflects real compression or an artifact of multiple-choice metrics that don't test multi-step generation. Listeners get a skeptical look at why a small-angle update isn't necessarily an unimportant one, illustrated by the last layer's FFN, whose removal sends perplexity from 7.60 to 12.35.

Sources:

1. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen, 2024

http://arxiv.org/abs/2403.03853

2. ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, Weipeng Chen, 2024

https://scholar.google.com/scholar?q=ShortGPT%3A+Layers+in+Large+Language+Models+are+More+Redundant+Than+You+Expect

3. The Unreasonable Ineffectiveness of the Deeper Layers — Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, Daniel A. Roberts, 2024 (arXiv; ICLR 2025)

https://scholar.google.com/scholar?q=The+Unreasonable+Ineffectiveness+of+the+Deeper+Layers

4. Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods — Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, Hyoung-Kyu Song, 2024

https://scholar.google.com/scholar?q=Shortened+LLaMA%3A+Depth+Pruning+for+Large+Language+Models+with+Comparison+of+Retraining+Methods

5. Compact Language Models via Pruning and Knowledge Distillation (Minitron) — Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, Pavlo Molchanov, 2024

https://scholar.google.com/scholar?q=Compact+Language+Models+via+Pruning+and+Knowledge+Distillation+%28Minitron%29

6. SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks — Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, Jae-Joon Kim, 2024

https://scholar.google.com/scholar?q=SLEB%3A+Streamlining+LLMs+through+Redundancy+Verification+and+Elimination+of+Transformer+Blocks

7. Reassessing Layer Pruning in LLMs: New Insights and Methods — Yao Lu, Hao Cheng, Yujie Fang, Zeyu Wang, Jiaheng Wei, Dongwei Xu, Qi Xuan, Xiaoniu Yang, Zhaowei Zhu, 2024

https://scholar.google.com/scholar?q=Reassessing+Layer+Pruning+in+LLMs%3A+New+Insights+and+Methods

8. Do Language Models Use Their Depth Efficiently? — Róbert Csordás, Christopher D. Manning, Christopher Potts, 2025

https://scholar.google.com/scholar?q=Do+Language+Models+Use+Their+Depth+Efficiently%3F

9. The Curse of Depth in Large Language Models — Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, Shiwei Liu, 2025

https://scholar.google.com/scholar?q=The+Curse+of+Depth+in+Large+Language+Models

10. Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN — Pengxiang Li, Lu Yin, Shiwei Liu, 2024

https://scholar.google.com/scholar?q=Mix-LN%3A+Unleashing+the+Power+of+Deeper+Layers+by+Combining+Pre-LN+and+Post-LN

11. Transformer Layers as Painters — Qi Sun, Marc Pickett, Aakash Kumar Nain, Llion Jones, 2024

https://scholar.google.com/scholar?q=Transformer+Layers+as+Painters

12. Massive Activations in Large Language Models — Mingjie Sun, Xinlei Chen, J. Zico Kolter, Zhuang Liu, 2024

https://scholar.google.com/scholar?q=Massive+Activations+in+Large+Language+Models

13. A Deeper Look at Depth Pruning of LLMs — Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, Pavlo Molchanov, 2024

https://scholar.google.com/scholar?q=A+Deeper+Look+at+Depth+Pruning+of+LLMs

14. What Matters in Transformers? Not All Attention is Needed — Shwai He, Guoheng Sun, Zheyu Shen, Ang Li, 2024

https://scholar.google.com/scholar?q=What+Matters+in+Transformers%3F+Not+All+Attention+is+Needed

15. Residual Networks Behave Like Ensembles of Relatively Shallow Networks — Andreas Veit, Michael Wilber, Serge Belongie, 2016

https://scholar.google.com/scholar?q=Residual+Networks+Behave+Like+Ensembles+of+Relatively+Shallow+Networks

16. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — Mostafa Elhoushi et al., 2024

https://scholar.google.com/scholar?q=LayerSkip%3A+Enabling+Early+Exit+Inference+and+Self-Speculative+Decoding

17. Compressing LLMs: The Truth is Rarely Pure and Never Simple — Ajay Jaiswal, Zhenyu Zhang, Zhangyang Wang, Yinfei Yang, Shiwei Liu, Yi Zhang, et al., 2023

https://scholar.google.com/scholar?q=Compressing+LLMs%3A+The+Truth+is+Rarely+Pure+and+Never+Simple

Interactive Visualization: ShortGPT: Deleting Redundant LLM Layers, Free Lunch or Artifact?