AI Post Transformers

This episode examines JustAskJev, a reinforcement-learning-trained detector that flags AI alignment failures using a single generic yes-or-no question asked in a zero-shot setting. It contrasts this approach with conventional generative LLM-as-judge and classifier-based detectors like Llama Guard, explaining how RLCD (reinforcement learning for calibrated decisions) lets one model process typed questions—yes/no, categorical, ordinal—against a shared "state" in a single call rather than requiring separate passes per failure type. The discussion covers the paper's headline result: a 0.886 median AUROC across ten distinct failure modes (sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, bias, reward hacking, concealed uncertainty, and power seeking) spanning forty-four benchmarks and five target models. It also unpacks calibration as a property distinct from accuracy, using sycophancy as a case study for why a model's stated confidence needs to track real-world correctness. Listeners interested in AI safety evaluation, model auditing costs, or the mechanics of confidence calibration will find the paper's claims about cheaper, unified failure detection a compelling departure from today's fragmented screening tools.

Sources:

1. Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures — Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, Leo Yu Zhang, 2026

http://arxiv.org/abs/2609.29429

2. On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, 2017

https://scholar.google.com/scholar?q=On+Calibration+of+Modern+Neural+Networks

3. Language Models (Mostly) Know What They Know — Saurav Kadavath, Tom Conerly, Amanda Askell, and others (Anthropic), 2022

https://scholar.google.com/scholar?q=Language+Models+%28Mostly%29+Know+What+They+Know

4. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback — Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D. Manning, 2023

https://scholar.google.com/scholar?q=Just+Ask+for+Calibration%3A+Strategies+for+Eliciting+Calibrated+Confidence+Scores+from+Language+Models+Fine-Tuned+with+Human+Feedback

5. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations — Hakan Inan, Kartikeya Upasani, and others (Meta), 2023

https://scholar.google.com/scholar?q=Llama+Guard%3A+LLM-based+Input-Output+Safeguard+for+Human-AI+Conversations

6. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models — Carson Denison, Monte MacDiarmid, Fazl Barez, et al., 2024

https://scholar.google.com/scholar?q=Sycophancy+to+Subterfuge%3A+Investigating+Reward-Tampering+in+Large+Language+Models

7. The MASK Benchmark: Disentangling Honesty from Accuracy in AI Systems — Richard Ren, Arunim Agarwal, Mantas Mazeika, et al., 2025

https://scholar.google.com/scholar?q=The+MASK+Benchmark%3A+Disentangling+Honesty+from+Accuracy+in+AI+Systems

8. Confident Learning: Estimating Uncertainty in Dataset Labels — Curtis G. Northcutt, Lu Jiang, Isaac L. Chuang, 2021

https://scholar.google.com/scholar?q=Confident+Learning%3A+Estimating+Uncertainty+in+Dataset+Labels

Interactive Visualization: JustAskJev: One Question Detects Ten AI Failure Modes

This episode explores a paper arguing that KV cache compressibility is not an inherent property of an input, but a structural property of how a transformer's weights happen to represent a computation. Using a histogram-counting thought experiment, the authors prove that two transformers can compute the identical function while one is trivially compressible and the other is fundamentally not, exposing a blind spot in existing inference-time compression techniques like heavy-hitter eviction, attention-sink methods, and optimization-based approaches such as Cartridges and Attention Matching. The discussion surveys the broader landscape of memory-saving strategies, contrasting architectural fixes like linear attention and state space models against post-hoc interventions on standard transformers, and explains why long-context and agentic workloads make cache size a serious bottleneck. The hosts debate whether the paper's formal construction is just a toy example or a meaningful warning, concluding that it reframes compressibility as something that must be deliberately trained for rather than assumed to emerge naturally from ordinary pretraining. Listeners interested in efficient LLM serving will find this useful for understanding why some models resist compression no matter how sophisticated the algorithm applied to them.

Sources:

1. Training Transformers for KV Cache Compressibility — Yoav Gelberg, Yam Eitan, Michael Bronstein, Yarin Gal, Haggai Maron, 2026

http://arxiv.org/abs/2605.05971

2. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai, 2023

https://scholar.google.com/scholar?q=GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints

3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, 2023

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models

4. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

5. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI (Aixin Liu et al.), 2024

https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model

6. Cartridges: Lightweight and general-purpose long context representations via self-study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré, 2025

https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study

7. Fast KV compaction via attention matching — Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim, 2026

https://scholar.google.com/scholar?q=Fast+KV+compaction+via+attention+matching

8. Dynamic chunking for end-to-end hierarchical sequence modeling — Sukjun Hwang, Brandon Wang, Albert Gu, 2025

https://scholar.google.com/scholar?q=Dynamic+chunking+for+end-to-end+hierarchical+sequence+modeling

9. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads — Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, Song Han, 2025

https://scholar.google.com/scholar?q=DuoAttention%3A+Efficient+long-context+LLM+inference+with+retrieval+and+streaming+heads

10. Lexico: Extreme KV cache compression via sparse coding over universal dictionaries — Junhyuck Kim, Jongho Park, Jaewoong Cho, Dimitris Papailiopoulos, 2025

https://scholar.google.com/scholar?q=Lexico%3A+Extreme+KV+cache+compression+via+sparse+coding+over+universal+dictionaries

Interactive Visualization: Training Transformers for Compressible KV Caches

This episode examines the Qwen team's design-rationale report for Qwen3.8-Flash-Next, a 125B-total, 6B-activated base model with 51B of n-gram embeddings held in host memory. It walks through the five components: a hybrid of Gated DeltaNet layers with periodic full attention, Qwen's sparse attention variant (QSA) as a response to the quadratic cost of DeepSeek-style indexers, a four-branch gated residual stream, a hashed n-gram lookup layer, and the Muon optimizer. The recurring question is whether loss, benchmark accuracy, cost, and stability agree. The paper reports that larger n-gram vocabularies always lower loss while accuracy stays flat. The hosts also note that most ablations are single runs with no reported seeds, and that the paper itself says top-setting margins likely fall within evaluation noise. The headline claim, matching or beating a 397B-A17B predecessor on eight of fourteen benchmarks at roughly a ninth of the training compute, is set against the ablation evidence behind it. The episode is useful for anyone who wants to see how the design choices trace to prior work (highway networks, Hyper-Connections, Engram, LongCat) and how much weight those choices can bear.

Sources:

1. On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability — Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu, 2026

http://arxiv.org/abs/2608.30320v1

2. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram) — Xin Cheng et al. (DeepSeek-AI, with Peking University), 2026

https://scholar.google.com/scholar?q=Conditional+Memory+via+Scalable+Lookup%3A+A+New+Axis+of+Sparsity+for+Large+Language+Models+%28Engram%29

3. Scaling Embedding Layers in Language Models (SCONE) — Da Yu, Edith Cohen, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Chiyuan Zhang (Google), 2025

https://scholar.google.com/scholar?q=Scaling+Embedding+Layers+in+Language+Models+%28SCONE%29

4. Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling — Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, Xun Zhou (ByteDance Seed), 2025

https://scholar.google.com/scholar?q=Over-Tokenized+Transformer%3A+Vocabulary+is+Generally+Worth+Scaling

5. Memory Layers at Scale — Vincent-Pierre Berges, Barlas Oguz, Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, Gargi Ghosh (Meta FAIR), 2024

https://scholar.google.com/scholar?q=Memory+Layers+at+Scale

6. Scaling Embeddings Outperforms Scaling Experts in Language Models — Hong Liu, Jiaqi Zhang, Chao Wang, ... Xunliang Cai (Meituan LongCat), 2026

https://scholar.google.com/scholar?q=Scaling+Embeddings+Outperforms+Scaling+Experts+in+Language+Models

7. N-Grammer: Augmenting Transformers with Latent n-grams — Aurko Roy, Rohan Anil, Guangda Lai, et al., 2022

https://scholar.google.com/scholar?q=N-Grammer%3A+Augmenting+Transformers+with+Latent+n-grams

8. Scaling Embedding Layers in Language Models — Da Yu, Edith Cohen, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Chiyuan Zhang, 2025

https://scholar.google.com/scholar?q=Scaling+Embedding+Layers+in+Language+Models

9. L3: Large Lookup Layers — Albert Tseng, Christopher De Sa, 2026

https://scholar.google.com/scholar?q=L3%3A+Large+Lookup+Layers

10. STEM: Scaling Transformers with Embedding Modules — Ranajoy Sadhukhan, Sheng Cao, Harry Dong, Changsheng Zhao, Attiano Purpura-Pontoniere, Yuandong Tian, Zechun Liu, Beidi Chen, 2026

https://scholar.google.com/scholar?q=STEM%3A+Scaling+Transformers+with+Embedding+Modules

11. Large Memory Layers with Product Keys — Guillaume Lample, Alexandre Sablayrolles, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou, 2019

https://scholar.google.com/scholar?q=Large+Memory+Layers+with+Product+Keys

12. Mixture of A Million Experts (PEER) — Xu Owen He, 2024

https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts+%28PEER%29

13. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — Zeyuan Allen-Zhu, Yuanzhi Li, 2024

https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.3%2C+Knowledge+Capacity+Scaling+Laws

14. Quantifying Memorization Across Neural Language Models — Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, Chiyuan Zhang, 2023

https://scholar.google.com/scholar?q=Quantifying+Memorization+Across+Neural+Language+Models

15. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan et al. (DeepSeek), 2025

https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention

16. Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Team, 2025

https://scholar.google.com/scholar?q=Kimi+Linear%3A+An+Expressive%2C+Efficient+Attention+Architecture

17. Small-scale Proxies for Large-scale Transformer Training Instabilities — Mitchell Wortsman et al., 2023

https://scholar.google.com/scholar?q=Small-scale+Proxies+for+Large-scale+Transformer+Training+Instabilities

Interactive Visualization: Qwen3.8-Next Design: Hybrid Attention, Residuals, and N-gram Embeddings

This episode examines DeepSeek-AI's mHC: Manifold-Constrained Hyper-Connections, which tries to improve the transformer's residual connection, a piece of the architecture that has barely changed in a decade. It first explains why the plain skip path x + F(x) has been so hard to displace. It then covers ByteDance's Hyper-Connections, which widen the residual stream into four parallel streams with learnable read, write and mixing maps. The mixing matrix alone accounts for most of the reported loss gain. The discussion turns to the failure mode: the product of unconstrained mixing matrices across layers lets signal gain climb to roughly 3000 in a 27B model, alongside a loss surge and a memory-bandwidth bill. The fix constrains the mixing matrix to be doubly stochastic using the 1967 Sinkhorn-Knopp algorithm, which keeps the composite mapping bounded. The hosts also weigh whether a doubly stochastic matrix, which is not identity, can preserve the gradient-flow argument for identity skip paths. They cover the added kernel and pipeline engineering, which reportedly costs about 6.7% extra training time.

Sources:

1. mHC: Manifold-Constrained Hyper-Connections — Zhenda Xie, Yixuan Wei, Huanqi Cao, Chenggang Zhao, Chengqi Deng, Jiashi Li, Damai Dai, Huazuo Gao, Jiang Chang, Kuai Yu, Liang Zhao, Shangyan Zhou, Zhean Xu, Zhengyan Zhang, Wangding Zeng, Shengding Hu, Yuqing Wang, Jingyang Yuan, Lean Wang, Wenfeng Liang, 2025

http://arxiv.org/abs/2512.24880

2. Hyper-Connections — Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, Xun Zhou, 2024

https://scholar.google.com/scholar?q=Hyper-Connections

3. Identity Mappings in Deep Residual Networks — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2016

https://scholar.google.com/scholar?q=Identity+Mappings+in+Deep+Residual+Networks

4. Concerning nonnegative matrices and doubly stochastic matrices — Richard Sinkhorn, Paul Knopp, 1967

https://scholar.google.com/scholar?q=Concerning+nonnegative+matrices+and+doubly+stochastic+matrices

5. Sinkformers: Transformers with Doubly Stochastic Attention — Michael E. Sander, Pierre Ablin, Mathieu Blondel, Gabriel Peyré, 2022

https://scholar.google.com/scholar?q=Sinkformers%3A+Transformers+with+Doubly+Stochastic+Attention

6. Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth — Yihe Dong, Jean-Baptiste Cordonnier, Andreas Loukas, 2021

https://scholar.google.com/scholar?q=Attention+is+Not+All+You+Need%3A+Pure+Attention+Loses+Rank+Doubly+Exponentially+with+Depth

7. Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning / Graph Neural Networks Exponentially Lose Expressive Power for Node Classification — Qimai Li, Zhichao Han, Xiao-Ming Wu; Kenta Oono, Taiji Suzuki, 2018 / 2020

https://scholar.google.com/scholar?q=Deeper+Insights+into+Graph+Convolutional+Networks+for+Semi-Supervised+Learning+%2F+Graph+Neural+Networks+Exponentially+Lose+Expressive+Power+for+Node+Classification

8. DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging — Matteo Pagliardini, Amirkeivan Mohtashami, François Fleuret, Martin Jaggi, 2024

https://scholar.google.com/scholar?q=DenseFormer%3A+Enhancing+Information+Flow+in+Transformers+via+Depth+Weighted+Averaging

9. MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections — Da Xiao, Qingye Meng, Shengping Li, Xingyuan Yuan, 2025

https://scholar.google.com/scholar?q=MUDDFormer%3A+Breaking+Residual+Bottlenecks+in+Transformers+via+Multiway+Dynamic+Dense+Connections

10. Residual Matrix Transformers: Scaling the Size of the Residual Stream — Brian Mak, Jeffrey Flanigan, 2025

https://scholar.google.com/scholar?q=Residual+Matrix+Transformers%3A+Scaling+the+Size+of+the+Residual+Stream

11. DeepNet: Scaling Transformers to 1,000 Layers — Hongyu Wang, Shuming Ma, Li Dong, et al., 2022

https://scholar.google.com/scholar?q=DeepNet%3A+Scaling+Transformers+to+1%2C000+Layers

12. ReZero is All You Need: Fast Convergence at Large Depth — Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, Julian McAuley, 2020

https://scholar.google.com/scholar?q=ReZero+is+All+You+Need%3A+Fast+Convergence+at+Large+Depth

13. The Curse of Depth in Large Language Models — Wenfang Sun et al., 2025

https://scholar.google.com/scholar?q=The+Curse+of+Depth+in+Large+Language+Models

14. Value Residual Learning — Zhanchao Zhou et al., 2024

https://scholar.google.com/scholar?q=Value+Residual+Learning

15. Resurrecting the Sigmoid in Deep Learning through Dynamical Isometry / Deep Information Propagation — Jeffrey Pennington, Samuel Schoenholz, Surya Ganguli; Samuel Schoenholz et al., 2017

https://scholar.google.com/scholar?q=Resurrecting+the+Sigmoid+in+Deep+Learning+through+Dynamical+Isometry+%2F+Deep+Information+Propagation

16. DeepSeek-V3 Technical Report — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report

Interactive Visualization: mHC: Manifold-Constrained Hyper-Connections Stabilize Wider Residual Streams

This episode examines "You Only Cache Once," a decoder-decoder architecture from Microsoft Research and Tsinghua University that computes one global key-value cache and reuses it across every upper layer. The hosts explain why the KV cache becomes a deployment bottleneck at long context. They cite a 65B model needing about 86 GB of cache at 512K tokens even with grouped-query attention and 8-bit quantization. They then walk through the design: a bottom self-decoder uses constant-state efficient attention (sliding-window or gated retention), and a top cross-decoder cross-attends to the single shared cache. The result stays causal like a decoder-only model but cuts cache memory roughly L-fold and lets prefill skip the cross-decoder. The paper claims about an 80x smaller cache, 71.8x faster prefill at a million tokens, and 9.6x higher throughput at 512K with quality on par with a Transformer. The hosts also set out to test what those multipliers are measured against and whether every layer reading the same memory costs quality. It suits listeners interested in long-context serving costs and in how architecture design can address them.

Sources:

1. You Only Cache Once: Decoder-Decoder Architectures for Language Models — Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei, 2024

http://arxiv.org/abs/2405.05254v2

2. You Only Cache Once: Decoder-Decoder Architectures for Language Models — Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei, 2024

https://scholar.google.com/scholar?q=You+Only+Cache+Once%3A+Decoder-Decoder+Architectures+for+Language+Models

3. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017

https://scholar.google.com/scholar?q=Attention+Is+All+You+Need

4. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) — Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, 2020

https://scholar.google.com/scholar?q=Exploring+the+Limits+of+Transfer+Learning+with+a+Unified+Text-to-Text+Transformer+%28T5%29

5. Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation (SambaY) — Liliang Ren, Congcong Chen, Haoran Xu, Young Jin Kim, Yelong Shen, Weizhu Chen, Jianfeng Gao and coauthors (Microsoft), 2025

https://scholar.google.com/scholar?q=Decoder-Hybrid-Decoder+Architecture+for+Efficient+Reasoning+with+Long+Generation+%28SambaY%29

6. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan-Kelley, 2024

https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention

7. Layer-Condensed KV Cache for Efficient Inference of Large Language Models — Haoyi Wu, Kewei Tu, 2024

https://scholar.google.com/scholar?q=Layer-Condensed+KV+Cache+for+Efficient+Inference+of+Large+Language+Models

8. Fast Transformer Decoding: One Write-Head is All You Need, and GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Noam Shazeer (2019); Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, Sumit Sanghai (2023), 2019 and 2023

https://scholar.google.com/scholar?q=Fast+Transformer+Decoding%3A+One+Write-Head+is+All+You+Need%2C+and+GQA%3A+Training+Generalized+Multi-Query+Transformer+Models+from+Multi-Head+Checkpoints

9. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model

10. SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformation — Aurick Qiao, Zhewei Yao, Samyam Rajbhandari, Yuxiong He (Snowflake AI Research), 2024

https://scholar.google.com/scholar?q=SwiftKV%3A+Fast+Prefill-Optimized+Inference+with+Knowledge-Preserving+Model+Transformation

11. Confident Adaptive Language Modeling (CALM) — Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, Donald Metzler, 2022

https://scholar.google.com/scholar?q=Confident+Adaptive+Language+Modeling+%28CALM%29

12. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A. Aly, Beidi Chen, Carole-Jean Wu, 2024

https://scholar.google.com/scholar?q=LayerSkip%3A+Enabling+Early+Exit+Inference+and+Self-Speculative+Decoding

13. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024

https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve

14. Retentive Network: A Successor to Transformer for Large Language Models — Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei, 2023

https://scholar.google.com/scholar?q=Retentive+Network%3A+A+Successor+to+Transformer+for+Large+Language+Models

15. Gated Linear Attention Transformers with Hardware-Efficient Training — Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim, 2023

https://scholar.google.com/scholar?q=Gated+Linear+Attention+Transformers+with+Hardware-Efficient+Training

16. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023

https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

17. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Francois Fleuret, 2020

https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention

18. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (Multi-head Latent Attention) — DeepSeek-AI, 2024

https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model+%28Multi-head+Latent+Attention%29

19. Jamba: A Hybrid Transformer-Mamba Language Model — Opher Lieber et al., 2024

https://scholar.google.com/scholar?q=Jamba%3A+A+Hybrid+Transformer-Mamba+Language+Model

20. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — Soham De et al., 2024

https://scholar.google.com/scholar?q=Griffin%3A+Mixing+Gated+Linear+Recurrences+with+Local+Attention+for+Efficient+Language+Models

21. Zoology: Measuring and Improving Recall in Efficient Language Models — Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, Christopher Ré, 2023

https://scholar.google.com/scholar?q=Zoology%3A+Measuring+and+Improving+Recall+in+Efficient+Language+Models

22. Efficient Streaming Language Models with Attention Sinks (StreamingLLM) — Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis, 2023

https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks+%28StreamingLLM%29

23. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models; and SnapKV — Zhenyu Zhang et al.; Yuhong Li et al., 2023/2024

https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models%3B+and+SnapKV

24. Efficiently Scaling Transformer Inference — Reiner Pope et al., 2022

https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference

25. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh et al., 2024

https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F

Interactive Visualization: YOCO: Caching Once for Decoder-Decoder Long-Context Language Models

This episode examines Hyper-Connections (ICLR 2025, ByteDance Seed), which replaces the fixed residual connection in transformers with learned connection strengths. It traces the "seesaw" between Pre-Norm and Post-Norm: Pre-Norm gives stable gradients but suffers representation collapse in deep layers, while Post-Norm does the reverse. The method carries n parallel residual streams and learns depth-connections and width-connections, optionally predicted per token. It initializes as exactly Pre-Norm, so it strictly generalizes both variants. The paper's headline claim is 1.8x faster convergence and about six extra points on ARC-Challenge for OLMoE-1B-7B, for roughly 0.03% more parameters and 0.2% more FLOPs. The hosts place the idea against Highway Networks, DenseNet, and DenseFormer, and they flag that the collapse evidence rests largely on a single cosine-similarity figure. Listeners get a clear view of how a small learned wiring matrix can change a core piece of transformer design, along with a skeptical read of the reported gains.

Sources:

1. Hyper-Connections — Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, Xun Zhou, 2024

http://arxiv.org/abs/2409.19606

2. Deep Residual Learning for Image Recognition — Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2016

https://scholar.google.com/scholar?q=Deep+Residual+Learning+for+Image+Recognition

3. On Layer Normalization in the Transformer Architecture — Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu, 2020

https://scholar.google.com/scholar?q=On+Layer+Normalization+in+the+Transformer+Architecture

4. Understanding the Difficulty of Training Transformers — Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Jiawei Han, 2020

https://scholar.google.com/scholar?q=Understanding+the+Difficulty+of+Training+Transformers

5. Densely Connected Convolutional Networks (DenseNet), with DenseFormer as the transformer analogue — Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger (DenseNet, 2017); Matteo Pagliardini, Amirkeivan Mohtashami, François Fleuret, Martin Jaggi (DenseFormer, 2024), 2017 / 2024

https://scholar.google.com/scholar?q=Densely+Connected+Convolutional+Networks+%28DenseNet%29%2C+with+DenseFormer+as+the+transformer+analogue

6. DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging — Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret, Martin Jaggi, 2024

https://scholar.google.com/scholar?q=DenseFormer%3A+Enhancing+Information+Flow+in+Transformers+via+Depth+Weighted+Averaging

7. Highway Networks / Training Very Deep Networks — Rupesh Srivastava, Klaus Greff, Jürgen Schmidhuber, 2015

https://scholar.google.com/scholar?q=Highway+Networks+%2F+Training+Very+Deep+Networks

8. Densely Connected Convolutional Networks (DenseNet) — Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Weinberger, 2017

https://scholar.google.com/scholar?q=Densely+Connected+Convolutional+Networks+%28DenseNet%29

9. DeepNet: Scaling Transformers to 1,000 Layers (DeepNorm) — Hongyu Wang et al., 2022

https://scholar.google.com/scholar?q=DeepNet%3A+Scaling+Transformers+to+1%2C000+Layers+%28DeepNorm%29

10. ReZero is All You Need: Fast Convergence at Large Depth — Thomas Bachlechner et al., 2021

https://scholar.google.com/scholar?q=ReZero+is+All+You+Need%3A+Fast+Convergence+at+Large+Depth

11. Peri-LN: Revisiting Layer Normalization in the Transformer Architecture / Sandwich-norm variants — Jeonghoon Kim et al., 2025

https://scholar.google.com/scholar?q=Peri-LN%3A+Revisiting+Layer+Normalization+in+the+Transformer+Architecture+%2F+Sandwich-norm+variants

12. The Curse of Depth in Large Language Models — Wenfang Sun et al., 2025

https://scholar.google.com/scholar?q=The+Curse+of+Depth+in+Large+Language+Models

13. Transformer Layers as Painters — Qi Sun, Marc Pickett, et al., 2024

https://scholar.google.com/scholar?q=Transformer+Layers+as+Painters

14. AltUp: Alternating Updates for Efficient Transformers — Cenk Baykal et al., 2023/2024

https://scholar.google.com/scholar?q=AltUp%3A+Alternating+Updates+for+Efficient+Transformers

15. ResiDual: Transformer with Dual Residual Connections — Shufang Xie et al., 2023

https://scholar.google.com/scholar?q=ResiDual%3A+Transformer+with+Dual+Residual+Connections

16. Value Residual Learning / ResFormer — Zhanchao Zhou et al., 2024

https://scholar.google.com/scholar?q=Value+Residual+Learning+%2F+ResFormer

17. A Mathematical Framework for Transformer Circuits (residual stream view) — Nelson Elhage et al., 2021

https://scholar.google.com/scholar?q=A+Mathematical+Framework+for+Transformer+Circuits+%28residual+stream+view%29

18. Reducing Activation Recomputation in Large Transformer Models — Vijay Korthikanti et al., 2022

https://scholar.google.com/scholar?q=Reducing+Activation+Recomputation+in+Large+Transformer+Models

19. mHC: Manifold-Constrained Hyper-Connections — DeepSeek-AI, 2025/2026

https://scholar.google.com/scholar?q=mHC%3A+Manifold-Constrained+Hyper-Connections

20. Frac-Connections / Virtual Width Networks (ByteDance Seed follow-ups) — ByteDance Seed team, 2025

https://scholar.google.com/scholar?q=Frac-Connections+%2F+Virtual+Width+Networks+%28ByteDance+Seed+follow-ups%29

Interactive Visualization: Hyper-Connections: Learning Residual Strengths for Faster Transformer Training

This episode explores "Subliminal Learning," a paper showing that a language model can pass behavioral traits to a student model through data that looks unrelated to the trait. In the headline experiment, a teacher prompted to love owls generates only number sequences. After filtering, a fresh copy of the base model is finetuned on those numbers, and its share of "owl" answers to a favorite-animal question rises from about 12% to over 60%. The hosts place this against earlier work: Hinton's "dark knowledge" in distillation, emergent misalignment from finetuning on insecure code, and adversarial examples as invisible predictive features. They also cover how the teacher-student pipeline works and why the effect seems to depend on a shared initialization. The episode matters for anyone who assumes that filtering distilled training data is enough to keep unwanted traits out, and it raises the possibility that some emergent misalignment is subliminal learning rather than a result of what the data says.

Sources:

1. Subliminal Learning: Language models transmit behavioral traits via hidden signals in data — Alex Cloud, Minh Le, James Chua, Jan Betley, Anna Sztyber-Betley, Jacob Hilton, Samuel Marks, Owain Evans, 2025

http://arxiv.org/abs/2507.14805

2. Distilling the Knowledge in a Neural Network — Geoffrey Hinton, Oriol Vinyals, Jeff Dean, 2015

https://scholar.google.com/scholar?q=Distilling+the+Knowledge+in+a+Neural+Network

3. Adversarial Examples Are Not Bugs, They Are Features — Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry, 2019

https://scholar.google.com/scholar?q=Adversarial+Examples+Are+Not+Bugs%2C+They+Are+Features

4. Preventing Language Models From Hiding Their Reasoning — Fabien Roger, Ryan Greenblatt, 2023

https://scholar.google.com/scholar?q=Preventing+Language+Models+From+Hiding+Their+Reasoning

5. AI Models Collapse When Trained on Recursively Generated Data — Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, Yarin Gal, 2024

https://scholar.google.com/scholar?q=AI+Models+Collapse+When+Trained+on+Recursively+Generated+Data

6. Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs — Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain Evans, 2025

https://scholar.google.com/scholar?q=Emergent+Misalignment%3A+Narrow+Finetuning+Can+Produce+Broadly+Misaligned+LLMs

7. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training — Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, et al., 2024

https://scholar.google.com/scholar?q=Sleeper+Agents%3A+Training+Deceptive+LLMs+that+Persist+Through+Safety+Training

8. Poisoning Web-Scale Training Datasets is Practical — Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, Florian Tramèr, 2023

https://scholar.google.com/scholar?q=Poisoning+Web-Scale+Training+Datasets+is+Practical

9. Persona Features Control Emergent Misalignment — Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, et al. (OpenAI), 2025

https://scholar.google.com/scholar?q=Persona+Features+Control+Emergent+Misalignment

10. Born Again Neural Networks — Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar, 2018

https://scholar.google.com/scholar?q=Born+Again+Neural+Networks

11. Distillation Robustifies Unlearning — Bruce W. Lee, Addie Foote, Alex Infanger, Leni Shor, Harish Kamath, Jacob Goldman-Wetzler, Bryce Woodworth, Alex Cloud, Alexander Matt Turner, 2025

https://scholar.google.com/scholar?q=Distillation+Robustifies+Unlearning

12. Model Organisms for Emergent Misalignment — Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, Neel Nanda, 2025

https://scholar.google.com/scholar?q=Model+Organisms+for+Emergent+Misalignment

13. Alignment Faking in Large Language Models — Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, et al., 2024

https://scholar.google.com/scholar?q=Alignment+Faking+in+Large+Language+Models

14. Unnatural Languages Are Not Bugs but Features for LLMs — Keyu Duan, Yiran Zhao, Zhili Feng, et al., 2025

https://scholar.google.com/scholar?q=Unnatural+Languages+Are+Not+Bugs+but+Features+for+LLMs

15. Linear Mode Connectivity and the Lottery Ticket Hypothesis — Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin, 2020

https://scholar.google.com/scholar?q=Linear+Mode+Connectivity+and+the+Lottery+Ticket+Hypothesis

16. What is being transferred in transfer learning? — Behnam Neyshabur, Hanie Sedghi, Chiyuan Zhang, 2020

https://scholar.google.com/scholar?q=What+is+being+transferred+in+transfer+learning%3F

17. Sequence-Level Knowledge Distillation — Yoon Kim, Alexander M. Rush, 2016

https://scholar.google.com/scholar?q=Sequence-Level+Knowledge+Distillation

18. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks — Ali Shafahi, W. Ronny Huang, Mahyar Najibi, et al., 2018

https://scholar.google.com/scholar?q=Poison+Frogs%21+Targeted+Clean-Label+Poisoning+Attacks+on+Neural+Networks

19. Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer (follow-up analyses of the effect) — Schrodi et al., 2025

https://scholar.google.com/scholar?q=Towards+Understanding+Subliminal+Learning%3A+When+and+How+Hidden+Biases+Transfer+%28follow-up+analyses+of+the+effect%29

Interactive Visualization: Subliminal Learning: Hidden Behavioral Traits Transmitted Through Model-Generated Data

This episode explores "Next-Latent Prediction Transformers Learn Compact World Models," a Microsoft Research paper proposing a small auxiliary loss that pushes an ordinary transformer to compress its history into a compact belief state, with no architecture changes. It covers why transformers lack the built-in compression that recurrent networks get from a fixed-size state, using the Manhattan taxi study where models reached 100 percent next-turn accuracy while their internal street maps were incoherent. It also covers the Clever Hans failure of myopic next-token training and the earlier fixes from the same group. The Belief State Transformer offers a guarantee at more than double the parameters, and joint multi-token prediction is cheaper but depends on an unknown k-observability horizon. NextLat borrows from reinforcement learning by training a small dynamics network to predict the next hidden state from the current state and token, which also allows self-speculative decoding with flexible draft lengths. The discussion then turns to the theorem behind the method and the conditions needed for the latents to converge to belief states.

Sources:

1. Next-Latent Prediction Transformers Learn Compact World Models — Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford, 2025

http://arxiv.org/abs/2511.05963

2. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning — Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, et al., 2020

https://scholar.google.com/scholar?q=Bootstrap+Your+Own+Latent%3A+A+New+Approach+to+Self-Supervised+Learning

3. Data-Efficient Reinforcement Learning with Self-Predictive Representations — Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, Philip Bachman, 2021

https://scholar.google.com/scholar?q=Data-Efficient+Reinforcement+Learning+with+Self-Predictive+Representations

4. Better & Faster Large Language Models via Multi-token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve, 2024

https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-token+Prediction

5. A Path Towards Autonomous Machine Intelligence (JEPA position paper) — Yann LeCun, 2022

https://scholar.google.com/scholar?q=A+Path+Towards+Autonomous+Machine+Intelligence+%28JEPA+position+paper%29

6. Planning and Acting in Partially Observable Stochastic Domains — Leslie Pack Kaelbling, Michael L. Littman, Anthony R. Cassandra, 1998

https://scholar.google.com/scholar?q=Planning+and+Acting+in+Partially+Observable+Stochastic+Domains

7. Predictive Representations of State — Michael L. Littman, Richard S. Sutton, Satinder Singh, 2001

https://scholar.google.com/scholar?q=Predictive+Representations+of+State

8. The Belief State Transformer — Edward S. Hu, Kwangjun Ahn, Qinghua Liu, Haoran Xu, Manan Tomar, Ada Langford, Dinesh Jayaraman, Alex Lamb, John Langford, 2024

https://scholar.google.com/scholar?q=The+Belief+State+Transformer

9. Transformers Represent Belief State Geometry in their Residual Stream — Adam S. Shai, Sarah E. Marzen, Lucas Teixeira, Alexander Gietelink Oldenziel, Paul M. Riechers, 2024

https://scholar.google.com/scholar?q=Transformers+Represent+Belief+State+Geometry+in+their+Residual+Stream

10. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023

https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding

11. Accelerating Large Language Model Decoding with Speculative Sampling — Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper, 2023

https://scholar.google.com/scholar?q=Accelerating+Large+Language+Model+Decoding+with+Speculative+Sampling

12. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao, 2024

https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads

13. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE%3A+Speculative+Sampling+Requires+Rethinking+Feature+Uncertainty

14. Efficient Joint Prediction of Multiple Future Tokens (JTP) — Kwangjun Ahn, Alex Lamb, John Langford, 2025

https://scholar.google.com/scholar?q=Efficient+Joint+Prediction+of+Multiple+Future+Tokens+%28JTP%29

15. Bridging State and History Representations: Understanding Self-Predictive RL — Tianwei Ni, Benjamin Eysenbach, Erfan SeyedSalehi, Michel Ma, Clement Gehring, Aditya Mahajan, Pierre-Luc Bacon, 2024

https://scholar.google.com/scholar?q=Bridging+State+and+History+Representations%3A+Understanding+Self-Predictive+RL

16. The Pitfalls of Next-Token Prediction — Gregor Bachmann, Vaishnavh Nagarajan, 2024

https://scholar.google.com/scholar?q=The+Pitfalls+of+Next-Token+Prediction

17. Evaluating the World Model Implicit in a Generative Model — Keyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon Kleinberg, Sendhil Mullainathan, 2024

https://scholar.google.com/scholar?q=Evaluating+the+World+Model+Implicit+in+a+Generative+Model

18. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty / EAGLE-2 — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024

https://scholar.google.com/scholar?q=EAGLE%3A+Speculative+Sampling+Requires+Rethinking+Feature+Uncertainty+%2F+EAGLE-2

19. Data-Efficient Reinforcement Learning with Self-Predictive Representations (SPR) — Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, Philip Bachman, 2021

https://scholar.google.com/scholar?q=Data-Efficient+Reinforcement+Learning+with+Self-Predictive+Representations+%28SPR%29

20. The Parallelism Tradeoff: Limitations of Log-Precision Transformers — William Merrill, Ashish Sabharwal, 2023

https://scholar.google.com/scholar?q=The+Parallelism+Tradeoff%3A+Limitations+of+Log-Precision+Transformers

21. The Illusion of State in State-Space Models — William Merrill, Jackson Petty, Ashish Sabharwal, 2025

https://scholar.google.com/scholar?q=The+Illusion+of+State+in+State-Space+Models

22. LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architectures — Hai Huang, Yann LeCun, Randall Balestriero, 2025

https://scholar.google.com/scholar?q=LLM-JEPA%3A+Large+Language+Models+Meet+Joint+Embedding+Predictive+Architectures

23. Emergent Representations of Program Semantics / Transformers Represent Belief State Geometry in their Residual Stream — Adam S. Shai, Sarah E. Marzen, Lucas Teixeira, Alexander Gietelmark Oldenziel, Paul M. Riechers, 2024

https://scholar.google.com/scholar?q=Emergent+Representations+of+Program+Semantics+%2F+Transformers+Represent+Belief+State+Geometry+in+their+Residual+Stream

24. Multi-token prediction in DeepSeek-V3 (Technical Report) — DeepSeek-AI (Aixin Liu et al.), 2024

https://scholar.google.com/scholar?q=Multi-token+prediction+in+DeepSeek-V3+%28Technical+Report%29

Interactive Visualization: Next-Latent Prediction Lets Transformers Learn Compact World Models

This episode dissects the BASED architecture from Arora, Eyuboglu, Zhang et al. (Stanford/Buffalo), examining how it tries to resolve the tradeoff between recall accuracy and inference throughput in sequence models. The discussion traces the lineage of alternatives to standard softmax attention — state space models like Mamba, gated-convolution approaches like H3 and Hyena, linear attention, and sliding window attention — explaining why a growing KV-cache makes attention memory-bound at scale, while fixed-size-state models trade away precise recall by construction. It highlights the MQAR benchmark from the Zoology paper as a tool for exposing this recall-versus-state-size Pareto frontier, showing that every architecture, including full attention, sits somewhere on that curve rather than escaping it. The hosts then unpack BASED's hybrid design, which combines a linear attention component for cheap global context with a small sliding window for exact local comparisons, and start evaluating whether this combination actually delivers on its claimed 24x throughput gain over FlashAttention-2. Listeners interested in efficient LLM inference will get a clear framework for why architecture choices around memory and recall are fundamentally linked rather than independently solvable engineering problems.

Sources:

1. Simple linear attention language models balance the recall-throughput tradeoff — Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, Christopher Ré, 2024

http://arxiv.org/abs/2402.18668

2. Zoology: Measuring and Improving Recall in Efficient Language Models — Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, Christopher Ré, 2023

https://scholar.google.com/scholar?q=Zoology%3A+Measuring+and+Improving+Recall+in+Efficient+Language+Models

3. The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry — Michael Zhang, Kush Bhatia, Hermann Kumbong, Christopher Ré, 2024

https://scholar.google.com/scholar?q=The+Hedgehog+%26+the+Porcupine%3A+Expressive+Linear+Attentions+with+Softmax+Mimicry

4. Gated Linear Attention Transformers with Hardware-Efficient Training — Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim, 2023

https://scholar.google.com/scholar?q=Gated+Linear+Attention+Transformers+with+Hardware-Efficient+Training

5. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023

https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces

6. Mistral 7B — Albert Q. Jiang et al., 2023

https://scholar.google.com/scholar?q=Mistral+7B

This episode examines SALI, a 2024 SIGMOD systems paper diagnosing why learned database indexes like ALEX and LIPP see throughput drop as thread counts rise, rather than scale with additional concurrency. The discussion traces the lineage from Google's 2018 Recursive Model Index through the buffer-based and model-based approaches that emerged to handle inserts, focusing on the shift-versus-chain tradeoff between ALEX's coarse-grained locking and LIPP's fine-grained, chain-based node design. The key finding is that fine-grained locking solves data contention but not statistics contention — shared counters tracking when a node needs reorganization become a cacheline-thrashing bottleneck under heavy concurrent writes. Listeners interested in database internals or concurrent data structures will find the breakdown of exactly where and why two well-regarded designs fail at scale particularly compelling, especially the setup for SALI's proposed fix using self-adapting nodes instead of shared bottleneck counters.

Sources:

1. SALI: A Scalable Adaptive Learned Index Framework based on Probability Models — Jiake Ge, Huanchen Zhang, Boyu Shi, Yuanhui Luo, Yunda Guo, Yunpeng Chai, Yuxing Chen, Anqun Pan, 2023

http://arxiv.org/abs/2308.15012

2. The Case for Learned Index Structures — Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis, 2018

https://scholar.google.com/scholar?q=The+Case+for+Learned+Index+Structures

3. ALEX: An Updatable Adaptive Learned Index — Jialin Ding, Umar Farooq Minhas, Jia Yu, Chi Wang, Jaeyoung Do, Yinan Li, Hantian Zhang, Badrish Chandramouli, Johannes Gehrke, Donald Kossmann, David Lomet, Tim Kraska, 2020

https://scholar.google.com/scholar?q=ALEX%3A+An+Updatable+Adaptive+Learned+Index

4. Updatable Learned Index with Precise Positions (LIPP) — Jiacheng Wu, Yong Zhang, Shimin Chen, Jin Wang, Yu Chen, Chunxiao Xing, 2021

https://scholar.google.com/scholar?q=Updatable+Learned+Index+with+Precise+Positions+%28LIPP%29

5. XIndex: A Scalable Learned Index for Multicore Data Storage — Chuzhe Tang, Youyun Wang, Zhiyuan Dong, Gansen Hu, Zhaoguo Wang, Minjie Wang, Haibo Chen, 2020

https://scholar.google.com/scholar?q=XIndex%3A+A+Scalable+Learned+Index+for+Multicore+Data+Storage

6. Are Updatable Learned Indexes Ready? — Chaichon Wongkham, Baotong Lu, Chris Liu, Zhicong Zhong, Eric Lo, Tianzheng Wang, 2022

https://scholar.google.com/scholar?q=Are+Updatable+Learned+Indexes+Ready%3F

7. Adaptive Hybrid Indexes — Christoph Anneser, Andreas Kipf, Huanchen Zhang, Thomas Neumann, Alfons Kemper, 2022

https://scholar.google.com/scholar?q=Adaptive+Hybrid+Indexes

8. DILI: A Distribution-Driven Learned Index — Pengfei Li, Hua Lu, Rong Zhu, Bolin Ding, Long Yang, Gang Pan, 2023

https://scholar.google.com/scholar?q=DILI%3A+A+Distribution-Driven+Learned+Index

9. Updatable Learned Indexes Meet Disk-Resident DBMS: From Evaluations to Design Choices — Hai Lan, Zhifeng Bao, J. Shane Culpepper, Renata Borovica-Gajic, 2023

https://scholar.google.com/scholar?q=Updatable+Learned+Indexes+Meet+Disk-Resident+DBMS%3A+From+Evaluations+to+Design+Choices

Interactive Visualization: SALI: Fixing Concurrency Bottlenecks in Learned Indexes