Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Stable Deep RL via Gaussian RepresentationsThis episode explores a 2026 paper on stabilizing deep reinforcement learning by pushing an agent’s hidden representations toward an isotropic Gaussian shape. It explains how nonstationarity in RL, from shifting data distributions, bootstrapped targets, and primacy bias, can make agents overfit early experience, lose plasticity, and accumulate dormant neurons. The discussion focuses on the paper’s core argument that a round, evenly used feature space makes linear readouts easier to keep tracking as targets drift, reducing collapse and improving adaptation, and it breaks down SIGReg as a lightweight way to enforce that geometry. Listeners would find it interesting because it links an abstract idea from representation geometry to a concrete engineering problem in making RL systems more stable and trainable.Sources:1. Stable Deep Reinforcement Learning via Isotropic Gaussian Representations — Ali Saheb Pasand, Johan Obando-Ceron, Aaron Courville, Pouya Bashivan, Pablo Samuel Castro, 2026http://arxiv.org/abs/2602.193732. Whitening for Self-Supervised Representation Learning — Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe, 2020https://scholar.google.com/scholar?q=Whitening+for+Self-Supervised+Representation+Learning3. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning4. The Dormant Neuron Phenomenon in Deep Reinforcement Learning — Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, Utku Evci, 2023https://scholar.google.com/scholar?q=The+Dormant+Neuron+Phenomenon+in+Deep+Reinforcement+Learning5. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — Randall Balestriero, Yann LeCun, 2025https://scholar.google.com/scholar?q=LeJEPA%3A+Provable+and+Scalable+Self-Supervised+Learning+Without+the+Heuristics6. The Primacy Bias in Deep Reinforcement Learning — Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, Aaron Courville, 2022https://scholar.google.com/scholar?q=The+Primacy+Bias+in+Deep+Reinforcement+Learning7. No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO — Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, Caglar Gulcehre, 2024https://scholar.google.com/scholar?q=No+Representation%2C+No+Trust%3A+Connecting+Representation%2C+Collapse%2C+and+Trust+Issues+in+PPO8. Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning — Roger Creus Castanyer, Johan Obando-Ceron, Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, Pablo Samuel Castro, 2025https://scholar.google.com/scholar?q=Stable+Gradients+for+Stable+Learning+at+Scale+in+Deep+Reinforcement+Learning9. Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks — Jesse Farebrother et al., 2023https://scholar.google.com/scholar?q=Proto-Value+Networks%3A+Scaling+Representation+Learning+with+Auxiliary+Tasks10. Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics — Raphael Bernas et al., 2026https://scholar.google.com/scholar?q=Revisiting+Anisotropy+in+Language+Transformers%3A+The+Geometry+of+Learning+Dynamics11. Emergence of Quantised Representations Isolated to Anisotropic Functions — George Bird, 2025https://scholar.google.com/scholar?q=Emergence+of+Quantised+Representations+Isolated+to+Anisotropic+Functions12. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp313. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp30 min
- EMO: Emergent Modularity for Mixture-of-ExpertsHal Turing and Dr. Ada Shannon take a deep dive into EMO: Pretraining Mixture of Experts for Emergent Modularity, a May 7, 2026 paper by Ryan Wang and co-authors from UC Berkeley and the Allen Institute for AI. The episode centers on a practical deployment question: if a workload is mostly code, math, or biomed, why must operators keep an entire giant model in memory instead of loading only the relevant slice? They frame EMO against the broader rise of sparse Mixture-of-Experts systems and explain why industry progress on active-parameter efficiency is not the same as delivering clean, domain-specific modules that can stand on their own at inference time.The discussion carefully separates standard MoE behavior from the stronger notion of modularity that EMO is targeting. Hal and Ada walk through how sparse-gated MoE and Switch Transformer style routing already allow different tokens to activate different experts, but argue that this still leaves deployment looking monolithic because the router makes local token-level decisions rather than exposing stable task-level components. A biology prompt can still scatter across a messy set of experts, and the next sentence may hit a different set entirely. The hosts use that distinction to unpack the paper’s core concepts: emergent modularity from unlabeled data, semantic expert specialization around meaningful domains like code or math, composable architecture, and the memory-accuracy frontier that determines whether smaller loaded expert pools can preserve real capability.The episode then gets into EMO’s training design and why the method is more than a single routing tweak. Ada explains the paper’s two-level routing scheme, where a document first selects a shared candidate pool of experts and individual tokens then choose active experts only within that pool, forcing document-consistent structure without removing all local flexibility. They also cover the supporting recipe: random pool-size sampling to expose the model to different memory budgets during training, global load balancing so a few experts do not dominate usage, and document-length-aware training so very long documents do not overwhelm the learning signal. The result is a focused discussion of whether MoE pretraining can produce expert groups that are not just sparsely activated, but genuinely deployable as modular tools.Sources:1. EMO: Emergent Modularity for Mixture-of-Expertshttps://allenai.org/papers/emo2. DEMix Layers: Disentangling Domains for Modular Language Modeling — Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, Luke Zettlemoyer, 2021https://scholar.google.com/scholar?q=DEMix+Layers%3A+Disentangling+Domains+for+Modular+Language+Modeling3. ModuleFormer: Modularity Emerges from Mixture-of-Experts — Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, Chuang Gan, 2023https://scholar.google.com/scholar?q=ModuleFormer%3A+Modularity+Emerges+from+Mixture-of-Experts4. OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models — Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, Yang You, 2024https://scholar.google.com/scholar?q=OpenMoE%3A+An+Early+Effort+on+Open+Mixture-of-Experts+Language+Models5. EMO: Pretraining Mixture of Experts for Emergent Modularity — Ryan Wang, Akshita Bhagia, Sewon Min, 2026https://scholar.google.com/scholar?q=EMO%3A+Pretraining+Mixture+of+Experts+for+Emergent+Modularity6. FlexOlmo: Open Language Models for Flexible Data Use — Weijia Shi et al., 2025https://scholar.google.com/scholar?q=FlexOlmo%3A+Open+Language+Models+for+Flexible+Data+Use7. Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM — Sainbayar Sukhbaatar et al., 2024https://scholar.google.com/scholar?q=Branch-Train-MiX%3A+Mixing+Expert+LLMs+into+a+Mixture-of-Experts+LLM8. The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise — Xi Wang, Soufiane Hayou, Eric Nalisnick, 2026https://scholar.google.com/scholar?q=The+Myth+of+Expert+Specialization+in+MoEs%3A+Why+Routing+Reflects+Geometry%2C+Not+Necessarily+Domain+Expertise9. Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations — Zican Dong et al., 2025https://scholar.google.com/scholar?q=Domain-Specific+Pruning+of+Large+Mixture-of-Experts+Models+with+Few-shot+Demonstrations10. Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs — Enshu Liu et al., 2024https://scholar.google.com/scholar?q=Efficient+Expert+Pruning+for+Sparse+Mixture-of-Experts+Language+Models%3A+Enhancing+Performance+and+Reducing+Inference+Costs11. MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router — Yanyue Xie et al., 2024https://scholar.google.com/scholar?q=MoE-Pruner%3A+Pruning+Mixture-of-Experts+Large+Language+Model+using+the+Hints+from+Its+Router12. AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models — Zihao Zeng et al., 2024https://scholar.google.com/scholar?q=AdaMoE%3A+Token-Adaptive+Routing+with+Null+Experts+for+Mixture-of-Experts+Language+Models13. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai et al., 2024https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models14. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev and Denis Mazur, 2023https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading15. Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding — Zhibin Wang et al., 2025https://scholar.google.com/scholar?q=Accelerating+Mixture-of-Experts+Inference+by+Hiding+Offloading+Latency+with+Speculative+Decoding16. AI Post Transformers: EMO: Emergent Modularity in Sparse Language Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-06-emo-emergent-modularity-in-sparse-langua-9551c4.mp317. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp318. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp319. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3Interactive Visualization: EMO: Emergent Modularity for Mixture-of-Experts0 min
- Modeling Financial Habits with Transaction TransformersThis episode explores how a transformer trained on raw bank transaction histories can model customer behavior for financial product recommendation, and why that may outperform pipelines built from hand-engineered tabular features alone. It explains the paper’s core idea of turning each transaction into a tokenized sequence that mixes inflow or outflow, amount buckets, calendar signals, source metadata, and natural-language merchant descriptions, then pretraining the model with self-supervised learning to produce reusable customer embeddings. The discussion argues that transaction text and long-range patterns such as pay cycles, bill timing, and abrupt behavior changes carry signal that conventional tabular systems often flatten away, while a practical deployment can still combine learned embeddings with legacy banking features downstream. A listener would find it interesting because it connects transformer-style representation learning to a concrete banking use case and shows how foundation-model ideas can be adapted to messy, real-world financial behavior.Sources:1. Your Spending Needs Attention: Modeling Financial Habits with Transformers — D. T. Braithwaite, Misael Cavalcanti, R. Austin McEver, Hiroto Udagawa, Daniel Silva, Rohan Ramanath, Felipe Meneses, Arissa Yoshida, Evan Wingert, Matheus Ramos, Brian Zanfelice, Aman Gupta, 2025http://arxiv.org/abs/2507.232672. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, 2019https://scholar.google.com/scholar?q=BERT%3A+Pre-training+of+Deep+Bidirectional+Transformers+for+Language+Understanding3. CoLES: Contrastive Learning for Event Sequences with Self-Supervision — Dmitrii Babaev, Ivan Kireev, Nikita Ovsov, Mariya Ivanova, Gleb Gusev, Ivan Nazarov, Alexander Tuzhilin, 2020https://scholar.google.com/scholar?q=CoLES%3A+Contrastive+Learning+for+Event+Sequences+with+Self-Supervision4. Dynamic Customer Embeddings for Financial Service Applications — Nima Chitsazan, Samuel Sharpe, Dwipam Katariya, Qianyu Cheng, Karthik Rajasethupathy, 2021https://scholar.google.com/scholar?q=Dynamic+Customer+Embeddings+for+Financial+Service+Applications5. Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences — Piotr Skalski, David Sutton, Stuart Burrell, Iker Perez, Jason Wong, 2024https://scholar.google.com/scholar?q=Towards+a+Foundation+Purchasing+Model%3A+Pretrained+Generative+Autoregression+on+Transaction+Sequences6. Self-Attentive Sequential Recommendation — Wang-Cheng Kang, Julian McAuley, 2018https://scholar.google.com/scholar?q=Self-Attentive+Sequential+Recommendation7. BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer — Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang, 2019https://scholar.google.com/scholar?q=BERT4Rec%3A+Sequential+Recommendation+with+Bidirectional+Encoder+Representations+from+Transformer8. S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization — Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, Ji-Rong Wen, 2020https://scholar.google.com/scholar?q=S%5E3-Rec%3A+Self-Supervised+Learning+for+Sequential+Recommendation+with+Mutual+Information+Maximization9. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba — Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, Wenwu Ou, 2019https://scholar.google.com/scholar?q=Behavior+Sequence+Transformer+for+E-commerce+Recommendation+in+Alibaba10. Text Is All You Need: Learning Language Representations for Sequential Recommendation — Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, Julian McAuley, 2023https://scholar.google.com/scholar?q=Text+Is+All+You+Need%3A+Learning+Language+Representations+for+Sequential+Recommendation11. PinnerFormer: Sequence Modeling for User Representation at Pinterest — Nikil Pancha, Andrew Zhai, Jure Leskovec, Charles Rosenberg, 2022https://scholar.google.com/scholar?q=PinnerFormer%3A+Sequence+Modeling+for+User+Representation+at+Pinterest12. Mamba4Rec: Towards Efficient Sequential Recommendation with Selective State Space Models — Chengkai Liu et al., 2024https://scholar.google.com/scholar?q=Mamba4Rec%3A+Towards+Efficient+Sequential+Recommendation+with+Selective+State+Space+Models13. SSD4Rec: A Structured State Space Duality Model for Efficient Sequential Recommendation — Haohao Qu et al., 2024https://scholar.google.com/scholar?q=SSD4Rec%3A+A+Structured+State+Space+Duality+Model+for+Efficient+Sequential+Recommendation14. DynLLM: When Large Language Models Meet Dynamic Graph Recommendation — Ziwei Zhao et al., 2024https://scholar.google.com/scholar?q=DynLLM%3A+When+Large+Language+Models+Meet+Dynamic+Graph+Recommendation15. Personalized Elastic Embedding Learning for On-Device Recommendation — Ruiqi Zheng et al., 2023https://scholar.google.com/scholar?q=Personalized+Elastic+Embedding+Learning+for+On-Device+Recommendation16. A Survey on Deep Tabular Learning — Shriyank Somvanshi et al., 2024https://scholar.google.com/scholar?q=A+Survey+on+Deep+Tabular+Learning17. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp318. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp30 min
- TransactionGPT as a Payments Foundation ModelThis episode explores TransactionGPT, a Visa Research paper that argues for a foundation-model approach to consumer transaction data spanning generation, anomaly detection, and representation learning. It explains why payment histories are fundamentally different from text or simple time series: each event mixes merchant IDs, amounts, timestamps, and engineered risk signals, creating a multi-modal, temporal, tabular structure that demands more schema-aware modeling. The discussion walks through the paper’s 1D, 2D, and 3D architecture progression, highlighting how separate transformers for transaction metadata, downstream features, and behavioral sequences aim to avoid the pitfalls of flattening everything into a single embedding space. Listeners would find it interesting for its clear debate over whether TransactionGPT is a genuine reusable backbone for payments or mainly a strong engineering response to real-world constraints like heterogeneity, scale, regulation, and low-latency fraud decisioning.Sources:1. TransactionGPT — Yingtong Dou, Zhimeng Jiang, Tianyi Zhang, Mingzhi Hu, Zhichao Xu, Shubham Jain, Uday Singh Saini, Xiran Fan, Jiarui Sun, Menghai Pan, Junpeng Wang, Xin Dai, Liang Wang, Chin-Chia Michael Yeh, Yujie Fan, Yan Zheng, Vineeth Rakesh, Huiyuan Chen, Guanchu Wang, Mangesh Bendre, Zhongfang Zhuang, Xiaoting Li, Prince Aboagye, Vivian Lai, Minghua Xu, Hao Yang, Yiwei Cai, Mahashweta Das, Yuzhong Chen, 2025http://arxiv.org/abs/2511.089392. TabDPT: Scaling Tabular Foundation Models on Real Data — Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Alex Labach, Hamidreza Kamkari, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini, Maksims Volkovs, 2024https://arxiv.org/abs/2410.181643. Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences — Piotr Skalski, David Sutton, Stuart Burrell, Iker Perez, Jason Wong, 2024https://arxiv.org/abs/2401.016414. Open Banking Foundational Model: Learning Language Representations from Few Financial Transactions — Gustavo Polleti, Marlesson Santana, Eduardo Fontes, 2025https://arxiv.org/abs/2511.121545. TransactionGPT — Yingtong Dou, Zhimeng Jiang, Tianyi Zhang, Mingzhi Hu, Zhichao Xu, Shubham Jain, Uday Singh Saini, Xiran Fan, Jiarui Sun, Menghai Pan, Junpeng Wang, Xin Dai, Liang Wang, Chin-Chia Michael Yeh, Yujie Fan, Vineeth Rakesh, Huiyuan Chen, Mangesh Bendre, Zhongfang Zhuang, Xiaoting Li, Prince Aboagye, Vivian Lai, Minghua Xu, Hao Yang, Yiwei Cai, Mahashweta Das, Yuzhong Chen, 2025https://arxiv.org/abs/2511.089396. TabTransformer: Tabular Data Modeling Using Contextual Embeddings — Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar Karnin, 2020https://arxiv.org/abs/2012.066787. Tabular Transformers for Modeling Multivariate Time Series — Inkit Padhi, Yair Schiff, Igor Melnyk, Mattia Rigotti, Youssef Mroueh, Pierre Dognin, Jerret Ross, Ravi Nair, Erik Altman, 2021https://arxiv.org/abs/2011.018438. FATA-Trans: Field And Time-Aware Transformer for Sequential Tabular Data — Dongyu Zhang, Liang Wang, Xin Dai, Shubham Jain, Junpeng Wang, Yujie Fan, Chin-Chia Michael Yeh, Yan Zheng, Zhongfang Zhuang, Wei Zhang, 2023https://arxiv.org/abs/2310.138189. Multi-modal Time Series Analysis: A Tutorial and Survey — Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, Dongjin Song, 2025https://arxiv.org/abs/2503.1370910. Credit card fraud detection using machine learning: A survey — Yvan Lucas, Johannes Jurgovsky, 2020https://arxiv.org/abs/2010.0647911. Interleaved Sequence RNNs for Fraud Detection — Bernardo Branco, Pedro Abreu, Ana Sofia Gomes, Mariana S. C. Almeida, João Tiago Ascensão, Pedro Bizarro, 2020https://arxiv.org/abs/2002.0598812. FraudTransformer: Time-Aware GPT for Transaction Fraud Detection — Gholamali Aminian, Andrew Elliott, Tiger Li, Timothy Cheuk Hin Wong, Victor Claude Dehon, Lukasz Szpruch, Carsten Maple, Christopher Read, Martin Brown, Gesine Reinert, Mo Mamouei, 2025https://arxiv.org/abs/2509.2371213. Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings — Xiran Fan, Zhimeng Jiang, Chin-Chia Michael Yeh, Yuzhong Chen, Yingtong Dou, Menghai Pan, Yan Zheng, 2025https://scholar.google.com/scholar?q=Enhancing+Foundation+Models+in+Transaction+Understanding+with+LLM-based+Sentence+Embeddings14. Self-Attentive Sequential Recommendation — Wang-Cheng Kang, Julian McAuley, 2018https://scholar.google.com/scholar?q=Self-Attentive+Sequential+Recommendation15. TabLLM: Few-shot Classification of Tabular Data with Large Language Models — Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, David Sontag, 2022https://scholar.google.com/scholar?q=TabLLM%3A+Few-shot+Classification+of+Tabular+Data+with+Large+Language+Models16. TimeGPT-1 — Azul Garza, Cristian Challu, Max Mergenthaler-Canseco, 2023https://scholar.google.com/scholar?q=TimeGPT-117. ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning — Junguang Jiang et al., 2023https://scholar.google.com/scholar?q=ForkMerge%3A+Mitigating+Negative+Transfer+in+Auxiliary-Task+Learning18. Identification of Negative Transfers in Multitask Learning Using Surrogate Models — Dongyue Li, Huy L. Nguyen, Hongyang R. Zhang, 2023https://scholar.google.com/scholar?q=Identification+of+Negative+Transfers+in+Multitask+Learning+Using+Surrogate+Models19. Enriching Tabular Data with Contextual LLM Embeddings: A Comprehensive Ablation Study for Ensemble Classifiers — Gjergji Kasneci, Enkelejda Kasneci, 2024https://scholar.google.com/scholar?q=Enriching+Tabular+Data+with+Contextual+LLM+Embeddings%3A+A+Comprehensive+Ablation+Study+for+Ensemble+Classifiers20. LLM Embeddings Improve Test-time Adaptation to Tabular Y|X-Shifts — Yibo Zeng et al., 2024https://scholar.google.com/scholar?q=LLM+Embeddings+Improve+Test-time+Adaptation+to+Tabular+Y%7CX-Shifts21. LLM Embeddings for Deep Learning on Tabular Data — Boshko Koloski et al., 2025https://scholar.google.com/scholar?q=LLM+Embeddings+for+Deep+Learning+on+Tabular+Data22. TableGPT2: A Large Multimodal Model with Tabular Data Integration — Aofeng Su et al., 2024https://scholar.google.com/scholar?q=TableGPT2%3A+A+Large+Multimodal+Model+with+Tabular+Data+Integration23. AI Post Transformers: KumoRFM for In-Context Relational Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-11-kumorfm-for-in-context-relational-learni-520d2b.mp324. AI Post Transformers: Relational Graph Transformer for Multi-Table Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-11-relational-graph-transformer-for-multi-t-57cce3.mp325. AI Post Transformers: Unembedding Matrices as Feature Lenses for Embeddings — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-08-unembedding-matrices-as-feature-lenses-f-4dc415.mp3Interactive Visualization: TransactionGPT as a Payments Foundation Model0 min
- When LeJEPA Truly Learns a World ModelThis episode explores the paper When Does LeJEPA Learn a World Model? and uses it to examine what should count as a genuine world model in latent predictive learning, contrasting JEPA-style representation prediction with generative reconstruction. It explains why good probe scores are not enough: the real standard is linear identifiability, where a single global linear map recovers the environment’s hidden state well enough to support planning and compositional generalization. The discussion centers on the paper’s main theorem that, under stationary additive-noise dynamics with Gaussian latent variables, LeJEPA’s alignment objective plus SIGReg recovers the true latent state up to an orthogonal rotation, and on the sharper converse result that this universal guarantee fails for non-Gaussian latents. Listeners get a rigorous argument for when latent models are truly learning the world’s coordinates instead of merely extracting features that happen to be useful on downstream tasks.Sources:1. When LeJEPA Truly Learns a World Modelhttps://arxiv.org/pdf/2605.263792. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations — Francesco Locatello, Stefan Bauer, Mario Lucic, et al., 2018https://scholar.google.com/scholar?q=Challenging+Common+Assumptions+in+the+Unsupervised+Learning+of+Disentangled+Representations3. Variational Autoencoders and Nonlinear ICA: A Unifying Framework — Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, Aapo Hyvarinen, 2019https://scholar.google.com/scholar?q=Variational+Autoencoders+and+Nonlinear+ICA%3A+A+Unifying+Framework4. On Linear Identifiability of Learned Representations — Geoffrey Roeder, Luke Metz, Diederik P. Kingma, 2020https://scholar.google.com/scholar?q=On+Linear+Identifiability+of+Learned+Representations5. Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning — Aapo Hyvarinen, Ilyes Khemakhem, Hiroshi Morioka, 2023https://scholar.google.com/scholar?q=Nonlinear+Independent+Component+Analysis+for+Principled+Disentanglement+in+Unsupervised+Deep+Learning6. Auto-Encoding Variational Bayes — Diederik P. Kingma, Max Welling, 2013https://scholar.google.com/scholar?q=Auto-Encoding+Variational+Bayes7. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning8. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — Randall Balestriero, Yann LeCun, 2025https://scholar.google.com/scholar?q=LeJEPA%3A+Provable+and+Scalable+Self-Supervised+Learning+Without+the+Heuristics9. When Does LeJEPA Learn a World Model? — David Klindt, Yann LeCun, Randall Balestriero, 2026https://scholar.google.com/scholar?q=When+Does+LeJEPA+Learn+a+World+Model%3F10. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels — Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, Randall Balestriero, 2026https://scholar.google.com/scholar?q=LeWorldModel%3A+Stable+End-to-End+Joint-Embedding+Predictive+Architecture+from+Pixels11. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning — Mido Assran et al., 2025https://scholar.google.com/scholar?q=V-JEPA+2%3A+Self-Supervised+Video+Models+Enable+Understanding%2C+Prediction+and+Planning12. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning — Aapo Hyvarinen, Hiroaki Sasaki, Richard E. Turner, 2018https://scholar.google.com/scholar?q=Nonlinear+ICA+Using+Auxiliary+Variables+and+Generalized+Contrastive+Learning13. Joint Embedding Predictive Architectures Focus on Slow Features — Vlad Sobal, Jyothir S V, Siddhartha Jalagam, Nicolas Carion, Kyunghyun Cho, Yann LeCun, 2022https://scholar.google.com/scholar?q=Joint+Embedding+Predictive+Architectures+Focus+on+Slow+Features14. Cross-Entropy Is All You Need To Invert the Data Generating Process — Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, David Klindt, 2024https://scholar.google.com/scholar?q=Cross-Entropy+Is+All+You+Need+To+Invert+the+Data+Generating+Process15. Identifiability of latent-variable and structural-equation models: from linear to nonlinear — Aapo Hyvarinen, Ilyes Khemakhem, Ricardo Monti, 2023https://scholar.google.com/scholar?q=Identifiability+of+latent-variable+and+structural-equation+models%3A+from+linear+to+nonlinear16. On the Identifiability of Sparse ICA without Assuming Non-Gaussianity — Ignavier Ng, Yujia Zheng, Xinshuai Dong, Kun Zhang, 2024https://scholar.google.com/scholar?q=On+the+Identifiability+of+Sparse+ICA+without+Assuming+Non-Gaussianity17. Adaptive World Models: Learning Behaviors by Latent Imagination Under Non-Stationarity — Emiliyan Gospodinov, Vaisakh Shaj, Philipp Becker, Stefan Geyer, Gerhard Neumann, 2024https://scholar.google.com/scholar?q=Adaptive+World+Models%3A+Learning+Behaviors+by+Latent+Imagination+Under+Non-Stationarity18. Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors — Yong Liu, Chenyu Li, Jianmin Wang, Mingsheng Long, 2023https://scholar.google.com/scholar?q=Koopa%3A+Learning+Non-stationary+Time+Series+Dynamics+with+Koopman+Predictors19. Simplifying Latent Dynamics with Softly State-Invariant World Models — Tankred Saanum, Peter Dayan, Eric Schulz, 2024https://scholar.google.com/scholar?q=Simplifying+Latent+Dynamics+with+Softly+State-Invariant+World+Models20. Structured World Models from Human Videos — Russell Mendonca, Shikhar Bahl, Deepak Pathak, 2023https://scholar.google.com/scholar?q=Structured+World+Models+from+Human+Videos21. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp322. AI Post Transformers: Causal-JEPA for Object-Level World Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-causal-jepa-for-object-level-world-model-311a8b.mp323. AI Post Transformers: Learning Latent Action World Models from Video — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-learning-latent-action-world-models-from-1570a4.mp3Interactive Visualization: When LeJEPA Truly Learns a World Model0 min
- Benchmarking PEFT Techniques for Large Language ModelsHal Turing and Dr. Ada Shannon examine an empirical study of parameter-efficient fine-tuning for large language models, centered on a practical question: when does a small task-specific update beat retraining the entire model? Using FLAN-T5-XL as the test bed, they frame PEFT as a transfer-learning strategy that freezes most of the transformer while learning a compact adaptation layer, whether through LoRA’s low-rank weight updates, adapter-style modules, IA3 scaling vectors, BitFit bias updates, or learned soft prompts. The discussion keeps returning to the real systems tradeoff: quality matters, but so do training speed, storage cost, and the burden of maintaining separate model copies for many downstream tasks.The episode walks through the benchmark design in detail rather than treating PEFT as a vague category. The paper compares full tuning, LoRA, IA3, prompt tuning, and BitFit on the same backbone across classification tasks like AG News and CoLA, generation tasks like E2E and SAMSum, and data budgets of roughly 100, 1,000, and 10,000 examples. The hosts emphasize why those controls matter: same model, same stopping rule, and fixed method settings make it easier to see where each technique actually helps, while also limiting how far the results should be generalized to other architectures, especially decoder-only chat models.They then dig into the paper’s uneven but useful results. In low-resource settings, LoRA and BitFit frequently outperform full tuning, with LoRA posting a notably stronger CoLA score and BitFit leading on AG News, E2E, and SAMSum, while prompt tuning performs strikingly poorly on the generation benchmarks under this setup. In medium-resource settings, IA3, LoRA, and BitFit remain competitive, but at higher data scales full tuning starts reclaiming ground on some tasks even as LoRA and IA3 still win specific cases. The takeaway is not that one PEFT method universally dominates, but that the strengths and weaknesses of each approach shift with task type, data regime, and the exact adaptation recipe.Sources:1. Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs — George Pu, Anirudh Jain, Jihan Yin, Russell Kaplan, 2023http://arxiv.org/abs/2304.149992. Parameter-Efficient Transfer Learning for NLP — Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, et al., 2019https://arxiv.org/abs/1902.007513. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, et al., 2021https://arxiv.org/abs/2106.096854. Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning — Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, Anna Rumshisky, 2023https://arxiv.org/abs/2303.156475. Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs — George Pu, Anirudh Jain, Jihan Yin, Russell Kaplan, 2023https://arxiv.org/abs/2304.149996. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021https://arxiv.org/abs/2101.001907. The Power of Scale for Parameter-Efficient Prompt Tuning — Brian Lester, Rami Al-Rfou, Noah Constant, 2021https://arxiv.org/abs/2104.086918. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks — Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, Jie Tang, 2021https://arxiv.org/abs/2110.076029. SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer — Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, Daniel Cer, 2021https://arxiv.org/abs/2110.0790410. Revisiting Parameter-Efficient Tuning: Are We Really There Yet? — Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, Shangsong Liang, 2022https://scholar.google.com/scholar?q=Revisiting+Parameter-Efficient+Tuning%3A+Are+We+Really+There+Yet%3F11. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel, 2022https://scholar.google.com/scholar?q=Few-Shot+Parameter-Efficient+Fine-Tuning+is+Better+and+Cheaper+than+In-Context+Learning12. On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation — Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, Luo Si, 2021https://scholar.google.com/scholar?q=On+the+Effectiveness+of+Adapter-based+Tuning+for+Pretrained+Language+Model+Adaptation13. Scaling Instruction-Finetuned Language Models — Hyung Won Chung et al., 2022https://scholar.google.com/scholar?q=Scaling+Instruction-Finetuned+Language+Models14. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method — Biao Zhang, Zhongtao Liu, Colin Cherry, Orhan Firat, 2024https://arxiv.org/abs/2402.1719315. Parameter-Efficient Fine-Tuning Design Spaces — Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, Diyi Yang, 2023https://arxiv.org/abs/2301.0182116. AI Post Transformers: ForkKV for Multi-LoRA Agent Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-forkkv-for-multi-lora-agent-serving-ccafa4.mp317. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp318. AI Post Transformers: ZeRO-Offload: Democratizing Billion-Scale Model Training — Hal Turing & Dr. Ada Shannon, Fri,https://podcast.do-not-panic.com/episodes/zero-offload-democratizing-billion-scale-model-training/Interactive Visualization: Benchmarking PEFT Techniques for Large Language Models13 min
- Weak-SIGReg for Stable Vision Transformer TrainingThis episode explores Weak-SIGReg, a lightweight covariance regularizer designed to prevent representation collapse in fragile supervised training, especially for small-data Vision Transformers. It explains how the method uses a sketched covariance matrix and an identity-matching penalty to keep hidden features decorrelated and similarly scaled at much lower cost than full covariance regularization. The discussion centers on CIFAR-100 results, where Weak-SIGReg dramatically improves a deliberately unstable ViT setup and also boosts a plain MLP, while offering little change on an already stable ResNet18. It also digs into the paper’s main caveat: the biggest gains appear when the baseline training recipe is badly broken, so the most interesting question is not just whether the regularizer works, but when it adds real value beyond simply fixing optimization and initialization.Sources:1. Weak-SIGReg: Covariance Regularization for Stable Deep Learning — Habibullah Akbar, 2026http://arxiv.org/abs/2603.059242. Reducing Overfitting in Deep Networks by Decorrelating Representations — Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, Dhruv Batra, 2015https://scholar.google.com/scholar?q=Reducing+Overfitting+in+Deep+Networks+by+Decorrelating+Representations3. Barlow Twins: Self-Supervised Learning via Redundancy Reduction — Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, Stephane Deny, 2021https://scholar.google.com/scholar?q=Barlow+Twins%3A+Self-Supervised+Learning+via+Redundancy+Reduction4. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Adrien Bardes, Jean Ponce, Yann LeCun, 2021https://scholar.google.com/scholar?q=VICReg%3A+Variance-Invariance-Covariance+Regularization+for+Self-Supervised+Learning5. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics — Randall Balestriero, Yann LeCun, 2025https://scholar.google.com/scholar?q=LeJEPA%3A+Provable+and+Scalable+Self-Supervised+Learning+Without+the+Heuristics6. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al., 2021https://scholar.google.com/scholar?q=An+Image+is+Worth+16x16+Words%3A+Transformers+for+Image+Recognition+at+Scale7. Training data-efficient image transformers & distillation through attention — Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Herve Jegou, 2021https://scholar.google.com/scholar?q=Training+data-efficient+image+transformers+%26+distillation+through+attention8. How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers — Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, Lucas Beyer, 2021https://scholar.google.com/scholar?q=How+to+train+your+ViT%3F+Data%2C+Augmentation%2C+and+Regularization+in+Vision+Transformers9. Early Convolutions Help Transformers See Better — Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollar, Ross Girshick, 2021https://scholar.google.com/scholar?q=Early+Convolutions+Help+Transformers+See+Better10. Whitening for Self-Supervised Representation Learning — Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, Nicu Sebe, 2020https://arxiv.org/abs/2007.0634611. Understanding Dimensional Collapse in Contrastive Self-Supervised Learning — Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian, 2021https://arxiv.org/abs/2110.0934812. UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures — Triet M. Le, 2026https://arxiv.org/abs/2606.0144313. Stability of Transformers under Layer Normalization — Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Krishna Kumar, Markos A. Katsoulakis, 2025https://scholar.google.com/scholar?q=Stability+of+Transformers+under+Layer+Normalization14. Transformers without Normalization — Jiachen Zhu, Xinlei Chen, Kaiming He, Yann LeCun, Zhuang Liu, 2025https://scholar.google.com/scholar?q=Transformers+without+Normalization15. Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations — Yongyi Yang, Jacob Steinhardt, Wei Hu, 2023https://scholar.google.com/scholar?q=Are+Neurons+Actually+Collapsed%3F+On+the+Fine-Grained+Structure+in+Neural+Representations16. The Impact of Geometric Complexity on Neural Collapse in Transfer Learning — Michael Munn, Benoit Dherin, Javier Gonzalvo, 2024https://scholar.google.com/scholar?q=The+Impact+of+Geometric+Complexity+on+Neural+Collapse+in+Transfer+Learning17. AI Post Transformers: LeWorldModel: Stable Joint-Embedding World Models from Pixels — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-leworldmodel-stable-joint-embedding-worl-650f9f.mp318. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp311 min
- InfiniGen for Efficient Long-Context LLM InferenceThis episode explores InfiniGen, a systems approach to speeding up long-context language model inference by treating KV cache management, not raw compute, as the central bottleneck. It explains why decoding slows down when large caches have to shuttle between CPU and GPU memory, and contrasts InfiniGen with FlexGen, H2O, and PagedAttention to show how different serving setups create different memory problems. The discussion focuses on InfiniGen’s core idea: use a lightweight preview from the previous layer, along with offline-skewed query and key weights, to predict which exact cache entries will matter next and prefetch only those instead of moving the whole history. Listeners would find it interesting because the paper reports large practical gains, including up to 3x speedups and major accuracy improvements over weaker cache-selection methods, making it a concrete example of systems engineering reshaping how large models are served.Sources:1. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024http://arxiv.org/abs/2406.197072. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng et al., 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang et al., 2023https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models4. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention5. Efficient Streaming Language Models with Attention Sinks — Guangxuan Xiao et al., 2023https://scholar.google.com/scholar?q=Efficient+Streaming+Language+Models+with+Attention+Sinks6. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads — Guangxuan Xiao et al., 2024https://scholar.google.com/scholar?q=DuoAttention%3A+Efficient+Long-Context+LLM+Inference+with+Retrieval+and+Streaming+Heads7. SCBench: A KV Cache-Centric Analysis of Long-Context Methods — Yucheng Li et al., 2024https://scholar.google.com/scholar?q=SCBench%3A+A+KV+Cache-Centric+Analysis+of+Long-Context+Methods8. IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inference — Xintong Yang et al., 2026https://arxiv.org/abs/2605.254759. Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks — Zheng Wang et al., 2024https://arxiv.org/abs/2407.0845410. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — Coleman Hooper et al., 2024https://arxiv.org/abs/2401.1807911. TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization — Dingyu Yao et al., 2025https://arxiv.org/abs/2505.1958612. LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management — Yi Xiong et al., 2024https://arxiv.org/abs/2410.0042813. SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget — Zihao Wang et al., 2024https://arxiv.org/abs/2404.0479314. KeDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments — Junyoung Park et al., 2025https://arxiv.org/abs/2504.1536415. AI Post Transformers: IndexMem: Learned KV-Cache Eviction for Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-12-indexmem-learned-kv-cache-eviction-for-l-132c2a.mp316. AI Post Transformers: PackKV Lossy Compression for KV Caches — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-packkv-lossy-compression-for-kv-caches-b37bce.mp317. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp318. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp319. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp320. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp321. AI Post Transformers: QuantSpec: Hierarchical KV Cache for Self-Speculative Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/quantspec-hierarchical-kv-cache-for-self-speculative-decoding/0 min
- When Quantization Hurts Reasoning ModelsThis episode explores how quantization affects reasoning models, asking how much weights, activations, and KV caches can be compressed before multi-step reasoning starts to fail. It explains the main quantization strategies in practical serving terms, from weight-only methods like AWQ and GPTQ to weight-activation schemes such as W8A8 and W4A4, and KV cache compression for long decoding traces. The discussion argues that reasoning models are unusually fragile because small numerical errors can compound across long solution paths, making calibration quality and benchmark choice far more important than they are for ordinary chat models. Listeners would find it interesting for its concrete look at the tradeoff between cheaper inference and reliable reasoning, grounded in evaluations across model families from 1.5B to 70B and difficult benchmarks in math, science, and code.Sources:1. Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models — Ruikang Liu, Yuxuan Sun, Manyi Zhang, Haoli Bai, Xianzhi Yu, Tiezheng Yu, Chun Yuan, Lu Hou, 2025http://arxiv.org/abs/2504.048232. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, Song Han, 2023https://scholar.google.com/scholar?q=SmoothQuant%3A+Accurate+and+Efficient+Post-Training+Quantization+for+Large+Language+Models3. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, Song Han, 2024https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+LLM+Compression+and+Acceleration4. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization — Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami, 2024https://scholar.google.com/scholar?q=KVQuant%3A+Towards+10+Million+Context+Length+LLM+Inference+with+KV+Cache+Quantization5. Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning — Zhen Li, Yupeng Su, Runming Yang, Congkai Xie, Zheng Wang, Zhongwei Xie, Ngai Wong, Hongxia Yang, 2025https://scholar.google.com/scholar?q=Quantization+Meets+Reasoning%3A+Exploring+LLM+Low-Bit+Quantization+Degradation+for+Mathematical+Reasoning6. Evaluating Quantized Large Language Models — Shiyao Li et al., 2024https://scholar.google.com/scholar?q=Evaluating+Quantized+Large+Language+Models7. FlatQuant: Flatness Matters for LLM Quantization — Yuxuan Sun et al., 2024https://scholar.google.com/scholar?q=FlatQuant%3A+Flatness+Matters+for+LLM+Quantization8. s1: Simple Test-Time Scaling — Niklas Muennighoff et al., 2025https://scholar.google.com/scholar?q=s1%3A+Simple+Test-Time+Scaling9. What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study — Keyu Lv et al., 2026https://arxiv.org/abs/2601.1488810. Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation — Richard J. Young, 2026https://arxiv.org/abs/2603.2017211. On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models — Sree Harsha Tanneru et al., 2024https://arxiv.org/abs/2406.1062512. Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs — Pranav Kumar Kaliaperumal, 2026https://arxiv.org/abs/2603.0430813. KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems — Hancheng Ye et al., 2025https://arxiv.org/abs/2510.1287214. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp315. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp316. AI Post Transformers: IndexMem: Learned KV-Cache Eviction for Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-12-indexmem-learned-kv-cache-eviction-for-l-132c2a.mp317. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp30 min
- Ling and Ring 2.6 for Trillion-Scale AgentsThis episode explores Inclusion AI’s Ling and Ring 2.6 technical report, which asks how a trillion-parameter model can stay fast, handle very long contexts, and remain dependable in multi-step agent workflows. It explains why agentic AI makes latency and token costs much more painful than in ordinary chat, especially when models must carry long instruction traces, tool outputs, and large working contexts through repeated reasoning loops. The discussion breaks down the report’s core architectural changes, including a Lightning Attention and MLA hybrid with a 7:1 layer mix, designed to reduce attention cost and KV-cache memory without sacrificing model quality. It also examines the practical significance of retrofitting an existing trillion-scale checkpoint through continued pretraining and staged migration techniques, making the episode especially interesting for listeners who want a concrete look at how frontier model design is shifting from raw scale toward deployable systems engineering.Sources:1. Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale — Ang Li, Ben Liu, Bin Han, Bin Hu, Bin Jing, Binbin Hu, Bing Li, Cai Chen, Caizhi Tang, Changxin Tian, Chao Huang, Chao Zhang, Chen Liang, Chen Qian, Chengfu Tang, Chengyao Wen, Chilin Fu, Chunwei Wu, Cong Zhang, Cunyin Peng, Daixin Wang, Dalong Zhang, Deng Zhao, Dingnan Jin, Dingyuan Zhu, Donghao Zhang, Fan Yuan, Fangzheng Zhao, Fanzhuang Meng, Feifan Wu, Feng Xu, Fengbin Fang, Gangshan Wang, Guodong Yang, Hailin Zhao, Haitao Wang, Haitao Zhang, Hanxiao Zhang, Hanzi Wang, Hao Dai, Hao Liu, Hao Qian, Hao Wu, Haoxiong Liu, Haoyu Xu, Heng Zhang, Hong Liu, Hongliang Zhang, Hongrui Liu, Hongxun Li, Hongzhi Ruan, Huaidong Xiong, Huihuang Zheng, Huikang Tang, Jia Guo, Jia Li, Jia Liu, Jiameng Wang, Jiaming Liu, Jiannan Shi, Jianping Wei, Jiaolong Yang, Jiapeng Wang, Jie Gao, Jie Wang, Jiewei Wu, Jin Yang, Jinjin Li, Jinjing Huang, Jinquan Sun, Jinyao Chen, Juanhui Tu, Jun Liu, Jun Mei, Jun Xu, Jun Zhou, Junjie Ou, Junnan Sipan, Junpeng Fang, Kaihong Zhang, Kaiqin Hu, Ke Shi, Kuan Xu, Kun Tang, Kunlong Chen, Lanyin Mei, Lei Chen, Lei Liang, Lei Xu, Li Tang, Liang Jiang, Liangcheng Fu, Lihui Zhang, Linfeng Shi, Lintao Ma, Liyuan Liu, Longfei Li, Longfei Zheng, Lu Liu, Lu Yu, Man Li, Meiqi Zhu, Meng Li, Mengjie Gao, Mengshu Sun, Mingming Yin, Mingyang Zhang, Mingyuan Fan, Nuo Xu, Pan Tang, Peijie Jiang, Peilong Zhao, Peng Lin, Pingping Liu, Qi Zuo, Qian Zhao, Qiang Cheng, Qianggang Cao, Qiaoben Bao, Qing Cui, Qingyuan Yang, Qitao Shi, Qiyin Huang, Qizheng Zhou, Quan Wan, Runyuan Zhao, Shaomian Zheng, Shaowei Wei, Shengnan Zhang, Shuaicheng Li, Shujie Li, Shuo Zhang, Sikang Bian, Tianchu Yao, Tiange Xu, Tianshu Wang, Ting Guo, Tinghao Wang, Tingwei Huang, Tong Zhao, Tongkai Yang, Wang Hong, Wanli Gu, Wei Lu, Weichang Wu, Weiguang Han, Weiquan Li, Wenbo Shen, Wenjing Fang, Wenzhi Tang, Xiang Shu, Xiao Shi, Xiaodong Yan, Xiaolu Zhang, Xiaopei Wan, Xiaqing Sun, Xin Zhao, Xingyu Lu, Xinxing Yang, Xinyao Tang, Xinyu Kong, Xinyu Liu, Xiong Xu, Xuan Sun, Xudong Han, Xudong Wang, Xujie Shen, Yalin Zhang, Yangyang Hou, Yankun Ren, Yao Zhao, Ye Chen, Yeyang Chen, Yibo Cao, Yifan Zuo, Yijie Chen, Ying Li, Yingjie Song, Yingxue Li, Yiqi Wang, Yixuan Sun, Yizhu Xiao, Yongfei Xu, Yu Liu, Yuchen Fang, Yue Gao, Yue Yu, Yue Zhang, Yuqi Zhang, Yuxiao He, Yuxiao Lu, Yuxin Tian, Yuxuan Li, Yuzhuo Fu, Zhankai Xu, Zhaoxin Huan, Zhenduo Zhang, Zhengke Gui, Zhengyu Huang, Zhenjun Ma, Zhenxuan Pan, Zheping Qu, Zhibo Zhu, Zhidong Fan, Zhigang Huangfu, Zhihao Wang, Zhiqiang Zhang, Zhizhen Liu, Zhuyan Zhou, Zibin Lin, Zihang Zeng, Zihao Wang, Zilong Wang, Ziqi Liu, Zitao Xuan, Zixuan Cheng, Zujie Wen, Zuoli Tang, 2026http://arxiv.org/abs/2606.150792. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, 2022https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models3. Let's Verify Step by Step — Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe, 2023https://scholar.google.com/scholar?q=Let%27s+Verify+Step+by+Step4. CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific Reasoning — Kehua Feng, Keyan Ding, Zhihui Zhu, Lei Liang, Qiang Zhang, Huajun Chen, 2025https://scholar.google.com/scholar?q=CoT-Evo%3A+Evolutionary+Distillation+of+Chain-of-Thought+for+Scientific+Reasoning5. Chain Of Thought Compression: A Theoretical Analysis — Juncai Li, Ru Li, Yuxiang Zhou, Boxiang Ma, Jeff Z. Pan, 2026https://scholar.google.com/scholar?q=Chain+Of+Thought+Compression%3A+A+Theoretical+Analysis6. Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning (https://arxiv.org/abs/2510.19338) — Ling Team et al., 2025https://scholar.google.com/scholar?q=Every+Attention+Matters%3A+An+Efficient+Hybrid+Architecture+for+Long-Context+Reasoning+%28https%3A%2F%2Farxiv.org%2Fabs%2F2510.19338%297. Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention (https://arxiv.org/abs/2405.17381) — Zhen Qin et al., 2024https://scholar.google.com/scholar?q=Various+Lengths%2C+Constant+Speed%3A+Efficient+Language+Modeling+with+Lightning+Attention+%28https%3A%2F%2Farxiv.org%2Fabs%2F2405.17381%298. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (https://arxiv.org/abs/2405.04434) — DeepSeek-AI et al., 2024https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model+%28https%3A%2F%2Farxiv.org%2Fabs%2F2405.04434%299. Holistic Capability Preservation: Towards Compact Yet Comprehensive Reasoning Models (https://arxiv.org/abs/2504.07158) — Ling Team et al., 2025https://scholar.google.com/scholar?q=Holistic+Capability+Preservation%3A+Towards+Compact+Yet+Comprehensive+Reasoning+Models+%28https%3A%2F%2Farxiv.org%2Fabs%2F2504.07158%2910. Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model (https://arxiv.org/abs/2510.18855) — Ling Team et al., 2025https://scholar.google.com/scholar?q=Every+Step+Evolves%3A+Scaling+Reinforcement+Learning+for+Trillion-Scale+Thinking+Model+%28https%3A%2F%2Farxiv.org%2Fabs%2F2510.18855%2911. Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries (https://arxiv.org/abs/2409.12640) — Kiran Vodrahalli et al., 2024https://scholar.google.com/scholar?q=Michelangelo%3A+Long+Context+Evaluations+Beyond+Haystacks+via+Latent+Structure+Queries+%28https%3A%2F%2Farxiv.org%2Fabs%2F2409.12640%2912. HyperAttention: Long-context Attention in Near-Linear Time — Insu Han et al., 2023https://arxiv.org/abs/2310.0586913. MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling — MiniCPM Team / Wenhao An et al., 2026https://arxiv.org/abs/2602.1176114. Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning — Wenkai Yang et al., 2025https://arxiv.org/abs/2502.1808015. Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs — Mohammad Ali Alomrani et al., 2025https://arxiv.org/abs/2507.0207616. Anyprefer: An Agentic Framework for Preference Data Synthesis — Yiyang Zhou et al., 2025https://arxiv.org/abs/2504.1927617. Towards Comprehensive Preference Data Collection for Reward Modeling — Yulan Hu et al., 2024https://arxiv.org/abs/2406.1648618. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp319. AI Post Transformers: DeepSeek-V4 and Practical Million-Token Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-deepseek-v4-and-practical-million-token-6f4de1.mp320. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp321. AI Post Transformers: Self-Improving Pretraining With Post-Trained Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-02-self-improving-pretraining-with-post-tra-e37460.mp322. AI Post Transformers: Agentic Discovery for Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-12-agentic-discovery-for-test-time-scaling-f9a81f.mp323. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp324. AI Post Transformers: Kimi K2.5 and Visual Agent Swarms — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-24-kimi-k25-and-visual-agent-swarms-7d04d7.mp3Interactive Visualization: Ling and Ring 2.6 for Trillion-Scale Agents0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.