Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- ELF and Continuous Language DiffusionThis episode explores ELF: Embedded Language Flows, a continuous-time diffusion language model that stays in embedding space until the final decoding step instead of repeatedly snapping back to discrete tokens during generation. It explains how that design lets the model borrow flow-matching and guidance techniques from image diffusion, while arguing that earlier continuous text models may have underperformed because of token-level constraints rather than any fundamental weakness. The discussion highlights reported results on OpenWebText, where a 105M-parameter ELF model achieves better generative perplexity than 170M baselines with far fewer training tokens and fewer sampling steps, while also extending to translation and summarization. It also digs into the main caveat: whether the gains really come from late discretization and continuous-time modeling, or from a bundle of confounded training and inference choices, making the episode interesting both as a technical walkthrough and as a skeptical evaluation of a bold research claim.Sources:1. ELF and Continuous Language Diffusionhttps://arxiv.org/pdf/2605.109382. Structured Denoising Diffusion Models in Discrete State-Spaces — Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, Rianne van den Berg, 2021https://scholar.google.com/scholar?q=Structured+Denoising+Diffusion+Models+in+Discrete+State-Spaces3. Diffusion-LM Improves Controllable Text Generation — Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, Tatsunori B. Hashimoto, 2022https://scholar.google.com/scholar?q=Diffusion-LM+Improves+Controllable+Text+Generation4. Simple and Effective Masked Diffusion Language Models — Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander M. Rush, Volodymyr Kuleshov, 2024https://scholar.google.com/scholar?q=Simple+and+Effective+Masked+Diffusion+Language+Models5. Large Language Diffusion Models — Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li, 2025https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models6. Flow Matching for Generative Modeling — Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, Matthew Le, 2023https://scholar.google.com/scholar?q=Flow+Matching+for+Generative+Modeling7. Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport — Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, Yoshua Bengio, 2024https://scholar.google.com/scholar?q=Improving+and+Generalizing+Flow-Based+Generative+Models+with+Minibatch+Optimal+Transport8. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis — Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, Robin Rombach, 2024https://scholar.google.com/scholar?q=Scaling+Rectified+Flow+Transformers+for+High-Resolution+Image+Synthesis9. Discrete Flow Matching — Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, Yaron Lipman, 2024https://scholar.google.com/scholar?q=Discrete+Flow+Matching10. Self-conditioned Embedding Diffusion for Text Generation — Robin Strudel, Corentin Tallec, Florent Altche, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, Remi Leblond, 2022https://scholar.google.com/scholar?q=Self-conditioned+Embedding+Diffusion+for+Text+Generation11. Difformer: Empowering Diffusion Models on the Embedding Space for Text Generation — Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, Linli Xu, 2022https://scholar.google.com/scholar?q=Difformer%3A+Empowering+Diffusion+Models+on+the+Embedding+Space+for+Text+Generation12. LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling — Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu, 2026https://scholar.google.com/scholar?q=LangFlow%3A+Continuous+Diffusion+Rivals+Discrete+in+Language+Modeling13. Classifier-Free Diffusion Guidance — Jonathan Ho, Tim Salimans, 2021https://scholar.google.com/scholar?q=Classifier-Free+Diffusion+Guidance14. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models — Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen, 2021https://scholar.google.com/scholar?q=GLIDE%3A+Towards+Photorealistic+Image+Generation+and+Editing+with+Text-Guided+Diffusion+Models15. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding — Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, Mohammad Norouzi, 2022https://scholar.google.com/scholar?q=Photorealistic+Text-to-Image+Diffusion+Models+with+Deep+Language+Understanding16. High-Resolution Image Synthesis with Latent Diffusion Models — Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjorn Ommer, 2021https://scholar.google.com/scholar?q=High-Resolution+Image+Synthesis+with+Latent+Diffusion+Models17. MDLM: Masked Diffusion Language Models — likely the MDLM authors cited as [56] in the paper, 2024https://scholar.google.com/scholar?q=MDLM%3A+Masked+Diffusion+Language+Models18. Duo — likely the Duo authors cited as [57] in the paper, 2025https://scholar.google.com/scholar?q=Duo19. Latent Diffusion Models — Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjorn Ommer, 2022https://scholar.google.com/scholar?q=Latent+Diffusion+Models20. LangFlow — the LangFlow authors cited as [10] in the paper, 2026https://scholar.google.com/scholar?q=LangFlow21. FLM — the FLM authors cited as [30] in the paper, 2026https://scholar.google.com/scholar?q=FLM22. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — Aaron Lou, Chenlin Meng, Stefano Ermon, 2024https://scholar.google.com/scholar?q=Discrete+Diffusion+Modeling+by+Estimating+the+Ratios+of+the+Data+Distribution23. Scaling Behavior of Discrete Diffusion Language Models — Dimitri von Rutte, Janis Fluri, Omead Pooladzandi, Bernhard Scholkopf, Thomas Hofmann, Antonio Orvieto, 2025https://scholar.google.com/scholar?q=Scaling+Behavior+of+Discrete+Diffusion+Language+Models24. Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasoner — Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, Dinghuai Zhang, 2025https://scholar.google.com/scholar?q=Coevolutionary+Continuous+Discrete+Diffusion%3A+Make+Your+Diffusion+Language+Model+a+Latent+Reasoner25. Stay on Topic with Classifier-Free Guidance — Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, Stella Biderman, 2023https://scholar.google.com/scholar?q=Stay+on+Topic+with+Classifier-Free+Guidance26. Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking — Pengxiang Li, Shilin Yan, Joey Tsai, Renrui Zhang, Ruichuan An, Ziyu Guo, Xiaowei Gao, 2025https://scholar.google.com/scholar?q=Adaptive+Classifier-Free+Guidance+via+Dynamic+Low-Confidence+Masking27. Studying Classifier(-Free) Guidance From a Classifier-Centric Perspective — Xiaoming Zhao, Alexander G. Schwing, 2025https://scholar.google.com/scholar?q=Studying+Classifier%28-Free%29+Guidance+From+a+Classifier-Centric+Perspective28. DEPT: Decoupled Embeddings for Pre-training Language Models — Alex Iacob, Lorenzo Sani, Meghdad Kurmanji, William F. Shen, Xinchi Qiu, Dongqi Cai, Yan Gao, Nicholas D. Lane, 2024https://scholar.google.com/scholar?q=DEPT%3A+Decoupled+Embeddings+for+Pre-training+Language+Models29. AI Post Transformers: Generative Modeling via Drifting in One Step — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-generative-modeling-via-drifting-in-one-671da0.mp330. AI Post Transformers: VL-JEPA for Vision-Language Semantic Prediction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-vl-jepa-for-vision-language-semantic-pre-69c9f4.mp331. AI Post Transformers: Why Transformers Fail at Counting — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-why-transformers-fail-at-counting-137924.mp332. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp3Interactive Visualization: ELF and Continuous Language Diffusion0 min
- Agentic Discovery for Test-Time ScalingThis episode explores a paper on test-time scaling that asks whether an LLM agent can automatically discover better inference-time control policies than the hand-built heuristics researchers usually rely on. It explains the core search framework in concrete terms: a controller decides when to branch, continue, probe, prune, or stop, balancing reasoning depth, breadth, and compute budget rather than simply generating more tokens. The discussion highlights the paper’s main technical argument that offline replay over logged reasoning traces, combined with a compact controller parameterization and detailed execution feedback, makes policy discovery cheap enough to be practical. Listeners would find it interesting because it connects abstract ideas about reasoning agents to a very specific claim: smarter inference may come not from larger models, but from better learned strategies for spending compute.Sources:1. LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling — Tong Zheng, Haolin Liu, Chengsong Huang, Huiwen Bao, Sheng Zhang, Rui Liu, Runpeng Dai, Ruibo Chen, Chenxi Liu, Tianyi Xiong, Xidong Wu, Hongming Zhang, Heng Huang, 2026http://arxiv.org/abs/2605.080832. Supervisory Control of a Class of Discrete Event Systems — Peter J. Ramadge, Walter M. Wonham, 1987https://scholar.google.com/scholar?q=Supervisory+Control+of+a+Class+of+Discrete+Event+Systems3. On the Synthesis of a Reactive Module — Amir Pnueli, Roni Rosner, 1989https://scholar.google.com/scholar?q=On+the+Synthesis+of+a+Reactive+Module4. Formal Methods for Control Synthesis: An Optimization Perspective — Calin Belta, Sadra Sadraddini, 2019https://scholar.google.com/scholar?q=Formal+Methods+for+Control+Synthesis%3A+An+Optimization+Perspective5. Formal Synthesis of Controllers for Safety-Critical Autonomous Systems: Developments and Challenges — Xiang Yin, Bingzhao Gao, Xiao Yu, 2024https://scholar.google.com/scholar?q=Formal+Synthesis+of+Controllers+for+Safety-Critical+Autonomous+Systems%3A+Developments+and+Challenges6. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems — Sergey Levine, Aviral Kumar, George Tucker, Justin Fu, 2020https://scholar.google.com/scholar?q=Offline+Reinforcement+Learning%3A+Tutorial%2C+Review%2C+and+Perspectives+on+Open+Problems7. Off-Policy Deep Reinforcement Learning without Exploration — Scott Fujimoto, David Meger, Doina Precup, 2019https://scholar.google.com/scholar?q=Off-Policy+Deep+Reinforcement+Learning+without+Exploration8. Conservative Q-Learning for Offline Reinforcement Learning — Aviral Kumar, Aurick Zhou, George Tucker, Sergey Levine, 2020https://scholar.google.com/scholar?q=Conservative+Q-Learning+for+Offline+Reinforcement+Learning9. D4RL: Datasets for Deep Data-Driven Reinforcement Learning — Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine, 2020https://scholar.google.com/scholar?q=D4RL%3A+Datasets+for+Deep+Data-Driven+Reinforcement+Learning10. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters11. Parallel-Probe: Towards Efficient Parallel Thinking via 2D Probing — Tong Zheng, Chengsong Huang, Runpeng Dai, Yun He, Rui Liu, Xin Ni, Huiwen Bao, Kaishen Wang, Hongtu Zhu, Jiaxin Huang, Furong Huang, Heng Huang, 2026https://scholar.google.com/scholar?q=Parallel-Probe%3A+Towards+Efficient+Parallel+Thinking+via+2D+Probing12. Scaling Test-time Compute for LLM Agents — King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, Changwang Zhang, Chenghua Lin, Jun Wang, Ge Zhang, Wangchunshu Zhou, 2025https://scholar.google.com/scholar?q=Scaling+Test-time+Compute+for+LLM+Agents13. Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic — Yichuan Ma, Linyang Li, Yongkang Chen, Peiji Li, Xiaozhe Li, Qipeng Guo, Dahua Lin, Kai Chen, 2026https://scholar.google.com/scholar?q=Timely+Machine%3A+Awareness+of+Time+Makes+Test-Time+Scaling+Agentic14. Predicting and improving test-time scaling laws via reward tail-guided search — Muheng Li, Jian Qian, Wenlong Mou, 2026https://scholar.google.com/scholar?q=Predicting+and+improving+test-time+scaling+laws+via+reward+tail-guided+search15. TTRL: Test-Time Reinforcement Learning — Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, et al., 2025https://scholar.google.com/scholar?q=TTRL%3A+Test-Time+Reinforcement+Learning16. CTRLS: Chain-of-Thought Reasoning via Latent State-Transition — Junda Wu, Yuxin Xiong, Xintong Li, Zhengmian Hu, Tong Yu, Rui Wang, Xiang Chen, Jingbo Shang, Julian McAuley, 2025https://scholar.google.com/scholar?q=CTRLS%3A+Chain-of-Thought+Reasoning+via+Latent+State-Transition17. Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification — Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, He He, 2025https://scholar.google.com/scholar?q=Reasoning+Models+Know+When+They%27re+Right%3A+Probing+Hidden+States+for+Self-Verification18. Calibrated Reasoning: An Explanatory Verifier for Dynamic and Efficient Problem-Solving — Anisha Garg, Engin Tekin, Yash More, David Bick, Nishit Neema, Ganesh Venkatesh, 2025https://scholar.google.com/scholar?q=Calibrated+Reasoning%3A+An+Explanatory+Verifier+for+Dynamic+and+Efficient+Problem-Solving19. Contextual Drag: How Errors in the Context Affect LLM Reasoning — Yun Cheng, Xingyu Zhu, Haoyu Zhao, Sanjeev Arora, 2026https://scholar.google.com/scholar?q=Contextual+Drag%3A+How+Errors+in+the+Context+Affect+LLM+Reasoning20. Reinforcement Learning Teachers of Test Time Scaling — Edoardo Cetin, Tianyu Zhao, Yujin Tang, 2025https://scholar.google.com/scholar?q=Reinforcement+Learning+Teachers+of+Test+Time+Scaling21. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp322. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp323. AI Post Transformers: Generalist Reward Modeling with Inference-Time Scaling — Hal Turing & Dr. Ada Shannon, Tue,https://podcast.do-not-panic.com/episodes/generalist-reward-modeling-with-inference-time-scaling/24. AI Post Transformers: Reasoning Theater and Unfaithful Chain-of-Thought — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-reasoning-theater-and-unfaithful-chain-o-a4507e.mp325. AI Post Transformers: SGLang for Faster Structured LLM Programs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-06-sglang-for-faster-structured-llm-program-c59f1c.mp3Interactive Visualization: Agentic Discovery for Test-Time Scaling0 min
- TIDE and the Rare Token ProblemThis episode explores TIDE, a transformer variant that lets every layer re-access the original token identity instead of relying entirely on contextual hidden states to preserve that information. It explains how this design targets rare-token failures and “contextual collapse,” where tokens appearing in similar contexts can become too hard for the model to distinguish, especially in scientific, biomedical, or code-heavy text. The discussion walks through TIDE’s mechanism of token-indexed memory tables and layer-wise routing, framing it as a lightweight side channel rather than retrieval or mixture-of-experts. Listeners would find it interesting because it gets at a basic but rarely questioned assumption in modern transformers and asks whether a small architectural change could improve how models handle the long tail of language.Sources:1. TIDE: Every Layer Knows the Token Beneath the Context — Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho, 2026http://arxiv.org/abs/2605.062162. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones and others, 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need3. Transformer Feed-Forward Layers Are Key-Value Memories — Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy, 2021https://scholar.google.com/scholar?q=Transformer+Feed-Forward+Layers+Are+Key-Value+Memories4. Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022https://scholar.google.com/scholar?q=Locating+and+Editing+Factual+Associations+in+GPT5. TIDE: Every Layer Knows the Token Beneath the Context — Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho, 2026https://scholar.google.com/scholar?q=TIDE%3A+Every+Layer+Knows+the+Token+Beneath+the+Context6. Adaptive Input Representations for Neural Language Modeling — Alexei Baevski, Michael Auli, 2018https://scholar.google.com/scholar?q=Adaptive+Input+Representations+for+Neural+Language+Modeling7. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation — Jonathan H. Clark, Dan Garrette, Iulia Turc, John Wieting, 2021https://scholar.google.com/scholar?q=CANINE%3A+Pre-training+an+Efficient+Tokenization-Free+Encoder+for+Language+Representation8. ByT5: Towards a token-free future with pre-trained byte-to-byte models — Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, Colin Raffel, 2021https://scholar.google.com/scholar?q=ByT5%3A+Towards+a+token-free+future+with+pre-trained+byte-to-byte+models9. XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models — Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa, 2023https://scholar.google.com/scholar?q=XLM-V%3A+Overcoming+the+Vocabulary+Bottleneck+in+Multilingual+Masked+Language+Models10. Knowledge Neurons in Pretrained Transformers — Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei, 2022https://scholar.google.com/scholar?q=Knowledge+Neurons+in+Pretrained+Transformers11. Improving Language Models by Retrieving from Trillions of Tokens — Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford and colleagues, 2022https://scholar.google.com/scholar?q=Improving+Language+Models+by+Retrieving+from+Trillions+of+Tokens12. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin and colleagues, 2020https://scholar.google.com/scholar?q=Retrieval-Augmented+Generation+for+Knowledge-Intensive+NLP+Tasks13. Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-Based Pre-Trained Language Models — approx. recent NLP authors; exact list unclear from snippet, recent, likely 2020shttps://scholar.google.com/scholar?q=Reconsidering+Degeneration+of+Token+Embeddings+with+Definitions+for+Encoder-Based+Pre-Trained+Language+Models14. MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers — approx. recent LLM/memory-systems authors; exact list unclear from snippet, recent, likely 2020shttps://scholar.google.com/scholar?q=MemoryLLM%3A+Plug-n-Play+Interpretable+Feed-Forward+Memory+for+Transformers15. The FFN as a Key-Value Memory: Functional Specialization in Transformer Computation — approx. recent mechanistic-interpretability authors; exact list unclear from snippet, recent, likely 2020shttps://scholar.google.com/scholar?q=The+FFN+as+a+Key-Value+Memory%3A+Functional+Specialization+in+Transformer+Computation16. MrT5: Dynamic Token Merging for Efficient Byte-Level Language Models — approx. recent efficiency/LM authors; exact list unclear from snippet, recent, likely 2020shttps://scholar.google.com/scholar?q=MrT5%3A+Dynamic+Token+Merging+for+Efficient+Byte-Level+Language+Models17. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp318. AI Post Transformers: Nemotron 3 Super Hybrid Mamba-Transformer MoE — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-nemotron-3-super-hybrid-mamba-transforme-31ac75.mp319. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp320. AI Post Transformers: Parallelizing DeltaNet Linear Transformers over Sequence Length — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-parallelizing-deltanet-linear-transforme-2d0377.mp321. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp3Interactive Visualization: TIDE and the Rare Token Problem0 min
- Metacognition Against Confident HallucinationsThis episode explores a 2026 paper arguing that the real trust problem in language models is not error alone, but confident error, and that improving trust may depend more on metacognition than on simply scaling up knowledge. It unpacks key distinctions such as knowledge boundaries, calibration, discrimination, and the gap between intrinsic uncertainty and the uncertainty a model expresses in words, using factoid question answering as a clean test bed where correctness is measurable. The discussion also situates the paper within prior work on self-knowledge, verbalized uncertainty, and self-correction, while stressing that many apparent factuality gains may come from expanded knowledge or external tools rather than genuine awareness of limits. A listener would find it interesting because it reframes hallucinations as a trust and decision-making problem, and offers a sharper way to judge whether AI systems actually know when they should hedge, abstain, or seek evidence.Sources:1. Hallucinations Undermine Trust; Metacognition is a Way Forward — Gal Yona, Mor Geva, Yossi Matias, 2026http://arxiv.org/abs/2605.014282. Language Models (Mostly) Know What They Know — Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Ethan Perez, Deep Ganguli, Dario Amodei, Jack Clark, Jared Kaplan and collaborators, 2022https://scholar.google.com/scholar?q=Language+Models+%28Mostly%29+Know+What+They+Know3. Teaching Models to Express Their Uncertainty in Words — Stephanie Lin, Jacob Hilton, Owain Evans, 2022https://scholar.google.com/scholar?q=Teaching+Models+to+Express+Their+Uncertainty+in+Words4. What Large Language Models Know and What People Think They Know — Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Sheer Karny, Xinyue Hu, Lukas W. Mayer, Padhraic Smyth, 2025https://scholar.google.com/scholar?q=What+Large+Language+Models+Know+and+What+People+Think+They+Know5. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs — Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, Rui Zhang, 2024https://scholar.google.com/scholar?q=When+Can+LLMs+Actually+Correct+Their+Own+Mistakes%3F+A+Critical+Survey+of+Self-Correction+of+LLMs6. Can LLMs Express Their Uncertainty in Their Generated Responses? — Gal Yona, Roee Aharoni, Mor Geva, Yossi Matias, 2024https://scholar.google.com/scholar?q=Can+LLMs+Express+Their+Uncertainty+in+Their+Generated+Responses%3F7. Faithful or Fluent? Evaluating Natural Language Explanations of Uncertainty — Ghafouri et al., 2024https://scholar.google.com/scholar?q=Faithful+or+Fluent%3F+Evaluating+Natural+Language+Explanations+of+Uncertainty8. TruthfulQA: Measuring How Models Mimic Human Falsehoods — Stephanie Lin, Jacob Hilton, Owain Evans, 2022https://scholar.google.com/scholar?q=TruthfulQA%3A+Measuring+How+Models+Mimic+Human+Falsehoods9. Survey of Hallucination in Natural Language Generation — Ziwei Ji, et al., 2023https://scholar.google.com/scholar?q=Survey+of+Hallucination+in+Natural+Language+Generation10. The Geometry of Truth: Emergent Linear Structure in LLM Representations of Factuality — Marks and Tegmark, 2023https://scholar.google.com/scholar?q=The+Geometry+of+Truth%3A+Emergent+Linear+Structure+in+LLM+Representations+of+Factuality11. The Internal State of an LLM Knows When It's Lying — Levinstein and Herrmann, 2023https://scholar.google.com/scholar?q=The+Internal+State+of+an+LLM+Knows+When+It%27s+Lying12. Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations — Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, Nicola Cancedda, 2025https://scholar.google.com/scholar?q=Calibrating+Verbal+Uncertainty+as+a+Linear+Feature+to+Reduce+Hallucinations13. Calibrating the Voice of Doubt: How LLMs Diverge from Humans in Verbal Uncertainty — Jinhao Duan, Zicheng Liu, Zijie Liu, Kaidi Xu, Tianlong Chen, 2025https://scholar.google.com/scholar?q=Calibrating+the+Voice+of+Doubt%3A+How+LLMs+Diverge+from+Humans+in+Verbal+Uncertainty14. More Is Not Better: Visual Uncertainty Cues and the Fragility of Trust Calibration in LLM-Assisted Decision Making — authors not recovered from snippet, 2026https://scholar.google.com/scholar?q=More+Is+Not+Better%3A+Visual+Uncertainty+Cues+and+the+Fragility+of+Trust+Calibration+in+LLM-Assisted+Decision+Making15. Do Large Language Models Know What They Don't Know? — Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Xuanjing Huang, 2023https://scholar.google.com/scholar?q=Do+Large+Language+Models+Know+What+They+Don%27t+Know%3F16. KnowRL: Teaching Language Models to Know What They Know — Sahil Kale, Devendra Singh Dhami, 2025https://scholar.google.com/scholar?q=KnowRL%3A+Teaching+Language+Models+to+Know+What+They+Know17. What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say "I Don't Know" — Joosung Lee, Hwiyeol Jo, Donghyeon Ko, Kyubyung Chae, Cheonbok Park, Jeonghoon Kim, 2026https://scholar.google.com/scholar?q=What+Models+Know%2C+How+Well+They+Know+It%3A+Knowledge-Weighted+Fine-Tuning+for+Learning+When+to+Say+%22I+Don%27t+Know%2218. Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models — Hiyori Yoshikawa, Naoaki Okazaki, 2023https://scholar.google.com/scholar?q=Selective-LAMA%3A+Selective+Prediction+for+Confidence-Aware+Evaluation+of+Language+Models19. Selective Generation for Controllable Language Models — Minjae Lee, Kyungmin Kim, Taesoo Kim, Sangdon Park, 2024https://scholar.google.com/scholar?q=Selective+Generation+for+Controllable+Language+Models20. Dynamic Uncertainty Ranking: Enhancing Retrieval-Augmented In-Context Learning for Long-Tail Knowledge in LLMs — Shuyang Yu, Runxue Bao, Parminder Bhatia, Taha Kass-Hout, Jiayu Zhou, Cao Xiao, 2025https://scholar.google.com/scholar?q=Dynamic+Uncertainty+Ranking%3A+Enhancing+Retrieval-Augmented+In-Context+Learning+for+Long-Tail+Knowledge+in+LLMs21. UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation — Zixuan Li, Jing Xiong, Fanghua Ye, Chuanyang Zheng, Xun Wu, Jianqiao Lu, Zhongwei Wan, Xiaodan Liang, Chengming Li, Zhenan Sun, Lingpeng Kong, Ngai Wong, 2024https://scholar.google.com/scholar?q=UncertaintyRAG%3A+Span-Level+Uncertainty+Enhanced+Long-Context+Modeling+for+Retrieval-Augmented+Generation22. AI Post Transformers: Can LLMs Judge Their Own Capabilities? — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-01-can-llms-judge-their-own-capabilities-d78fed.mp323. AI Post Transformers: Do Language Models Know Their Limits — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-01-do-language-models-know-their-limits-48e444.mp324. AI Post Transformers: Teaching Language Models to Verbalize Uncertainty — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-01-teaching-language-models-to-verbalize-un-a1d774.mp325. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp3Interactive Visualization: Metacognition Against Confident Hallucinations0 min
- Vistara Brings CXL Memory to HyperscaleThis episode explores whether CXL memory expansion has finally become practical for hyperscale production, using the 2025 Vistara system as a case study. It explains the core ideas behind CXL, tiered memory, and memory disaggregation, then argues that the real comparison is not against swap but against transparent page placement that keeps hot data in local DRAM and colder pages in a slower expanded tier. The discussion highlights Vistara’s full-stack design, from a custom low-latency ASIC and Linux support to workload-specific tuning on a production server with 768 GB of local DDR5 and 256 GB of CXL-attached DDR4. Listeners would find it interesting because the episode moves past industry hype and examines the concrete tradeoffs around latency, bandwidth, operational complexity, and whether memory can finally be managed as a flexible datacenter resource rather than a fixed property of a single machine.Sources:1. Vistara Brings CXL Memory to Hyperscalehttps://aisystemcodesign.github.io/papers/isca26/vistara_camera_ready.pdf2. Software-Defined Far Memory in Warehouse-Scale Computers — H. Andres Lagar-Cavilla, Junwhan Ahn, Suleiman Souhlal, Neha Agarwal, Junaid Shahid, Greg Thelen, Parthasarathy Ranganathan, and others, 2019https://scholar.google.com/scholar?q=Software-Defined+Far+Memory+in+Warehouse-Scale+Computers3. TMO: Transparent Memory Offloading in Datacenters — Johannes Weiner, Niket Agarwal, Dan Schatzberg, Leon Yang, Hao Wang, Blaise Sanouillet, Bikash Sharma, Tejun Heo, Mayank Jain, Chunqiang Tang, Dimitrios Skarlatos, 2022https://scholar.google.com/scholar?q=TMO%3A+Transparent+Memory+Offloading+in+Datacenters4. Pond: CXL-Based Memory Pooling Systems for Cloud Platforms — Huaicheng Li, Daniel S. Berger, Stanko Novakovic, Lisa Hsu, Dan Ernst, Pantea Zardoshti, Monish Shah, Samir Rajadnya, Scott Lee, Ishwar Agarwal, Mark D. Hill, Marcus Fontoura, Ricardo Bianchini, 2023https://scholar.google.com/scholar?q=Pond%3A+CXL-Based+Memory+Pooling+Systems+for+Cloud+Platforms5. TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory — Hasan Al Maruf, Hao Wang, Abhishek Dhanotia, Johannes Weiner, Niket Agarwal, Pallab Bhattacharya, Chris Petersen, Mosharaf Chowdhury, Shobhit Kanaujia, Prakash Chauhan, 2023https://scholar.google.com/scholar?q=TPP%3A+Transparent+Page+Placement+for+CXL-Enabled+Tiered-Memory6. Managing Memory Tiers with CXL in Virtualized Environments — Yuhong Zhong, Daniel S. Berger, Carl Waldspurger, Richard Wee, Ishan Agarwal, Raghav Agarwal, Fred Hady, K. Kumar, Mark D. Hill, Mosharaf Chowdhury, Ahmed Cidon, 2024https://scholar.google.com/scholar?q=Managing+Memory+Tiers+with+CXL+in+Virtualized+Environments7. Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices — Yongjun Sun, Ye Yuan, Ziming Yu, Ryan Kuper, Chao Song, Jiyong Huang, Honggyu Ji, Saurabh Agarwal, Jingwen Lou, Inhwan Jeong, Rui Wang, Joonho H. Ahn, Tianyin Xu, Nam Sung Kim, 2023https://scholar.google.com/scholar?q=Demystifying+CXL+Memory+with+Genuine+CXL-Ready+Systems+and+Devices8. M5: Mastering Page Migration and Memory Management for CXL-based Tiered Memory Systems — Yongjun Sun, Jiyoon Kim, Ziming Yu, Joonwon Zhang, Sangho Chai, Minjae J. Kim, Hyeonsu Nam, Jihye Park, Euna Na, Ye Yuan, Rui Wang, Joonho H. Ahn, Tianyin Xu, Nam Sung Kim, 2025https://scholar.google.com/scholar?q=M5%3A+Mastering+Page+Migration+and+Memory+Management+for+CXL-based+Tiered+Memory+Systems9. Dissecting CXL Memory Performance at Scale: Analysis, Modeling, and Optimization — approx. recent systems authors, 2024/2025https://scholar.google.com/scholar?q=Dissecting+CXL+Memory+Performance+at+Scale%3A+Analysis%2C+Modeling%2C+and+Optimization10. Improving Key-Value Cache Performance with Heterogeneous Memory Tiering: A Case Study of CXL-Based Memory Expansion — approx. recent cache/tiering authors, 2024/2025https://scholar.google.com/scholar?q=Improving+Key-Value+Cache+Performance+with+Heterogeneous+Memory+Tiering%3A+A+Case+Study+of+CXL-Based+Memory+Expansion11. Tolerate It if You Cannot Reduce It: Handling Latency in Tiered Memory — approx. recent tiered-memory authors, 2024/2025https://scholar.google.com/scholar?q=Tolerate+It+if+You+Cannot+Reduce+It%3A+Handling+Latency+in+Tiered+Memory12. Can Hardware Outsmart Software in Tiered Memory Management? A CMM-H Case Study — approx. recent tiered-memory authors, 2024/2025https://scholar.google.com/scholar?q=Can+Hardware+Outsmart+Software+in+Tiered+Memory+Management%3F+A+CMM-H+Case+Study13. NeoMem: Hardware/Software Co-Design for CXL-Native Memory Tiering — approx. recent CXL co-design authors, 2024/2025https://scholar.google.com/scholar?q=NeoMem%3A+Hardware%2FSoftware+Co-Design+for+CXL-Native+Memory+Tiering14. Survey of Disaggregated Memory: Cross-Layer Technique Insights for Next-Generation Datacenters — approx. survey authors, 2024/2025https://scholar.google.com/scholar?q=Survey+of+Disaggregated+Memory%3A+Cross-Layer+Technique+Insights+for+Next-Generation+Datacenters15. Disaggregated Memory in the Datacenter: A Survey — approx. survey authors, 2024/2025https://scholar.google.com/scholar?q=Disaggregated+Memory+in+the+Datacenter%3A+A+Survey16. AI Post Transformers: CXL Computational Memory Offloading for Lower Runtime — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-cxl-computational-memory-offloading-for-3b2124.mp317. AI Post Transformers: FengHuang for Rack-Scale LLM Inference Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-fenghuang-for-rack-scale-llm-inference-m-62708e.mp318. AI Post Transformers: Splitwise: Phase-Split LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-26-splitwise-phase-split-llm-inference-e8945b.mp3Interactive Visualization: Vistara Brings CXL Memory to Hyperscale13 min
- Optimization, Credit Assignment, and ConsciousnessThis episode explores a historical argument that simple Hebbian learning rules are too weak to explain real intelligence, because they mainly capture local correlations rather than solving multivariate credit-assignment problems. It examines how that critique points toward global optimization methods such as backpropagation and, in some settings, reinforcement learning, while contrasting their engineering success with the biological appeal of local synaptic updates. The discussion uses examples like XOR and later Hebbian variants such as Oja and BCM to show that the real issue is not whether Hebbian ideas are useless, but what kind of optimization principle is powerful enough to support complex learning. A listener would find it interesting for its mix of AI history, mathematical intuition, and an early attempt to connect learning theory to broader questions about consciousness.Sources:1. Optimization, Credit Assignment, and Consciousnesshttps://gwern.net/doc/ai/nn/rnn/1998-werbos.pdf2. The Organization of Behavior: A Neuropsychological Theory — Donald O. Hebb, 1949https://scholar.google.com/scholar?q=The+Organization+of+Behavior%3A+A+Neuropsychological+Theory3. A Simplified Neuron Model as a Principal Component Analyzer — Erkki Oja, 1982https://scholar.google.com/scholar?q=A+Simplified+Neuron+Model+as+a+Principal+Component+Analyzer4. Theory for the Development of Neuron Selectivity: Orientation Specificity and Binocular Interaction in Visual Cortex — Elie L. Bienenstock, Leon N. Cooper, Paul W. Munro, 1982https://scholar.google.com/scholar?q=Theory+for+the+Development+of+Neuron+Selectivity%3A+Orientation+Specificity+and+Binocular+Interaction+in+Visual+Cortex5. Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network — Xiaohui Xie, H. Sebastian Seung, 2003https://scholar.google.com/scholar?q=Equivalence+of+Backpropagation+and+Contrastive+Hebbian+Learning+in+a+Layered+Network6. The Organization of Behavior — Donald O. Hebb, 1949https://scholar.google.com/scholar?q=The+Organization+of+Behavior7. Optimization: A Foundation for Understanding Consciousness — Paul J. Werbos, 1996https://scholar.google.com/scholar?q=Optimization%3A+A+Foundation+for+Understanding+Consciousness8. Learning Representations by Back-Propagating Errors — David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986https://scholar.google.com/scholar?q=Learning+Representations+by+Back-Propagating+Errors9. A Theory of Cerebral Neocortex — Elie L. Bienenstock, Leon N. Cooper, Paul W. Munro, 1982https://scholar.google.com/scholar?q=A+Theory+of+Cerebral+Neocortex10. Reinforcement Learning: An Introduction — Richard S. Sutton, Andrew G. Barto, 1998https://scholar.google.com/scholar?q=Reinforcement+Learning%3A+An+Introduction11. Backpropagation-free spiking neural networks with the forward-forward algorithm — authors not identifiable from the snippet, recent; exact year not identifiable from the snippethttps://scholar.google.com/scholar?q=Backpropagation-free+spiking+neural+networks+with+the+forward-forward+algorithm12. AI Post Transformers: Reverse-Mode Differentiation Across AD and Neural Nets — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-29-reverse-mode-differentiation-across-ad-a-5c1f77.mp313. AI Post Transformers: Backpropagation Through Time Explained — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-backpropagation-through-time-explained-eea44a.mp314. AI Post Transformers: Reinforcement Learning in 2025: An Overview — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-reinforcement-learning-in-2025-an-overvi-e7a4ce.mp315. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp316. AI Post Transformers: Deep Learning in Spiking Neural Networks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-24-deep-learning-in-spiking-neural-networks-bf558f.mp3Interactive Visualization: Optimization, Credit Assignment, and Consciousness8 min
- Why Transformers Fail at CountingThis episode explores a mechanistic interpretability paper arguing that transformers often fail at counting not because they lack an internal notion of quantity, but because the pathway that converts that latent count into digit tokens is poorly aligned. It explains key ideas like linear probes, the logit lens, attention, LoRA, constrained next-token evaluation, and autoregressive generation to show how the authors separate “the model knows” from “the model can say.” The discussion highlights striking evidence that intermediate hidden states can encode counts almost perfectly while the corresponding digit readout directions remain nearly orthogonal, creating a readout bottleneck. Listeners would find it interesting because it reframes a familiar model weakness into a precise geometric and causal diagnosis, with implications for how to fix generation failures in modern model families like Pythia, Qwen3, and Mistral.Sources:1. Why Transformers Fail at Countinghttps://arxiv.org/pdf/2605.032582. Teaching Arithmetic to Small Transformers — Andrew McLeish, David Irving, Simon Sokota, Max Black, Berlin Chen, et al., 2024https://scholar.google.com/scholar?q=Teaching+Arithmetic+to+Small+Transformers3. Language Models Use Trigonometry to Do Addition — Stephen McLeish, et al., 2024https://scholar.google.com/scholar?q=Language+Models+Use+Trigonometry+to+Do+Addition4. Faithfulness of Linear Probes in Transformers — Various probe-critique literature; a representative reference should be cited explicitly by the author, 2019-2024https://scholar.google.com/scholar?q=Faithfulness+of+Linear+Probes+in+Transformers5. ROME: Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022https://scholar.google.com/scholar?q=ROME%3A+Locating+and+Editing+Factual+Associations+in+GPT6. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets — Wes Gurnee, et al., 2023https://scholar.google.com/scholar?q=The+Geometry+of+Truth%3A+Emergent+Linear+Structure+in+Large+Language+Model+Representations+of+True%2FFalse+Datasets7. A Mathematical Framework for Transformer Circuits — Nelson Elhage, et al., 2021https://scholar.google.com/scholar?q=A+Mathematical+Framework+for+Transformer+Circuits8. Finding Transformer Circuits with Edge-Level Attribution Patching — Neel Nanda, et al., 2023https://scholar.google.com/scholar?q=Finding+Transformer+Circuits+with+Edge-Level+Attribution+Patching9. Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — Aaditya K. Singh, DJ Strouse, 2024https://scholar.google.com/scholar?q=Tokenization+counts%3A+the+impact+of+tokenization+on+arithmetic+in+frontier+LLMs10. Efficient numeracy in language models through single-token number embeddings — Linus Kreitner, Paul Hager, Jonathan Mengedoht, Georgios Kaissis, Daniel Rueckert, Martin J. Menten, 2025https://scholar.google.com/scholar?q=Efficient+numeracy+in+language+models+through+single-token+number+embeddings11. Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models — Dominic Petrak, Nafise Sadat Moosavi, Iryna Gurevych, 2023https://scholar.google.com/scholar?q=Arithmetic-Based+Pretraining+Improving+Numeracy+of+Pretrained+Language+Models12. Rethinking Weight Tying: Pseudo-Inverse Tying for Stable LM Training and Updates — Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang, 2026https://scholar.google.com/scholar?q=Rethinking+Weight+Tying%3A+Pseudo-Inverse+Tying+for+Stable+LM+Training+and+Updates13. Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data — Charles Jin, 2024https://scholar.google.com/scholar?q=Latent+Causal+Probing%3A+A+Formal+Perspective+on+Probing+with+Causal+Models+of+Data14. AI Post Transformers: Linear Classifier Probes for Intermediate Layers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-linear-classifier-probes-for-intermediat-927ae3.mp315. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp316. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp317. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp318. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3Interactive Visualization: Why Transformers Fail at Counting11 min
- Split Personality Training Reveals Latent KnowledgeThis episode explores a 2026 paper on “split personality training,” a method for attaching an internal reviewer to a language model that can reveal what the model knows about its own deceptive or reward-hacking behavior without changing the answer shown to the user. It situates the work in the broader lineage of latent knowledge elicitation, alignment faking, and mechanistic interpretability, explaining why a model’s hidden state may contain more honest information than its final text output. The discussion focuses on the paper’s use of a LoRA-based “honest persona” that activates only after the main response, and on benchmark setups like Anthropic’s auditing game that test whether internal representations expose hidden objectives that outside observers cannot infer. Listeners would find it interesting because it tackles a central safety problem: whether models can be audited for strategic deception using their own internal signals rather than their polished outward behavior.Sources:1. Split Personality Training Reveals Latent Knowledgehttps://arxiv.org/pdf/2602.055322. Discovering Latent Knowledge in Language Models Without Supervision — Collin Burns, Haotian Ye, Dan Klein, Jacob Steinhardt, 2023https://scholar.google.com/scholar?q=Discovering+Latent+Knowledge+in+Language+Models+Without+Supervision3. Eliciting Latent Knowledge from Quirky Language Models — Alex Mallen, Nora Belrose, 2023https://scholar.google.com/scholar?q=Eliciting+Latent+Knowledge+from+Quirky+Language+Models4. Challenges with Unsupervised LLM Knowledge Discovery — Sebastian Farquhar, Vikrant Varma, Zachary Kenton, Johannes Gasteiger, Vladimir Mikulik, Rohin Shah, 2023https://scholar.google.com/scholar?q=Challenges+with+Unsupervised+LLM+Knowledge+Discovery5. LatentQA: Teaching LLMs to Decode Activations Into Natural Language — Alexander Pan, Lijie Chen, Jacob Steinhardt, 2024https://scholar.google.com/scholar?q=LatentQA%3A+Teaching+LLMs+to+Decode+Activations+Into+Natural+Language6. Language Models Mostly Know What They Know — Collin Burns, Haotian Ye, Dan Klein, Jacob Steinhardt, 2023https://scholar.google.com/scholar?q=Language+Models+Mostly+Know+What+They+Know7. ELK Report: Eliciting Latent Knowledge — Paul Christiano, Ajeya Cotra, Mark Xu, et al., 2021https://scholar.google.com/scholar?q=ELK+Report%3A+Eliciting+Latent+Knowledge8. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets — Samuel Marks, Max Tegmark, 2024https://scholar.google.com/scholar?q=The+Geometry+of+Truth%3A+Emergent+Linear+Structure+in+Large+Language+Model+Representations+of+True%2FFalse+Datasets9. LatentQA: Teaching Language Models to Decode Activations into Natural Language — Yida Pan et al., 2024https://scholar.google.com/scholar?q=LatentQA%3A+Teaching+Language+Models+to+Decode+Activations+into+Natural+Language10. Activation Oracles — Jesse Karvonen et al., 2026https://scholar.google.com/scholar?q=Activation+Oracles11. Confessions — Nikhil Joglekar et al., 2025https://scholar.google.com/scholar?q=Confessions12. Self-Report Fine-Tuning — Li et al., 2025https://scholar.google.com/scholar?q=Self-Report+Fine-Tuning13. Auditing Language Models for Hidden Objectives — Anthropic, 2025https://scholar.google.com/scholar?q=Auditing+Language+Models+for+Hidden+Objectives14. Alignment Faking in Large Language Models — Ryan Greenblatt et al., 2024https://scholar.google.com/scholar?q=Alignment+Faking+in+Large+Language+Models15. Towards Eliciting Latent Knowledge from LLMs with Mechanistic Interpretability — Bartosz Cywinski, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda, 2025https://scholar.google.com/scholar?q=Towards+Eliciting+Latent+Knowledge+from+LLMs+with+Mechanistic+Interpretability16. Quantifying Elicitation of Latent Capabilities in Language Models — Elizabeth Donoway et al., 2025https://scholar.google.com/scholar?q=Quantifying+Elicitation+of+Latent+Capabilities+in+Language+Models17. BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models — Yi Zeng, Weiyu Sun, Tran Huynh, Dawn Song, Bo Li, Ruoxi Jia, 2024https://scholar.google.com/scholar?q=BEEAR%3A+Embedding-based+Adversarial+Removal+of+Safety+Backdoors+in+Instruction-tuned+Language+Models18. Investigating Adversarial Trigger Transfer in Large Language Models — Nicholas Meade, Arkil Patel, Siva Reddy, 2024https://scholar.google.com/scholar?q=Investigating+Adversarial+Trigger+Transfer+in+Large+Language+Models19. When Thinking LLMs Lie: Unveiling the Strategic Deception in Representations of Reasoning Models — Kai Wang, Yihao Zhang, Meng Sun, 2025https://scholar.google.com/scholar?q=When+Thinking+LLMs+Lie%3A+Unveiling+the+Strategic+Deception+in+Representations+of+Reasoning+Models20. Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort — Xinpeng Wang, Nitish Joshi, Barbara Plank, Rico Angell, He He, 2025https://scholar.google.com/scholar?q=Is+It+Thinking+or+Cheating%3F+Detecting+Implicit+Reward+Hacking+by+Measuring+Reasoning+Effort21. AI Post Transformers: Linear Classifier Probes for Intermediate Layers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-linear-classifier-probes-for-intermediat-927ae3.mp322. AI Post Transformers: CLUE: Hidden-State Clustering for Non-parametric Verification — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/clue-hidden-state-clustering-for-non-parametric-verification/23. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp324. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp3Interactive Visualization: Split Personality Training Reveals Latent Knowledge17 min
- RAPTOR: Stable Concept Directions From Logistic ProbesThis episode explores RAPTOR, a method for extracting concept directions from language model hidden states using ridge-regularized logistic probes, with the goal of making those directions accurate enough for interpretation and stable enough for activation steering. It explains the core probe-then-steer workflow, why linear probes can reveal what a model has encoded, and why good classification accuracy does not necessarily produce a reliable control vector. The discussion situates the paper within broader debates in mechanistic interpretability, including concerns about brittle probes, distribution shift, and whether a single direction can really capture a concept like sentiment, refusal, or honesty. A listener would find it interesting because the episode turns an abstract interpretability question into a concrete engineering tradeoff about robustness, causal usefulness, and whether cheap white-box methods could become practical tools for controlling large models.Sources:1. RAPTOR: Ridge-Adaptive Logistic Probes — Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, Kaize Ding, 2026http://arxiv.org/abs/2602.001582. Plug and Play Language Models: a Simple Approach to Controlled Text Generation — Sumanth Dathathri, Andrea Madotto, Janice Lan, Jason Yosinski, Rosanne Liu, et al., 2019https://arxiv.org/abs/1912.021643. Activation Addition: Steering Language Models Without Optimization — Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Ulisse Mini, Monte MacDiarmid, 2023https://arxiv.org/abs/2308.102484. Representation Engineering: A Top-Down Approach to AI Transparency — Andy Zou, Long Phan, Sarah Chen, James Campbell, Dan Hendrycks, J. Zico Kolter, et al., 2023https://arxiv.org/abs/2310.014055. Refusal in Language Models Is Mediated by a Single Direction — Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, Neel Nanda, 2024https://arxiv.org/abs/2406.117176. Understanding Intermediate Layers Using Linear Classifier Probes — Guillaume Alain, Yoshua Bengio, 2016https://openreview.net/forum?id=HJ4-rAVtl7. Designing and Interpreting Probes with Control Tasks — John Hewitt, Percy Liang, 2019https://aclanthology.org/D19-1275/8. Information-Theoretic Probing with Minimum Description Length — Elena Voita, Ivan Titov, 2020https://aclanthology.org/2020.emnlp-main.14/9. RAPTOR: Ridge-Adaptive Logistic Probes — Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, Kaize Ding, 2026https://arxiv.org/abs/2602.0015810. Steering Llama 2 via Contrastive Activation Addition — Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, Alexander Turner, 2024https://scholar.google.com/scholar?q=Steering+Llama+2+via+Contrastive+Activation+Addition11. Analysing the Generalisation and Reliability of Steering Vectors — Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga-Alonso, Robert Kirk, 2024https://scholar.google.com/scholar?q=Analysing+the+Generalisation+and+Reliability+of+Steering+Vectors12. Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distribution — Haiyan Zhao, Heng Zhao, Bo Shen, Ali Payani, Fan Yang, Mengnan Du, 2025https://scholar.google.com/scholar?q=Beyond+Single+Concept+Vector%3A+Modeling+Concept+Subspace+in+LLMs+with+Gaussian+Distribution13. Controlling Large Language Models Through Concept Activation Vectors — Hanyu Zhang, Xiting Wang, Chengao Li, Xiang Ao, Qing He, 2025https://scholar.google.com/scholar?q=Controlling+Large+Language+Models+Through+Concept+Activation+Vectors14. Token prepending: A training-free approach for eliciting better sentence embeddings from llms — authors not confirmed from provided snippet, recent, unverifiedhttps://scholar.google.com/scholar?q=Token+prepending%3A+A+training-free+approach+for+eliciting+better+sentence+embeddings+from+llms15. Rep2Text: Decoding Full Text from a Single LLM Token Representation — authors not confirmed from provided snippet, recent, unverifiedhttps://scholar.google.com/scholar?q=Rep2Text%3A+Decoding+Full+Text+from+a+Single+LLM+Token+Representation16. Context Matters: Analyzing the Generalizability of Linear Probing and Steering Across Diverse Scenarios — authors not confirmed from provided snippet, recent, unverifiedhttps://scholar.google.com/scholar?q=Context+Matters%3A+Analyzing+the+Generalizability+of+Linear+Probing+and+Steering+Across+Diverse+Scenarios17. Angular steering: Behavior control via rotation in activation space — authors not confirmed from provided snippet, recent, unverifiedhttps://scholar.google.com/scholar?q=Angular+steering%3A+Behavior+control+via+rotation+in+activation+space18. Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistency — authors not confirmed from provided snippet, recent, unverifiedhttps://scholar.google.com/scholar?q=Global+Evolutionary+Steering%3A+Refining+Activation+Steering+Control+via+Cross-Layer+Consistency19. Fine-Grained Activation Steering: Steering Less, Achieving More — authors not confirmed from provided snippet, recent, unverifiedhttps://scholar.google.com/scholar?q=Fine-Grained+Activation+Steering%3A+Steering+Less%2C+Achieving+More20. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp321. AI Post Transformers: Learning to Reason with 13 Parameters — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-learning-to-reason-with-13-parameters-54c87f.mp322. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp3Interactive Visualization: RAPTOR: Stable Concept Directions From Logistic Probes16 min
- Reasoning Theater and Unfaithful Chain-of-ThoughtThis episode explores a paper arguing that a model’s internal answer belief can form well before its visible chain-of-thought reveals it, raising doubts about whether reasoning traces are true explanations or polished post hoc narratives. It explains core ideas such as chain-of-thought faithfulness, activation monitoring, mechanistic interpretability, confidence calibration, and dynamic inference, framing the broader safety question of whether text reasoning can really serve as an audit trail. The discussion focuses on the paper’s method of comparing internal activation probes, forced early answers, and monitors of partial written reasoning to test whether models “know” the answer before their text shows it. Listeners would find it interesting because it connects interpretability research to practical concerns about oversight, trust, and compute efficiency, while contrasting easy recall-heavy benchmarks with harder multistep science questions where genuine belief updates may still happen during inference.Sources:1. Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought — Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow, Atticus Geiger, Owen Lewis, Jack Merullo, 2026http://arxiv.org/abs/2603.054882. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, 2022https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models3. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Miles Turpin, Julian Michael, Ethan Perez, Samuel R. Bowman, 2023https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think%3A+Unfaithful+Explanations+in+Chain-of-Thought+Prompting4. Measuring Faithfulness in Chain-of-Thought Reasoning — Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion and others, 2023https://scholar.google.com/scholar?q=Measuring+Faithfulness+in+Chain-of-Thought+Reasoning5. Reasoning Models Don't Always Say What They Think — Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, Ethan Perez, 2025https://scholar.google.com/scholar?q=Reasoning+Models+Don%27t+Always+Say+What+They+Think6. Performative Thinking? The Brittle Correlation between CoT Length and Problem Complexity — Vivek Palod, Karthik Valmeekam, Kyle Stechly, Subbarao Kambhampati, 2025https://scholar.google.com/scholar?q=Performative+Thinking%3F+The+Brittle+Correlation+between+CoT+Length+and+Problem+Complexity7. Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety — Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, et al., 2025https://scholar.google.com/scholar?q=Chain+of+Thought+Monitorability%3A+A+New+and+Fragile+Opportunity+for+AI+Safety8. Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification — Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, He He, 2025https://scholar.google.com/scholar?q=Reasoning+Models+Know+When+They%27re+Right%3A+Probing+Hidden+States+for+Self-Verification9. A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behavior — Harry Mayne, Justin Singh Kang, Dewi Gould, Kannan Ramchandran, Adam Mahdi, Noah Y. Siegel, 2026https://scholar.google.com/scholar?q=A+Positive+Case+for+Faithfulness%3A+LLM+Self-Explanations+Help+Predict+Model+Behavior10. Base Models Know How to Reason, Thinking Models Learn When — Constantin Venhoff, Ivan Arcuschin, Philip Torr, Arthur Conmy, Neel Nanda, 2025https://scholar.google.com/scholar?q=Base+Models+Know+How+to+Reason%2C+Thinking+Models+Learn+When11. Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps — Martin Tutek, Fateme Hashemi Chaleshtori, Ana Marasovic, Yonatan Belinkov, 2025https://scholar.google.com/scholar?q=Measuring+Chain+of+Thought+Faithfulness+by+Unlearning+Reasoning+Steps12. Faithful Chain-of-Thought Reasoning — Qing Lyu et al., 2023https://scholar.google.com/scholar?q=Faithful+Chain-of-Thought+Reasoning13. Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning — Wenkai Yang, Shuming Ma, Yankai Lin, Furu Wei, 2025https://scholar.google.com/scholar?q=Towards+Thinking-Optimal+Scaling+of+Test-Time+Compute+for+LLM+Reasoning14. AI Post Transformers: Do Language Models Know Their Limits — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-01-do-language-models-know-their-limits-48e444.mp315. AI Post Transformers: Selective Classification with Deep Neural Networks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-01-selective-classification-with-deep-neura-bed8cb.mp316. AI Post Transformers: Stabilizing Efficient Reasoning with Step-Level Advantage Selection — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-29-stabilizing-efficient-reasoning-with-ste-1e589d.mp317. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp3Interactive Visualization: Reasoning Theater and Unfaithful Chain-of-Thought13 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.