Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Discrete Representations for Continual Reinforcement LearningThis episode explores whether learned discrete representations actually improve reinforcement learning, world modeling, and continual adaptation compared with standard continuous latent spaces. It explains how vector-quantized codebook latents, sparse binary-style features, and older ideas like tile coding relate to modern world models, and why the real advantage may come from reduced interference rather than discreteness alone. The discussion centers on three evaluation settings: predicting future dynamics in latent space, improving downstream control in model-free RL, and helping agents adapt to shifting tasks without forgetting earlier behavior. Listeners would find it interesting because it cuts through the “discrete vs. continuous” hype and turns the paper into a sharper engineering question about which representation bottlenecks produce more stable, reusable abstractions under changing conditions.Sources:1. Harnessing Discrete Representations For Continual Reinforcement Learning — Edan Meyer, Adam White, Marlos C. Machado, 2023http://arxiv.org/abs/2312.012032. Neural Discrete Representation Learning — Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, 2017https://papers.nips.cc/paper/2017/hash/7a98af17e63a0ac09ce2e96d03992fbc-Abstract.html3. Mastering Atari with Discrete World Models — Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy Ba, 2021https://openreview.net/forum?id=0oabwyZbOu4. Transformers are Sample-Efficient World Models — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2023https://openreview.net/forum?id=vhFu1Acb0xb5. Harnessing Discrete Representations for Continual Reinforcement Learning — Edan Jacob Meyer, Adam White, Marlos C. Machado, 2024https://openreview.net/forum?id=tCXURNlAZ36. Mastering Diverse Domains through World Models — Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy P. Lillicrap, 2023https://scholar.google.com/scholar?q=Mastering+Diverse+Domains+through+World+Models7. Fuzzy Tiling Activations: A Simple Approach to Learning Sparse Representations Online — Yangchen Pan, Kirby Banman, Martha White, 2021https://scholar.google.com/scholar?q=Fuzzy+Tiling+Activations%3A+A+Simple+Approach+to+Learning+Sparse+Representations+Online8. Investigating the Properties of Neural Network Representations in Reinforcement Learning — Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, Adam White, 2022https://scholar.google.com/scholar?q=Investigating+the+Properties+of+Neural+Network+Representations+in+Reinforcement+Learning9. Smaller World Models for Reinforcement Learning — Jan Robine, Tobias Uelwer, Stefan Harmeling, 2021https://scholar.google.com/scholar?q=Smaller+World+Models+for+Reinforcement+Learning10. Continual Learning as Computationally Constrained Reinforcement Learning — Saurabh Kumar, Henrik Marklund, Ashish Rao, Yifan Zhu, Hong Jun Jeon, Yueyang Liu, Benjamin Van Roy, 2023https://scholar.google.com/scholar?q=Continual+Learning+as+Computationally+Constrained+Reinforcement+Learning11. Efficient World Models with Context-Aware Tokenization — Vincent Micheli, Eloi Alonso, Francois Fleuret, 2024https://scholar.google.com/scholar?q=Efficient+World+Models+with+Context-Aware+Tokenization12. AdaWorld: Learning Adaptable World Models with Latent Actions — authors not identifiable from the snippet, recenthttps://scholar.google.com/scholar?q=AdaWorld%3A+Learning+Adaptable+World+Models+with+Latent+Actions13. A Survey of Continual Reinforcement Learning — authors not identifiable from the snippet, recenthttps://scholar.google.com/scholar?q=A+Survey+of+Continual+Reinforcement+Learning14. Stable Continual Reinforcement Learning via Diffusion-Based Trajectory Replay — authors not identifiable from the snippet, recenthttps://scholar.google.com/scholar?q=Stable+Continual+Reinforcement+Learning+via+Diffusion-Based+Trajectory+Replay15. AI Post Transformers: DreamerV3 World Models Across 150 Tasks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-dreamerv3-world-models-across-150-tasks-af5edb.mp316. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp317. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp315 min
- Teaching Language Models to Verbalize UncertaintyThis episode explores whether language models can express uncertainty in natural language in a way that is actually calibrated and useful, rather than merely sounding cautious or confident. It focuses on the paper’s central distinction between token uncertainty and epistemic uncertainty, arguing that next-token probabilities are a poor proxy for whether a model truly knows an answer. The discussion situates this idea alongside earlier work on Bayesian approximations, dataset shift, and newer hallucination-detection methods such as semantic entropy, all pointing to the same challenge: uncertainty should attach to claims, not just strings. A listener would find it interesting because it connects a seemingly simple design choice, having models state confidence in words, to the much larger problem of building AI systems that can warn users when they are likely guessing.Sources:1. Teaching Models to Express Their Uncertainty in Words — Stephanie Lin, Jacob Hilton, Owain Evans, 2022http://arxiv.org/abs/2205.143342. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods — Eyke Hullermeier, Willem Waegeman, 2021https://scholar.google.com/scholar?q=Aleatoric+and+Epistemic+Uncertainty+in+Machine+Learning%3A+An+Introduction+to+Concepts+and+Methods3. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — Yarin Gal, Zoubin Ghahramani, 2016https://scholar.google.com/scholar?q=Dropout+as+a+Bayesian+Approximation%3A+Representing+Model+Uncertainty+in+Deep+Learning4. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift — Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, Jasper Snoek, 2019https://scholar.google.com/scholar?q=Can+You+Trust+Your+Model%27s+Uncertainty%3F+Evaluating+Predictive+Uncertainty+Under+Dataset+Shift5. Detecting Hallucinations in Large Language Models Using Semantic Entropy — Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and colleagues including Yarin Gal, 2024https://scholar.google.com/scholar?q=Detecting+Hallucinations+in+Large+Language+Models+Using+Semantic+Entropy6. Language Models (Mostly) Know What They Know — Katherine Kadavath, Roberta Raileanu, Akul Arora, et al., 2022https://scholar.google.com/scholar?q=Language+Models+%28Mostly%29+Know+What+They+Know7. On Calibration of Modern Neural Networks — Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger, 2017https://scholar.google.com/scholar?q=On+Calibration+of+Modern+Neural+Networks8. Can Language Models Be Too Honest? The Curious Case of Adaptive Honesty — Owain Evans, Jacob Hilton, Lukas Heim, et al., 2021https://scholar.google.com/scholar?q=Can+Language+Models+Be+Too+Honest%3F+The+Curious+Case+of+Adaptive+Honesty9. TruthfulQA: Measuring How Models Mimic Human Falsehoods — Stephanie Lin, Jacob Hilton, Owain Evans, 2021https://scholar.google.com/scholar?q=TruthfulQA%3A+Measuring+How+Models+Mimic+Human+Falsehoods10. Measuring Calibration in Deep Learning — Mahdi Pakdaman Naeini, Gregory Cooper, Milos Hauskrecht, 2015https://scholar.google.com/scholar?q=Measuring+Calibration+in+Deep+Learning11. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation — Sebastian Kuhn, Yarin Gal, et al., 2023https://scholar.google.com/scholar?q=Semantic+Uncertainty%3A+Linguistic+Invariances+for+Uncertainty+Estimation+in+Natural+Language+Generation12. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models — Potsawee Manakul, Adian Liusie, Mark Gales, 2023https://scholar.google.com/scholar?q=SelfCheckGPT%3A+Zero-Resource+Black-Box+Hallucination+Detection+for+Generative+Large+Language+Models13. Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence? — Shiyu Ni, Keping Bi, Lulu Yu, Jiafeng Guo, 2024https://scholar.google.com/scholar?q=Are+Large+Language+Models+More+Honest+in+Their+Probabilistic+or+Verbalized+Confidence%3F14. Calibrating Verbalized Probabilities for Large Language Models — Cheng Wang, Gyuri Szarvas, Georges Balazs, Pavel Danchenko, Patrick Ernst, 2024https://scholar.google.com/scholar?q=Calibrating+Verbalized+Probabilities+for+Large+Language+Models15. Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuning — Jiaqi Li, Yixuan Tang, Yi Yang, 2025https://scholar.google.com/scholar?q=Know+the+Unknown%3A+An+Uncertainty-Sensitive+Method+for+LLM+Instruction+Tuning16. From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered — Siddartha Devic, Tejas Srinivasan, Jesse Thomason, Willie Neiswanger, Vatsal Sharan, 2025https://scholar.google.com/scholar?q=From+Calibration+to+Collaboration%3A+LLM+Uncertainty+Quantification+Should+Be+More+Human-Centered17. Closing the Confidence-Faithfulness Gap in Large Language Models — Miranda Muqing Miao, Lyle Ungar, 2026https://scholar.google.com/scholar?q=Closing+the+Confidence-Faithfulness+Gap+in+Large+Language+Models18. AI Post Transformers: CLUE: Hidden-State Clustering for Non-parametric Verification — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/clue-hidden-state-clustering-for-non-parametric-verification/19. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp320. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3Interactive Visualization: Teaching Language Models to Verbalize Uncertainty15 min
- ChartNet for Robust Multimodal Chart UnderstandingThis episode explores ChartNet, a 1.5 million-sample multimodal dataset designed to improve how vision-language models read and reason about charts. It explains why chart understanding is harder than OCR or captioning alone, because models must connect visual marks, axes, legends, numerical values, and language-based reasoning with high precision. The discussion places ChartNet in the context of earlier benchmarks like DVQA, PlotQA, ChartQA, and UniChart, arguing that past datasets were too small or too narrow to teach robust chart comprehension. It also examines ChartNet’s code-guided pipeline, where models reconstruct plotting code from seed charts, generate structurally varied new examples, and align each chart with images, code, tables, summaries, and QA, making the episode interesting for listeners who want to understand whether scale and multimodal alignment can produce more reliable chart-reading AI.Sources:1. ChartNet for Robust Multimodal Chart Understandinghttps://arxiv.org/pdf/2603.270642. DVQA: Understanding Data Visualizations via Question Answering — Kushal Kafle, Brian Price, Scott Cohen, Christopher Kanan, 2018https://scholar.google.com/scholar?q=DVQA%3A+Understanding+Data+Visualizations+via+Question+Answering3. PlotQA: Reasoning over Scientific Plots — Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, Pratyush Kumar, 2019https://scholar.google.com/scholar?q=PlotQA%3A+Reasoning+over+Scientific+Plots4. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning — Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, Enamul Hoque, 2022https://scholar.google.com/scholar?q=ChartQA%3A+A+Benchmark+for+Question+Answering+about+Charts+with+Visual+and+Logical+Reasoning5. UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning — Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, Shafiq Joty, 2023https://scholar.google.com/scholar?q=UniChart%3A+A+Universal+Vision-language+Pretrained+Model+for+Chart+Comprehension+and+Reasoning6. TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learning — L. Zhang, A. Hu, H. Xu, M. Yan, Y. Xu, Q. Jin, J. Zhang, and F. Huang, 2024https://scholar.google.com/scholar?q=TinyChart%3A+Efficient+Chart+Understanding+with+Visual+Token+Merging+and+Program-of-Thoughts+Learning7. ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering — A. Masry, M. S. Islam, M. Ahmed, A. Bajaj, F. Kabir, A. Kartha, M. T. R. Laskar, M. Rahman, S. Rahman, M. Shahmohammadi, et al., 2025https://scholar.google.com/scholar?q=ChartQAPro%3A+A+More+Diverse+and+Challenging+Benchmark+for+Chart+Question+Answering8. EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart Understanding — M. Huang, H. Lai, X. Zhang, W. Wu, J. Ma, L. Zhang, and J. Liu, 2025https://scholar.google.com/scholar?q=EvoChart%3A+A+Benchmark+and+a+Self-Training+Approach+Towards+Real-World+Chart+Understanding9. ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation — C. Yang, C. Shi, Y. Liu, B. Shui, J. Wang, M. Jing, L. Xu, X. Zhu, S. Li, Y. Zhang, G. Liu, X. Nie, D. Cai, and Y. Yang, 2025https://scholar.google.com/scholar?q=ChartMimic%3A+Evaluating+LMM%27s+Cross-Modal+Reasoning+Capability+via+Chart-to-Code+Generation10. OpenCQA: Open-Ended Question Answering with Charts — S. Kantharaj, X. L. Do, R. T. K. Leong, J. Q. Tan, E. Hoque, and S. Joty, 2022https://scholar.google.com/scholar?q=OpenCQA%3A+Open-Ended+Question+Answering+with+Charts11. Effective Training Data Synthesis for Improving MLLM Chart Understanding — approximate; unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=Effective+Training+Data+Synthesis+for+Improving+MLLM+Chart+Understanding12. From Charts to Code: A Hierarchical Benchmark for Multimodal Models — approximate; unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=From+Charts+to+Code%3A+A+Hierarchical+Benchmark+for+Multimodal+Models13. GRAFT: GRaPH and Table Reasoning for Textual Alignment — approximate; unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=GRAFT%3A+GRaPH+and+Table+Reasoning+for+Textual+Alignment14. ChartQA-X: Generating Explanations for Visual Chart Reasoning — approximate; unknown from snippet, 2024/2025https://scholar.google.com/scholar?q=ChartQA-X%3A+Generating+Explanations+for+Visual+Chart+Reasoning15. AI Post Transformers: Procgen Benchmark: Measuring Generalization in Reinforcement Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/procgen-benchmark-measuring-generalization-in-reinforcement-learning/16. AI Post Transformers: The Endless Gym: Training Terminal Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/the-endless-gym-training-terminal-agents/17. AI Post Transformers: Evaluating Large Language Models Trained on Code — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/evaluating-large-language-models-trained-on-code/18. AI Post Transformers: VL-JEPA for Vision-Language Semantic Prediction — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-vl-jepa-for-vision-language-semantic-pre-69c9f4.mp319. AI Post Transformers: IMO-Bench for Robust Mathematical Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-imo-bench-for-robust-mathematical-reason-143489.mp3Interactive Visualization: ChartNet for Robust Multimodal Chart Understanding17 min
- Can LLMs Judge Their Own Capabilities?This episode explores whether large language models can estimate their chances of success before acting, update those estimates during a task, and use them to decide when to abstain from costly work. It explains why that matters for agentic systems in coding and software environments, where overconfidence can lead to wasted effort, unsafe actions, or expensive mistakes, and connects the paper to earlier work on calibration, uncertainty, and selective abstention. The discussion highlights the paper’s focus on three settings, including single-step coding, sequential decisions with feedback, and multi-step software engineering, while also stressing the distinction between raw capability, calibration, and rational decision-making. Listeners would find it interesting because it treats self-assessment not as a philosophical question, but as a practical requirement for building AI systems that know when not to act.Sources:1. Do Large Language Models Know What They Are Capable Of? — Casey O. Barkan, Sid Black, Oliver Sourbut, 2025http://arxiv.org/abs/2512.246612. Language Models (Mostly) Know What They Know — Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Ethan Perez, Nicholas Joseph, and collaborators, 2022https://scholar.google.com/scholar?q=Language+Models+%28Mostly%29+Know+What+They+Know3. Do Large Language Models Know What They Don’t Know? — Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, Xuanjing Huang, 2023https://scholar.google.com/scholar?q=Do+Large+Language+Models+Know+What+They+Don%E2%80%99t+Know%3F4. Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception — Shiyu Ni, Keping Bi, Jiafeng Guo, Lulu Yu, Baolong Bi, Xueqi Cheng, 2025https://scholar.google.com/scholar?q=Towards+Fully+Exploiting+LLM+Internal+States+to+Enhance+Knowledge+Boundary+Perception5. What Large Language Models Know and What People Think They Know — Mark Steyvers, Heliodoro Tejeda, Aakriti Kumar, Catarina Belem, Lukas W. Mayer, Padhraic Smyth, and collaborators, 2025https://scholar.google.com/scholar?q=What+Large+Language+Models+Know+and+What+People+Think+They+Know6. Teaching Models to Express Their Uncertainty in Words — Stephanie Lin, Jacob Hilton, Owain Evans, 2022https://scholar.google.com/scholar?q=Teaching+Models+to+Express+Their+Uncertainty+in+Words7. A Survey of Confidence Estimation and Calibration in Large Language Models — Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, Iryna Gurevych, 2024https://scholar.google.com/scholar?q=A+Survey+of+Confidence+Estimation+and+Calibration+in+Large+Language+Models8. Calibrating Language Models with Adaptive Temperature Scaling — Johnathan Xie, Annie S. Chen, Yoonho Lee, Eric Mitchell, Chelsea Finn, 2024https://scholar.google.com/scholar?q=Calibrating+Language+Models+with+Adaptive+Temperature+Scaling9. Calibrating the Confidence of Large Language Models by Eliciting Fidelity — Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, Xipeng Qiu, 2024https://scholar.google.com/scholar?q=Calibrating+the+Confidence+of+Large+Language+Models+by+Eliciting+Fidelity10. Selective Classification for Deep Neural Networks — Yonatan Geifman, Ran El-Yaniv, 2017https://scholar.google.com/scholar?q=Selective+Classification+for+Deep+Neural+Networks11. SelectiveNet: A Deep Neural Network with an Integrated Reject Option — Yonatan Geifman, Ran El-Yaniv, 2019https://scholar.google.com/scholar?q=SelectiveNet%3A+A+Deep+Neural+Network+with+an+Integrated+Reject+Option12. Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models — Hiyori Yoshikawa, Naoaki Okazaki, 2023https://scholar.google.com/scholar?q=Selective-LAMA%3A+Selective+Prediction+for+Confidence-Aware+Evaluation+of+Language+Models13. Confidence-Driven Multi-Scale Model Selection for Cost-Efficient Inference — Bo-Wei Chen, Chung-Chi Chen, An-Zi Yen, 2026https://scholar.google.com/scholar?q=Confidence-Driven+Multi-Scale+Model+Selection+for+Cost-Efficient+Inference14. Quantifying Uncert-AI-nty: Testing the Accuracy of LLMs' Confidence Judgments — Trent N. Cash, Daniel M. Oppenheimer, Sara Christie, Mira Devgan, 2025https://scholar.google.com/scholar?q=Quantifying+Uncert-AI-nty%3A+Testing+the+Accuracy+of+LLMs%27+Confidence+Judgments15. Credence Calibration Game? Calibrating Large Language Models Through Structured Play — Ke Fang, Tianyi Zhao, Lu Cheng, 2025https://scholar.google.com/scholar?q=Credence+Calibration+Game%3F+Calibrating+Large+Language+Models+Through+Structured+Play16. Calibration and Correctness of Language Models for Code — Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, Toufique Ahmed, 2025https://scholar.google.com/scholar?q=Calibration+and+Correctness+of+Language+Models+for+Code17. Large Language Models Must Be Taught to Know What They Don't Know — Sanyam Kapoor, Nate Gruver, Manley Roberts, Katie Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, Andrew G. Wilson, 2024https://scholar.google.com/scholar?q=Large+Language+Models+Must+Be+Taught+to+Know+What+They+Don%27t+Know18. SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration — Yuanhao Shen, Xiaodan Zhu, Lei Chen, 2024https://scholar.google.com/scholar?q=SMARTCAL%3A+An+Approach+to+Self-Aware+Tool-Use+Evaluation+and+Calibration19. Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations — Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schutze, Benjamin Roth, 2026https://scholar.google.com/scholar?q=Calibration+Is+Not+Enough%3A+Evaluating+Confidence+Estimation+Under+Language+Variations20. CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thought — Boxuan Zhang, Ruqi Zhang, 2025https://scholar.google.com/scholar?q=CoT-UQ%3A+Improving+Response-wise+Uncertainty+Quantification+in+LLMs+with+Chain-of-Thought21. Structured Uncertainty guided Clarification for LLM Agents — Manan Suri, Puneet Mathur, Nedim Lipka, Franck Dernoncourt, Ryan A. Rossi, Dinesh Manocha, 2025https://scholar.google.com/scholar?q=Structured+Uncertainty+guided+Clarification+for+LLM+Agents22. CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty — Johannes Kirmayr, Lukas Stappen, Elisabeth Andre, 2026https://scholar.google.com/scholar?q=CAR-bench%3A+Evaluating+the+Consistency+and+Limit-Awareness+of+LLM+Agents+under+Real-World+Uncertainty23. Improving Interactive In-Context Learning from Natural Language Feedback — Martin Klissarov, Jonathan Cook, Diego Antognini, Hao Sun, Jingling Li, Natasha Jaques, Claudiu Musat, Edward Grefenstette, 2026https://scholar.google.com/scholar?q=Improving+Interactive+In-Context+Learning+from+Natural+Language+Feedback24. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp325. AI Post Transformers: Program Synthesis with Large Language Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-program-synthesis-with-large-language-mo-b962ec.mp326. AI Post Transformers: SkillsBench for Evaluating Agent Skills — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-skillsbench-for-evaluating-agent-skills-58bb1e.mp327. AI Post Transformers: Stabilizing Efficient Reasoning with Step-Level Advantage Selection — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-29-stabilizing-efficient-reasoning-with-ste-1e589d.mp328. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp3Interactive Visualization: Can LLMs Judge Their Own Capabilities?13 min
- Convolutional Sequence to Sequence Learning for TranslationThis episode explores the 2017 ConvS2S paper from Facebook AI Research, which argued that sequence-to-sequence models for machine translation did not need recurrence and could instead use fully convolutional encoder-decoder networks with attention. It explains the seq2seq and neural machine translation setup, clarifies that the paper replaces recurrent computation rather than attention, and breaks down the architecture’s core ideas: stacked convolutions, expanding receptive fields, positional embeddings, residual connections, and gated linear units. The discussion highlights why the approach was provocative at the time: it challenged the LSTM-based consensus, reported stronger BLEU scores, and promised much faster, more parallelizable decoding on GPUs. Listeners would find it interesting as a key pre-transformer moment that shows how researchers were already rethinking sequential modeling, while also surfacing the tradeoff between efficiency and limited context windows for long-range dependencies.Sources:1. Convolutional Sequence to Sequence Learning — Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017http://arxiv.org/abs/1705.031222. Convolutional Sequence to Sequence Learning — Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017https://scholar.google.com/scholar?q=Convolutional+Sequence+to+Sequence+Learning3. ByteNet: Generating High-Resolution Discrete Sequences with Multiscale Dilation — Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, Koray Kavukcuoglu, 2016https://scholar.google.com/scholar?q=ByteNet%3A+Generating+High-Resolution+Discrete+Sequences+with+Multiscale+Dilation4. Language Modeling with Gated Convolutional Networks — Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier, 2017https://scholar.google.com/scholar?q=Language+Modeling+with+Gated+Convolutional+Networks5. WaveNet: A Generative Model for Raw Audio — Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, et al., 2016https://scholar.google.com/scholar?q=WaveNet%3A+A+Generative+Model+for+Raw+Audio6. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks7. Neural Machine Translation by Jointly Learning to Align and Translate — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014https://scholar.google.com/scholar?q=Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate8. Effective Approaches to Attention-based Neural Machine Translation — Minh-Thang Luong, Hieu Pham, Christopher D. Manning, 2015https://scholar.google.com/scholar?q=Effective+Approaches+to+Attention-based+Neural+Machine+Translation9. A Survey on Deep Learning for Neural Machine Translation — Antonio Toral, Víctor M. Sánchez-Cartagena, et al. (representative survey literature varies by edition), 2018https://scholar.google.com/scholar?q=A+Survey+on+Deep+Learning+for+Neural+Machine+Translation10. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation — Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, et al., 2016https://scholar.google.com/scholar?q=Google%27s+Neural+Machine+Translation+System%3A+Bridging+the+Gap+between+Human+and+Machine+Translation11. Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units12. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need13. Neural Machine Translation and Sequence-to-sequence Models: A Tutorial — Graham Neubig, 2017https://scholar.google.com/scholar?q=Neural+Machine+Translation+and+Sequence-to-sequence+Models%3A+A+Tutorial14. Neural Machine Translation in Linear Time — Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin, 2016https://scholar.google.com/scholar?q=Neural+Machine+Translation+in+Linear+Time15. ByteNet: Neural Machine Translation in Linear Time — Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, Koray Kavukcuoglu, 2016https://scholar.google.com/scholar?q=ByteNet%3A+Neural+Machine+Translation+in+Linear+Time16. Quasi-Recurrent Neural Networks — James Bradbury, Stephen Merity, Caiming Xiong, Richard Socher, 2016https://scholar.google.com/scholar?q=Quasi-Recurrent+Neural+Networks17. Convolutional Neural Network for Machine Translation — Fandong Meng, Zhengdong Lu, Hang Li, Qun Liu, 2015https://scholar.google.com/scholar?q=Convolutional+Neural+Network+for+Machine+Translation18. Resurrecting Recurrent Neural Networks for Long Sequences — approx. Orvieto et al., 2023https://scholar.google.com/scholar?q=Resurrecting+Recurrent+Neural+Networks+for+Long+Sequences19. A Comprehensive Survey on Long Context Language Modeling — approx. recent survey authors, exact authorship to verify, 2024https://scholar.google.com/scholar?q=A+Comprehensive+Survey+on+Long+Context+Language+Modeling20. Linear Recurrent Models for Robust and Interpretable Long-context Modeling — approx. recent long-context modeling authors, exact authorship to verify, 2024https://scholar.google.com/scholar?q=Linear+Recurrent+Models+for+Robust+and+Interpretable+Long-context+Modeling21. Cross-layer Attention Sharing for Pre-trained Large Language Models — approx. recent LLM systems authors, exact authorship to verify, 2024https://scholar.google.com/scholar?q=Cross-layer+Attention+Sharing+for+Pre-trained+Large+Language+Models22. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — approx. recent efficiency-focused LLM authors, exact authorship to verify, 2024https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention23. AI Post Transformers: RoPE — Hal Turing & Dr. Ada Shannon, Thu,https://podcast.do-not-panic.com/episodes/rope/24. AI Post Transformers: Apple's Speculative Streaming: Fast LLM Inference without Auxiliary Models — Hal Turing & Dr. Ada Shannon, Sat,https://podcast.do-not-panic.com/episodes/apples-speculative-streaming-fast-llm-inference-without-auxiliary-models/25. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp326. AI Post Transformers: FlatAttention for Tile-Based Accelerator Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-flatattention-for-tile-based-accelerator-56e6ca.mp313 min
- Bahdanau Attention for Neural Machine TranslationThis episode explores the 2014–2015 breakthrough paper that introduced attention to neural machine translation, framing it as a solution to a specific flaw in early encoder-decoder models: forcing an entire source sentence into one fixed-length vector. It explains how pre-attention RNN-based seq2seq systems struggled on long or complex sentences, and how Bahdanau et al.’s “soft alignment” let the decoder focus on different source words at each generation step instead of relying on a single compressed summary. Along the way, it situates the paper against phrase-based statistical translation and earlier LSTM/GRU seq2seq models, showing why attention was more than a performance tweak—it was a durable structural idea. Listeners would find it interesting for its clear account of what was actually broken before attention, what changed technically, and why this paper became a foundational step toward modern language models.Sources:1. Neural Machine Translation by Jointly Learning to Align and Translate — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014http://arxiv.org/abs/1409.04732. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks3. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, Yoshua Bengio, 2014https://scholar.google.com/scholar?q=Learning+Phrase+Representations+using+RNN+Encoder-Decoder+for+Statistical+Machine+Translation4. A Neural Network for Machine Translation, at Production Scale — Nal Kalchbrenner, Phil Blunsom, 2013https://scholar.google.com/scholar?q=A+Neural+Network+for+Machine+Translation%2C+at+Production+Scale5. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches — Kyunghyun Cho, Bart van Merrienboer, Çağlar Gülçehre, Fethi Bougares, Holger Schwenk, Yoshua Bengio, 2014https://scholar.google.com/scholar?q=On+the+Properties+of+Neural+Machine+Translation%3A+Encoder-Decoder+Approaches6. Moses: Open Source Toolkit for Statistical Machine Translation — Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al., 2007https://scholar.google.com/scholar?q=Moses%3A+Open+Source+Toolkit+for+Statistical+Machine+Translation7. Neural Machine Translation by Jointly Learning to Align and Translate — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014https://scholar.google.com/scholar?q=Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate8. Effective Approaches to Attention-based Neural Machine Translation — Minh-Thang Luong, Hieu Pham, Christopher D. Manning, 2015https://scholar.google.com/scholar?q=Effective+Approaches+to+Attention-based+Neural+Machine+Translation9. Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch, 2016https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units10. Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation — approx. Guerreiro et al., 2023https://scholar.google.com/scholar?q=Looking+for+a+Needle+in+a+Haystack%3A+A+Comprehensive+Study+of+Hallucinations+in+Neural+Machine+Translation11. Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques — approx. recent systems/LLM authors, 2024https://scholar.google.com/scholar?q=Key%2C+Value%2C+Compress%3A+A+Systematic+Exploration+of+KV+Cache+Compression+Techniques12. DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity — approx. recent systems/LLM authors, 2024https://scholar.google.com/scholar?q=DeltaKV%3A+Residual-Based+KV+Cache+Compression+via+Long-Range+Similarity13. A Survey on Large Language Model Acceleration Based on KV Cache Management — approx. recent survey authors, 2024https://scholar.google.com/scholar?q=A+Survey+on+Large+Language+Model+Acceleration+Based+on+KV+Cache+Management14. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp315. AI Post Transformers: Mamba-3 for Efficient Sequence Modeling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-16-mamba-3-for-efficient-sequence-modeling-97a22a.mp316. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp317. AI Post Transformers: Doc-to-LoRA: Internalizing Context as LoRA — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-29-doc-to-lora-internalizing-context-as-lor-8dd5ec.mp319 min
- Sequence to Sequence Learning with Neural NetworksThis episode explores the 2014 seq2seq paper by Sutskever and colleagues as a turning point in machine translation, asking what problem it actually solved and why it mattered at the time. It explains how deep LSTM encoder-decoder models reframed translation as end-to-end learning of \(p(\text{output}|\text{input})\), replacing hand-built phrase tables, alignment models, and decoding heuristics with a single learned system. The discussion highlights both the breakthrough and the limitation: compressing an entire source sentence into one fixed-length vector was elegant but created a severe bottleneck, which later made attention mechanisms so important. Listeners would find it interesting because the episode separates the paper’s real contribution from its mythology, situating it against phrase-based SMT, earlier encoder-decoder work, and the practical role of LSTMs and beam search in making the approach viable.Sources:1. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014http://arxiv.org/abs/1409.32152. Sequence to Sequence Learning with Neural Networks — Ilya Sutskever, Oriol Vinyals, Quoc V. Le, 2014https://scholar.google.com/scholar?q=Sequence+to+Sequence+Learning+with+Neural+Networks3. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation — Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, Yoshua Bengio and collaborators, 2014https://scholar.google.com/scholar?q=Learning+Phrase+Representations+using+RNN+Encoder-Decoder+for+Statistical+Machine+Translation4. Neural Machine Translation by Jointly Learning to Align and Translate — Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014https://scholar.google.com/scholar?q=Neural+Machine+Translation+by+Jointly+Learning+to+Align+and+Translate5. A Survey of the Usages of Deep Learning for Natural Language Processing — Tom Young, Devamanyu Hazarika, Soujanya Poria, Erik Cambria, 2018https://scholar.google.com/scholar?q=A+Survey+of+the+Usages+of+Deep+Learning+for+Natural+Language+Processing6. Long Short-Term Memory — Sepp Hochreiter, Jürgen Schmidhuber, 1997https://scholar.google.com/scholar?q=Long+Short-Term+Memory7. Learning to Forget: Continual Prediction with LSTM — Felix A. Gers, Jürgen Schmidhuber, Fred Cummins, 2000https://scholar.google.com/scholar?q=Learning+to+Forget%3A+Continual+Prediction+with+LSTM8. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling — Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, Yoshua Bengio, 2014https://scholar.google.com/scholar?q=Empirical+Evaluation+of+Gated+Recurrent+Neural+Networks+on+Sequence+Modeling9. An Empirical Exploration of Recurrent Network Architectures — Razvan Pascanu, Tomas Mikolov, Yoshua Bengio, 2013https://scholar.google.com/scholar?q=An+Empirical+Exploration+of+Recurrent+Network+Architectures10. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation — Yonghui Wu and many Google Brain collaborators, 2016https://scholar.google.com/scholar?q=Google%27s+Neural+Machine+Translation+System%3A+Bridging+the+Gap+between+Human+and+Machine+Translation11. A Call for Clarity in Reporting BLEU Scores — Matt Post, 2018https://scholar.google.com/scholar?q=A+Call+for+Clarity+in+Reporting+BLEU+Scores12. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches — Kyunghyun Cho and collaborators, 2014https://scholar.google.com/scholar?q=On+the+Properties+of+Neural+Machine+Translation%3A+Encoder-Decoder+Approaches13. Beam Search Strategies for Neural Machine Translation — Markus Freitag, Yaser Al-Onaizan, 2017https://scholar.google.com/scholar?q=Beam+Search+Strategies+for+Neural+Machine+Translation14. A Systematic Comparison of Search Algorithms for Neural Machine Translation — Various later comparison studies, e.g. work by Felix Stahlberg and collaborators, 2018https://scholar.google.com/scholar?q=A+Systematic+Comparison+of+Search+Algorithms+for+Neural+Machine+Translation15. Skip-Thought Vectors — Ryan Kiros, Yukun Zhu, Russ R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, Sanja Fidler, 2015https://scholar.google.com/scholar?q=Skip-Thought+Vectors16. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data — Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, Antoine Bordes, 2017https://scholar.google.com/scholar?q=Supervised+Learning+of+Universal+Sentence+Representations+from+Natural+Language+Inference+Data17. Universal Sentence Encoder — Daniel Cer, Yinfei Yang, Sheng-yi Kong and collaborators, 2018https://scholar.google.com/scholar?q=Universal+Sentence+Encoder18. A Neural Network for Machine Translation, at Production Scale — Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom, 2014https://scholar.google.com/scholar?q=A+Neural+Network+for+Machine+Translation%2C+at+Production+Scale19. Sequence Transduction with Recurrent Neural Networks — Alex Graves, 2012https://scholar.google.com/scholar?q=Sequence+Transduction+with+Recurrent+Neural+Networks20. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks — Alex Graves, Santiago Fernandez, Faustino Gomez, Jürgen Schmidhuber, 2006https://scholar.google.com/scholar?q=Connectionist+Temporal+Classification%3A+Labelling+Unsegmented+Sequence+Data+with+Recurrent+Neural+Networks21. Statistical Machine Translation: The Mathematics of SMT — Philipp Koehn, 2010https://scholar.google.com/scholar?q=Statistical+Machine+Translation%3A+The+Mathematics+of+SMT22. Recurrent Neural Network Based Language Model — Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, Sanjeev Khudanpur, 2010https://scholar.google.com/scholar?q=Recurrent+Neural+Network+Based+Language+Model23. Effective Approaches to Attention-based Neural Machine Translation — Luong, Pham, Manning, 2015https://scholar.google.com/scholar?q=Effective+Approaches+to+Attention-based+Neural+Machine+Translation24. Learning Transductions and Alignments with RNN Seq2seq Models — approx. Britz, Goldie, Luong, Le, or related Google authors, 2017https://scholar.google.com/scholar?q=Learning+Transductions+and+Alignments+with+RNN+Seq2seq+Models25. Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow, Birch, 2016https://scholar.google.com/scholar?q=Neural+Machine+Translation+of+Rare+Words+with+Subword+Units26. On Using Very Large Target Vocabulary for Neural Machine Translation — Jean, Cho, Memisevic, Bengio, 2015https://scholar.google.com/scholar?q=On+Using+Very+Large+Target+Vocabulary+for+Neural+Machine+Translation27. Attention Is All You Need — Vaswani et al., 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need28. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp329. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp319 min
- Hopfield Networks and Transformer Attention as MemoryThis episode explores how the 2021 paper “Hopfield Networks Is All You Need” reframes transformer attention as a modern continuous Hopfield network, connecting attention, associative memory, and similarity-based retrieval under one mathematical lens. It explains the core idea of content-addressable memory, contrasts classical binary Hopfield networks with newer differentiable versions, and shows why attention can be understood not just as weighted averaging but as an energy-based retrieval process with fixed points and attractor states. The discussion highlights the paper’s major claims: one-step retrieval, exponential storage capacity, low retrieval error under assumptions, and distinct retrieval regimes such as global averaging and subset averaging. Listeners interested in AI theory will find it compelling because it offers a concrete, less mystical interpretation of transformer heads and suggests practical memory-layer designs grounded in formal guarantees.Sources:1. Hopfield Networks is All You Need — Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter, 2020http://arxiv.org/abs/2008.022172. https://isl.stanford.edu/~cover/papers/transIT/0021cove.pdfhttps://isl.stanford.edu/~cover/papers/transIT/0021cove.pdf3. Neural Networks and Physical Systems with Emergent Collective Computational Abilities — John J. Hopfield, 1982https://scholar.google.com/scholar?q=Neural+Networks+and+Physical+Systems+with+Emergent+Collective+Computational+Abilities4. A Neural Network with Locality-Sensitive Hashed Dynamics — Dmitry Krotov, John J. Hopfield, 2016https://scholar.google.com/scholar?q=A+Neural+Network+with+Locality-Sensitive+Hashed+Dynamics5. Hopfield Networks is All You Need — Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, Sepp Hochreiter, 2020https://scholar.google.com/scholar?q=Hopfield+Networks+is+All+You+Need6. Dense Associative Memory for Pattern Recognition — Dmitry Krotov, John J. Hopfield, 2020https://scholar.google.com/scholar?q=Dense+Associative+Memory+for+Pattern+Recognition7. Neural Turing Machines — Alex Graves, Greg Wayne, Ivo Danihelka, 2014https://scholar.google.com/scholar?q=Neural+Turing+Machines8. Memory Networks — Jason Weston, Sumit Chopra, Antoine Bordes, 2014https://scholar.google.com/scholar?q=Memory+Networks9. End-To-End Memory Networks — Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus, 2015https://scholar.google.com/scholar?q=End-To-End+Memory+Networks10. Hybrid computing using a neural network with dynamic external memory — Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, Demis Hassabis, 2016https://scholar.google.com/scholar?q=Hybrid+computing+using+a+neural+network+with+dynamic+external+memory11. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, 2017https://scholar.google.com/scholar?q=Attention+Is+All+You+Need12. Dense Associative Memory is Robust to Adversarial Inputs — Dmitry Krotov, John J. Hopfield, 2018https://scholar.google.com/scholar?q=Dense+Associative+Memory+is+Robust+to+Adversarial+Inputs13. A Robust Exponential Associative Memory with Fixed Point Analysis — Mert Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, Franck Vermet, 2017https://scholar.google.com/scholar?q=A+Robust+Exponential+Associative+Memory+with+Fixed+Point+Analysis14. Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks — Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, Yee Whye Teh, 2019https://scholar.google.com/scholar?q=Set+Transformer%3A+A+Framework+for+Attention-based+Permutation-Invariant+Neural+Networks15. Deep Sets — Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, Alexander Smola, 2017https://scholar.google.com/scholar?q=Deep+Sets16. Modern Hopfield Networks and Attention for Immune Repertoire Classification — Michael Widrich, Bernhard Schäfl, Milena Pavlović, Hubert Ramsauer, Lukas Gruber, Markus Holzleitner, Geir Kjetil Sandve, Victor Greiff, Sepp Hochreiter, et al., 2020https://scholar.google.com/scholar?q=Modern+Hopfield+Networks+and+Attention+for+Immune+Repertoire+Classification17. Attention Heads of Large Language Models — authors unclear from snippet, likely 2024-2025https://scholar.google.com/scholar?q=Attention+Heads+of+Large+Language+Models18. Causal Head Gating: A Framework for Interpreting Roles of Attention Heads in Transformers — authors unclear from snippet, likely 2024-2025https://scholar.google.com/scholar?q=Causal+Head+Gating%3A+A+Framework+for+Interpreting+Roles+of+Attention+Heads+in+Transformers19. Mechanistic Interpretability of Fine-Tuned Vision Transformers on Distorted Images: Decoding Attention Head Behavior for Transparent and Trustworthy AI — authors unclear from snippet, likely 2024-2025https://scholar.google.com/scholar?q=Mechanistic+Interpretability+of+Fine-Tuned+Vision+Transformers+on+Distorted+Images%3A+Decoding+Attention+Head+Behavior+for+Transparent+and+Trustworthy+AI20. Iterative Sparse Attention for Long-Sequence Recommendation — authors unclear from snippet, likely recenthttps://scholar.google.com/scholar?q=Iterative+Sparse+Attention+for+Long-Sequence+Recommendation21. An Evolved Universal Transformer Memory — authors unclear from snippet, likely recenthttps://scholar.google.com/scholar?q=An+Evolved+Universal+Transformer+Memory22. AI Post Transformers: In-Place Test-Time Training for Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-09-in-place-test-time-training-for-transfor-d0b976.mp323. AI Post Transformers: RoPE — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/rope/24. AI Post Transformers: Jet-Nemotron and PostNAS for Faster Long Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-24-jet-nemotron-and-postnas-for-faster-long-436381.mp325. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp326. AI Post Transformers: Doc-to-LoRA: Internalizing Context as LoRA — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-29-doc-to-lora-internalizing-context-as-lor-8dd5ec.mp322 min
- Scaling Test-Time Compute for Reasoning ModelsThis episode explores how test-time compute should be allocated in large language models, using a recent study that compares parallel sampling, majority voting, shortest- and longest-trace selection, and beam-style search under a common evaluation setup. It explains the paper’s central argument that there is no single best inference-time strategy: some model families behave like short-horizon reasoners that benefit from several concise attempts, while others act like long-horizon reasoners that can make productive use of longer sequential reasoning. The discussion also examines how the authors benchmark eight open models across demanding datasets such as AIME and GPQA Diamond, and why harder problems reveal whether extra trace length produces real progress or just more verbose failure. Listeners would find it interesting because it turns a vague idea of “letting models think longer” into a concrete engineering question about how reasoning systems should spend their runtime budget.Sources:1. The Art of Scaling Test-Time Compute for Large Language Models — Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty, 2025http://arxiv.org/abs/2512.020082. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters3. Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning — Michael Hassid, Gabriel Synnaeve, Yossi Adi, Roy Schwartz, 2025https://scholar.google.com/scholar?q=Don%27t+Overthink+it.+Preferring+Shorter+Thinking+Chains+for+Improved+LLM+Reasoning4. Inverse Scaling in Test-Time Compute — Aryo Pradipta Gema, Alexander Hagele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, Pasquale Minervini, Yanda Chen, Joe Benton, Ethan Perez, 2025https://scholar.google.com/scholar?q=Inverse+Scaling+in+Test-Time+Compute5. Qwen3 Technical Report — An Yang and the Qwen team, 2025https://scholar.google.com/scholar?q=Qwen3+Technical+Report6. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang et al., 2023https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models7. Tree of Thoughts: Deliberate Problem Solving with Large Language Models — Shunyu Yao et al., 2023https://scholar.google.com/scholar?q=Tree+of+Thoughts%3A+Deliberate+Problem+Solving+with+Large+Language+Models8. Graph of Thoughts: Solving Elaborate Problems with Large Language Models — Maciej Besta et al., 2024https://scholar.google.com/scholar?q=Graph+of+Thoughts%3A+Solving+Elaborate+Problems+with+Large+Language+Models9. short-m@k — Ranit Hassid et al., 2025https://scholar.google.com/scholar?q=short-m%40k10. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning — approx. recent process-verifier work, exact authors not confirmed from snippet, 2025https://scholar.google.com/scholar?q=Rewarding+Progress%3A+Scaling+Automated+Process+Verifiers+for+LLM+Reasoning11. Improving LLM Reasoning Through Scaling Inference Computation With Collaborative Verification — approx. recent collaborative-verification work, exact authors not confirmed from snippet, 2025https://scholar.google.com/scholar?q=Improving+LLM+Reasoning+Through+Scaling+Inference+Computation+With+Collaborative+Verification12. Graph of Verification: Structured Verification of LLM Reasoning With Directed Acyclic Graphs — approx. recent verification-structure work, exact authors not confirmed from snippet, 2025https://scholar.google.com/scholar?q=Graph+of+Verification%3A+Structured+Verification+of+LLM+Reasoning+With+Directed+Acyclic+Graphs13. Dynamic Parallel Tree Search for Efficient LLM Reasoning — approx. recent tree-search work, exact authors not confirmed from snippet, 2025https://scholar.google.com/scholar?q=Dynamic+Parallel+Tree+Search+for+Efficient+LLM+Reasoning14. Don't Get Lost in the Trees: Streamlining LLM Reasoning by Overcoming Tree Search Exploration Pitfalls — approx. recent tree-search analysis work, exact authors not confirmed from snippet, 2025https://scholar.google.com/scholar?q=Don%27t+Get+Lost+in+the+Trees%3A+Streamlining+LLM+Reasoning+by+Overcoming+Tree+Search+Exploration+Pitfalls15. REST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search — approx. recent MCTS/process-reward work, exact authors not confirmed from snippet, 2025https://scholar.google.com/scholar?q=REST-MCTS%2A%3A+LLM+Self-Training+via+Process+Reward+Guided+Tree+Search16. Large Language Models Cannot Self-Correct Reasoning Yet — approx. recent self-correction evaluation work, exact authors not confirmed from snippet, 2024https://scholar.google.com/scholar?q=Large+Language+Models+Cannot+Self-Correct+Reasoning+Yet17. AI Post Transformers: Test-Time Scaling — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/test-time-scaling/18. AI Post Transformers: Benchmarking Test-Time Scaling for General LLM Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-benchmarking-test-time-scaling-for-gener-8f14f9.mp319. AI Post Transformers: Agentic Aggregation for Long-Horizon AI Tasks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-agentic-aggregation-for-long-horizon-ai-4c1a71.mp320. AI Post Transformers: TUMIX Multi-Agent Test-Time Scaling with Tools — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-tumix-multi-agent-test-time-scaling-with-40671c.mp321. AI Post Transformers: Test-time Scaling for Multi-Agent Collaborative Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-22-test-time-scaling-for-multi-agent-collab-082570.mp322. AI Post Transformers: The Art of Scaling Reinforcement Learning Compute for LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/the-art-of-scaling-reinforcement-learning-compute-for-llms/12 min
- Reverse-Mode Differentiation Across AD and Neural NetsThis episode explores Paul Werbos’s 2004 review of reverse differentiation and argues that reverse-mode automatic differentiation, backpropagation, hand-coded adjoints, and adjoint circuits are largely the same core idea expressed in different technical communities. It explains the mechanics of automatic differentiation and reverse mode in clear terms, then traces how these methods diverged historically and why that fragmentation slowed progress in fields like neural networks, control, and scientific computing. The discussion highlights Werbos’s main claim that better integrated, derivative-aware software could make advanced nonlinear modeling and intelligent control far more practical, while also questioning how much evidence supports that agenda beyond synthesis and historical interpretation. Listeners would find it interesting for its sharp distinction between gradients as infrastructure versus models or optimizers, and for its perspective on how today’s differentiable programming ecosystem was once a contested software vision.Sources:1. Reverse-Mode Differentiation Across AD and Neural Netshttps://www.werbos.com/AD2004.pdf2. A Simple Automatic Derivative Evaluation Program — R. E. Wengert, 1964https://scholar.google.com/scholar?q=A+Simple+Automatic+Derivative+Evaluation+Program3. Taylor Expansion of the Accumulated Rounding Error — Seppo Linnainmaa, 1976https://scholar.google.com/scholar?q=Taylor+Expansion+of+the+Accumulated+Rounding+Error4. The Complexity of Partial Derivatives — Walter Baur and Volker Strassen, 1983https://scholar.google.com/scholar?q=The+Complexity+of+Partial+Derivatives5. Automatic Differentiation in Machine Learning: a Survey — Atılım Güneş Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, Jeffrey Mark Siskind, 2018https://scholar.google.com/scholar?q=Automatic+Differentiation+in+Machine+Learning%3A+a+Survey6. Learning Representations by Back-Propagating Errors — David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams, 1986https://scholar.google.com/scholar?q=Learning+Representations+by+Back-Propagating+Errors7. Backpropagation Through Time: What It Does and How to Do It — Paul J. Werbos, 1990https://scholar.google.com/scholar?q=Backpropagation+Through+Time%3A+What+It+Does+and+How+to+Do+It8. Backpropagation Applied to Handwritten Zip Code Recognition — Yann LeCun, Bernhard Boser, John S. Denker, Don Henderson, Richard E. Howard, Wayne Hubbard, Lawrence D. Jackel, 1989https://scholar.google.com/scholar?q=Backpropagation+Applied+to+Handwritten+Zip+Code+Recognition9. Gradient-Based Learning Applied to Document Recognition — Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, 1998https://scholar.google.com/scholar?q=Gradient-Based+Learning+Applied+to+Document+Recognition10. Neuro-Dynamic Programming: An Overview — Dimitri P. Bertsekas, John N. Tsitsiklis, 1995https://scholar.google.com/scholar?q=Neuro-Dynamic+Programming%3A+An+Overview11. Neuro-Dynamic Programming — Dimitri P. Bertsekas, John N. Tsitsiklis, 1996https://scholar.google.com/scholar?q=Neuro-Dynamic+Programming12. Approximate Dynamic Programming and Reinforcement Learning — Lucian Bușoniu, Bart De Schutter, Robert Babuška, 2010https://scholar.google.com/scholar?q=Approximate+Dynamic+Programming+and+Reinforcement+Learning13. An Approximate Dynamic Programming Algorithm for Large-Scale Fleet Management: A Case Application — Hugo P. Simão, Jeff Day, Abraham P. George, Ted Gifford, John Nienow, Warren B. Powell, 2009https://scholar.google.com/scholar?q=An+Approximate+Dynamic+Programming+Algorithm+for+Large-Scale+Fleet+Management%3A+A+Case+Application14. Some New Tools for Prediction and Analysis in the Behavioral Sciences — Paul J. Werbos, 1974https://scholar.google.com/scholar?q=Some+New+Tools+for+Prediction+and+Analysis+in+the+Behavioral+Sciences15. The Difficulty of Learning Long-Term Dependencies with Gradient Descent is Officially Overcome — Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, 2001https://scholar.google.com/scholar?q=The+Difficulty+of+Learning+Long-Term+Dependencies+with+Gradient+Descent+is+Officially+Overcome16. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation — Andreas Griewank, 2000https://scholar.google.com/scholar?q=Evaluating+Derivatives%3A+Principles+and+Techniques+of+Algorithmic+Differentiation17. Differential Dynamic Programming — David H. Jacobson, David Q. Mayne, 1970https://scholar.google.com/scholar?q=Differential+Dynamic+Programming18. Backpropagation-free training of deep physical neural networks — Ali Momeni, Babak Rahmani, Matthieu Mallejac, Philipp Del Hougne, Romain Fleury, 2023https://scholar.google.com/scholar?q=Backpropagation-free+training+of+deep+physical+neural+networks19. Fully forward mode training for optical neural networks — Zhiwei Xue, Tiankuang Zhou, Zhihao Xu, Shaoliang Yu, Qionghai Dai, Lu Fang, 2024https://scholar.google.com/scholar?q=Fully+forward+mode+training+for+optical+neural+networks20. Brain-like training of a pre-sensor optical neural network with a backpropagation-free algorithm — Zheng Huang, Conghe Wang, Caihua Zhang, Wanxin Shi, Shukai Wu, Sigang Yang, Hongwei Chen, 2025https://scholar.google.com/scholar?q=Brain-like+training+of+a+pre-sensor+optical+neural+network+with+a+backpropagation-free+algorithm21. Backpropagation-Free Deep Learning with Recursive Local Representation Alignment — Alexander G. Ororbia, Ankur Mali, Daniel Kifer, C. Lee Giles, 2023https://scholar.google.com/scholar?q=Backpropagation-Free+Deep+Learning+with+Recursive+Local+Representation+Alignment22. Exploring the Promise and Limits of Real-Time Recurrent Learning — Kazuki Irie, Anand Gopalakrishnan, Jürgen Schmidhuber, 2023https://scholar.google.com/scholar?q=Exploring+the+Promise+and+Limits+of+Real-Time+Recurrent+Learning23. Real-Time Recurrent Reinforcement Learning — Julian Lemmel, Radu Grosu, 2023/2025https://scholar.google.com/scholar?q=Real-Time+Recurrent+Reinforcement+Learning24. Second-order forward-mode optimization of recurrent neural networks for neuroscience — Youjing Yu, Rui Xia, Qingxi Ma, Máté Lengyel, Guillaume Hennequin, 2024https://scholar.google.com/scholar?q=Second-order+forward-mode+optimization+of+recurrent+neural+networks+for+neuroscience25. Dynamic predictive coding: A model of hierarchical sequence learning and prediction in the neocortex — Linxing Preston Jiang, Rajesh P. N. Rao, 2024https://scholar.google.com/scholar?q=Dynamic+predictive+coding%3A+A+model+of+hierarchical+sequence+learning+and+prediction+in+the+neocortex26. Predictive coding networks for temporal prediction — Beren Millidge, Mufeng Tang, Mahyar Osanlouy, Nicol S. Harper, Rafal Bogacz, 2024https://scholar.google.com/scholar?q=Predictive+coding+networks+for+temporal+prediction27. Where is the error? Hierarchical predictive coding through dendritic error computation — Fabian A. Mikulasch, Lucas Rudelt, Michael Wibral, Viola Priesemann, 2023https://scholar.google.com/scholar?q=Where+is+the+error%3F+Hierarchical+predictive+coding+through+dendritic+error+computation28. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp329. AI Post Transformers: When Spectral Gradient Updates Help Deep Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-when-spectral-gradient-updates-help-deep-9c8441.mp330. AI Post Transformers: ASI-Evolve for Data, Architectures, and RL — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-05-asi-evolve-for-data-architectures-and-rl-197b2b.mp3Interactive Visualization: Reverse-Mode Differentiation Across AD and Neural Nets11 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.