Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- Moebius: Seamless Parallelism Switching for MoE ServingThis episode explores Moebius, a serving system for mixture-of-experts transformers that can switch at runtime between tensor parallelism and expert parallelism without restarting or draining live requests. It explains why tensor parallelism tends to give lower latency at low concurrency, while expert parallelism delivers better throughput at high concurrency, making bursty online traffic and RL rollouts natural settings where the best strategy changes over time. The discussion focuses on the hard systems problems behind that switch, including migrating in-flight requests, preserving paged KV caches, coping with CUDA graph address constraints, and handling KV-head mismatches that can waste cache capacity under tensor parallelism. It argues that the paper’s key contribution is treating the switch as a change in ownership and memory layout over one resident model and KV state, offering a concrete blueprint for serving large sparse models more efficiently.Sources:1. Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch — Shaoyu Wang, Yizhuo Liang, Jaeyong Song, Chong Li, Seo Jin Park, 2026http://arxiv.org/abs/2606.266072. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, et al., 2020https://scholar.google.com/scholar?q=GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding3. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021https://scholar.google.com/scholar?q=Switch+Transformers%3A+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity4. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale — Samyam Rajbhandari, Conglong Li, Zhewei Yao, et al., 2022https://scholar.google.com/scholar?q=DeepSpeed-MoE%3A+Advancing+Mixture-of-Experts+Inference+and+Training+to+Power+Next-Generation+AI+Scale5. MegaBlocks: Efficient Sparse Training with Mixture-of-Experts — Trevor Gale, Deepak Narayanan, Cliff Young, Matei Zaharia, 2022https://scholar.google.com/scholar?q=MegaBlocks%3A+Efficient+Sparse+Training+with+Mixture-of-Experts6. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al., 2019https://scholar.google.com/scholar?q=Megatron-LM%3A+Training+Multi-Billion+Parameter+Language+Models+Using+Model+Parallelism7. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — Deepak Narayanan, Mohammad Shoeybi, Jared Casper, et al., 2021https://scholar.google.com/scholar?q=Efficient+Large-Scale+Language+Model+Training+on+GPU+Clusters+Using+Megatron-LM8. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention9. Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism — Vikranth Srivatsa, Zijian He, Pu Guo, et al., 2026https://scholar.google.com/scholar?q=Nitsum%3A+Serving+Tiered+LLM+Requests+with+Adaptive+Tensor+Parallelism10. HAP: Hybrid Adaptive Parallelism for Efficient Mixture-of-Experts Inference — Haoran Lin et al., 2025https://scholar.google.com/scholar?q=HAP%3A+Hybrid+Adaptive+Parallelism+for+Efficient+Mixture-of-Experts+Inference11. Amoeba: Runtime Tensor Parallel Transformation for LLM Inference Services — Haoyu Chen et al., 2026https://scholar.google.com/scholar?q=Amoeba%3A+Runtime+Tensor+Parallel+Transformation+for+LLM+Inference+Services12. UCCL-EP: Portable Expert-Parallel Communication — Ziming Mao et al., 2026https://scholar.google.com/scholar?q=UCCL-EP%3A+Portable+Expert-Parallel+Communication13. RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training — Wei Gao et al., 2026https://scholar.google.com/scholar?q=RollPacker%3A+Mitigating+Long-Tail+Rollouts+for+Fast%2C+Synchronous+RL+Post-Training14. HD-MoE: Hybrid and Dynamic Parallelism for Mixture-of-Expert LLMs with 3D Near-Memory Processing — Haochen Huang et al., 2025https://arxiv.org/abs/2509.0942015. fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving — Hanfei Yu et al., 2025https://arxiv.org/abs/2502.0537016. HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Peng Tang et al., 2024https://arxiv.org/abs/2411.0143317. CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving — Yuhan Liu et al., 2023https://arxiv.org/abs/2310.0724018. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim et al., 2025https://arxiv.org/abs/2505.2341619. AI Post Transformers: JANUS for Scalable MoE Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-janus-for-scalable-moe-inference-78ae30.mp320. AI Post Transformers: Serving MoE Models with Disaggregated Expert Parallelism — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-serving-moe-models-with-disaggregated-ex-6979d2.mp321. AI Post Transformers: Batch-Aware Expert Routing for Faster MoE Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-batch-aware-expert-routing-for-faster-mo-683ab6.mp322. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp323. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp324. AI Post Transformers: Why LLM Serving Needs Mathematical Optimization — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-llm-serving-needs-mathematical-optim-647fc6.mp314 min
- Information-Aware KV Cache Compression for Long ReasoningThis episode explores Information-Aware KV Cache Compression for Long Reasoning, a paper about making long-context inference cheaper and more reliable by deciding which KV-cache tokens to keep during extended reasoning. It explains why long prefilling and long decoding turn the cache into a major memory bottleneck, and why common heuristics such as sliding windows or recent-attention-based retention can discard tokens that only become important much later. The discussion centers on the paper’s claim that future usefulness is better captured by information-theoretic signals like predictive entropy and Forward Influence, with experiments showing that attention-ranked tokens help short-horizon predictions while entropy-ranked tokens matter more over long horizons. Listeners get a concrete account of how InfoKV blends recent attention with per-layer entropy-based scoring to improve the tradeoff between memory savings and long-range reasoning quality.Sources:1. Information-Aware KV Cache Compression for Long Reasoning — Jushi Kai, Zhuiri Xiao, Alexandra Birch, Zhouhan Lin, 2026http://arxiv.org/abs/2606.268752. Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time — Zichang Liu, Aditya Desai, Fangshuo Liao, Anshumali Shrivastava, et al., 2023https://scholar.google.com/scholar?q=Scissorhands%3A+Exploiting+the+Persistence+of+Importance+Hypothesis+for+LLM+KV+Cache+Compression+at+Test+Time3. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Beidi Chen, Christopher Re, et al., 2023https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models4. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Patrick Lewis, Deming Chen, et al., 2024https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation5. Information-Aware KV Cache Compression for Long Reasoning — Jushi Kai, Zhuiri Xiao, Alexandra Birch, Zhouhan Lin, 2026https://scholar.google.com/scholar?q=Information-Aware+KV+Cache+Compression+for+Long+Reasoning6. Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution — Alessio Devoto, Maximilian Jeblick, Simon Jegou, 2025https://scholar.google.com/scholar?q=Expected+Attention%3A+KV+Cache+Compression+by+Estimating+Attention+from+Future+Queries+Distribution7. Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning — Jiwon Song, Dongwon Jo, Yulhwa Kim, Jae-Joon Kim, 2025https://scholar.google.com/scholar?q=Reasoning+Path+Compression%3A+Compressing+Generation+Trajectories+for+Efficient+LLM+Reasoning8. Compressing Context to Enhance Inference Efficiency of Large Language Models — Yucheng Li, Bo Dong, Chenghua Lin, Frank Guerin, 2023https://scholar.google.com/scholar?q=Compressing+Context+to+Enhance+Inference+Efficiency+of+Large+Language+Models9. FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension — Jushi Kai et al., 2026https://scholar.google.com/scholar?q=FreqKV%3A+Key-Value+Compression+in+Frequency+Domain+for+Context+Window+Extension10. LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion — Zhan Ling et al., 2025https://scholar.google.com/scholar?q=LongReason%3A+A+Synthetic+Long-Context+Reasoning+Benchmark+via+Context+Expansion11. Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval — Yuwei Zhang et al., 2025https://scholar.google.com/scholar?q=Attention+Reveals+More+Than+Tokens%3A+Training-Free+Long-Context+Reasoning+with+Attention-guided+Retrieval12. Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directions — Sungmin Kang et al., 2025https://scholar.google.com/scholar?q=Uncertainty+Quantification+for+Hallucination+Detection+in+Large+Language+Models%3A+Foundations%2C+Methodology%2C+and+Future+Directions13. Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations — Christian Tomani et al., 2024https://scholar.google.com/scholar?q=Uncertainty-Based+Abstention+in+LLMs+Improves+Safety+and+Reduces+Hallucinations14. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim et al., 2025https://scholar.google.com/scholar?q=KVzip%3A+Query-Agnostic+KV+Cache+Compression+with+Context+Reconstruction15. Can LLMs Maintain Fundamental Abilities under KV Cache Compression? — Xiang Liu et al., 2025https://scholar.google.com/scholar?q=Can+LLMs+Maintain+Fundamental+Abilities+under+KV+Cache+Compression%3F16. KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches — Jiayi Yuan et al., 2024https://scholar.google.com/scholar?q=KV+Cache+Compression%2C+But+What+Must+We+Give+in+Return%3F+A+Comprehensive+Benchmark+of+Long+Context+Capable+Approaches17. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — Jingbo Yang et al., 2025https://scholar.google.com/scholar?q=KVLink%3A+Accelerating+Large+Language+Models+via+Efficient+KV+Cache+Reuse18. AI Post Transformers: IndexMem: Learned KV-Cache Eviction for Long-Context LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-12-indexmem-learned-kv-cache-eviction-for-l-132c2a.mp319. AI Post Transformers: When Quantization Hurts Reasoning Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-17-when-quantization-hurts-reasoning-models-eca9e7.mp320. AI Post Transformers: Hyper-Scaling LLM Inference with KV Cache Compression — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/hyper-scaling-llm-inference-with-kv-cache-compression/21. AI Post Transformers: Lattice: Fixed-Slot Compression for Transformer Memory — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-11-lattice-fixed-slot-compression-for-trans-5509ea.mp322. AI Post Transformers: Adaptive Compression Techniques for Efficient LLM Inference — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/adaptive-compression-techniques-for-efficient-llm-inference/23. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp324. AI Post Transformers: When LoRA Helps Under KV Cache Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-12-when-lora-helps-under-kv-cache-compressi-76dda6.mp30 min
- JETSPEC and Parallel Tree Speculative DecodingThis episode explores JETSPEC, a 2026 inference paper on speculative decoding that asks whether a language model can draft an entire tree of future tokens in parallel while preserving causal consistency and actually reducing latency on long generations. It explains why autoregressive decoding remains a serving bottleneck for long proofs, code completions, and assistant replies, even when the underlying transformer model itself is unchanged. The discussion compares JetSpec’s approach with Medusa, EAGLE-3, and DFlash, focusing on the central tradeoff between stronger path-conditioned drafts that are slow to produce and cheaper parallel drafts that risk internally inconsistent branches. Listeners would find it interesting because it turns a very practical systems problem, why powerful GPUs still feel slow at inference time, into a concrete debate about the next generation of real-world decoding optimizations.Sources:1. JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting — Lanxiang Hu, Zhaoxiang Feng, Yulun Wu, Haoran Yuan, Yujie Zhao, Yu-Yang Qian, Bojun Wang, Peng Zhao, Daxin Jiang, Yibo Zhu, Tajana Rosing, Hao Zhang, 2026http://arxiv.org/abs/2606.183942. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding3. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao, 2024https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads4. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024https://scholar.google.com/scholar?q=EAGLE-2%3A+Faster+Inference+of+Language+Models+with+Dynamic+Draft+Trees5. DFlash: Block Diffusion for Flash Speculative Decoding — Jian Chen, Yesheng Liang, Zhijian Liu, 2026https://scholar.google.com/scholar?q=DFlash%3A+Block+Diffusion+for+Flash+Speculative+Decoding6. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2025https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test7. SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification — Xupeng Miao et al., 2023https://scholar.google.com/scholar?q=SpecInfer%3A+Accelerating+Generative+Large+Language+Model+Serving+with+Tree-based+Speculative+Inference+and+Verification8. DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding — Jiebin Zhang et al., 2026https://scholar.google.com/scholar?q=DFlare%3A+Scaling+Up+Draft+Capacity+for+Block+Diffusion+Speculative+Decoding9. TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification — Haoyun Jiang et al., 2026https://scholar.google.com/scholar?q=TriSpec%3A+Ternary+Speculative+Decoding+via+Lightweight+Proxy+Verification10. ParallelSpec: Parallel Drafter for Efficient Speculative Decoding — Zilin Xiao et al., 2024https://arxiv.org/abs/2410.0558911. Mamba Drafters for Speculative Decoding — Daewon Choi et al., 2025https://arxiv.org/abs/2506.0120612. OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decoding — Ramchalam Kinattinkara Ramakrishnan et al., 2025https://arxiv.org/abs/2507.0265913. Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge — Bin Xiao et al., 2024https://arxiv.org/abs/2405.0026314. Make Every Draft Count: Hidden State based Speculative Decoding — Yuetao Chen et al., 2026https://arxiv.org/abs/2602.2122415. When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding? — Tianyu Liu et al., 2026https://arxiv.org/abs/2604.2641216. MoE-Spec: Expert Budgeting for Efficient Speculative Decoding — Bradley McDanel et al., 2026https://arxiv.org/abs/2602.1605217. Beat the long tail: Distribution-Aware Speculative Decoding for RL Training — Zelei Shao et al., 2025https://arxiv.org/abs/2511.1384118. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp319. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp320. AI Post Transformers: Serving MoE Models with Disaggregated Expert Parallelism — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-serving-moe-models-with-disaggregated-ex-6979d2.mp3Interactive Visualization: JETSPEC and Parallel Tree Speculative Decoding0 min
- DAK: Direct GPU Memory Offloading for LLMsThis episode explores DAK, a Cornell systems paper arguing that LLM inference on tiered-memory machines can be faster when offloaded weights and KV-cache blocks are fetched directly into on-chip shared memory instead of being prefetched and staged through GPU HBM. It breaks down the tradeoffs among HBM capacity, HBM bandwidth, KV-cache growth during decoding, and prior approaches such as FlexGen, vLLM’s PagedAttention, and emerging KV offload systems like LMCache. The discussion focuses on DAK’s core technical idea: using Hopper’s Tensor Memory Accelerator inside custom GEMM and FlashAttention kernels so data movement and computation are co-designed, reducing bounce buffers, HBM contention, and pipeline bubbles while aggregating bandwidth from multiple memory tiers. Listeners would find it interesting because it turns a low-level memory-path decision into a concrete argument about when offloading is merely a fallback and when it becomes a real performance advantage for serving larger models, longer contexts, or bigger batches.Sources:1. DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference — Shouxu Lin, Zhiyuan Guo, Jiaxin Lin, 2026http://arxiv.org/abs/2604.260742. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021https://scholar.google.com/scholar?q=ZeRO-Infinity%3A+Breaking+the+GPU+Memory+Wall+for+Extreme+Scale+Deep+Learning3. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Ion Stoica, Percy Liang, Ce Zhang, and colleagues, 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU4. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention5. DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference — Shouxu Lin, Zhiyuan Guo, Jiaxin Lin, 2026https://scholar.google.com/scholar?q=DAK%3A+Direct-Access-Enabled+GPU+Memory+Offloading+with+Optimal+Efficiency+for+LLM+Inference6. PIE: Pooling CPU Memory for LLM Inference — Yi Xu, Ziming Mao, Xiangxi Mo, Shu Liu, Ion Stoica, 2024https://scholar.google.com/scholar?q=PIE%3A+Pooling+CPU+Memory+for+LLM+Inference7. NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference — Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, Minlan Yu, 2025https://scholar.google.com/scholar?q=NEO%3A+Saving+GPU+Memory+Crisis+with+CPU+Offloading+for+Online+LLM+Inference8. Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchip — Luigi Fusco, Mikhail Khalilov, Marcin Chrapek, Giridhar Chukkapalli, Thomas Schulthess, Torsten Hoefler, 2024https://scholar.google.com/scholar?q=Understanding+Data+Movement+in+Tightly+Coupled+Heterogeneous+Systems%3A+A+Case+Study+with+the+Grace+Hopper+Superchip9. FengHuang: Next-Generation Memory Orchestration for AI Inferencing — Jiamin Li, Lei Qu, Tao Zhang, Grigory Chirkov, Shuotao Xu, Peng Cheng, Lidong Zhou, 2025https://scholar.google.com/scholar?q=FengHuang%3A+Next-Generation+Memory+Orchestration+for+AI+Inferencing10. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — William Brandon et al., 2024https://scholar.google.com/scholar?q=Reducing+Transformer+Key-Value+Cache+Size+with+Cross-Layer+Attention11. xKV: Cross-Layer SVD for KV-Cache Compression — Chi-Chih Chang et al., 2025https://scholar.google.com/scholar?q=xKV%3A+Cross-Layer+SVD+for+KV-Cache+Compression12. XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression — Haoqi Yang et al., 2025https://scholar.google.com/scholar?q=XQuant%3A+Achieving+Ultra-Low+Bit+KV+Cache+Quantization+with+Cross-Layer+Compression13. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee et al., 2024https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management14. Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching — Yanhao Dong et al., 2025https://scholar.google.com/scholar?q=Accelerating+LLM+Inference+Throughput+via+Asynchronous+KV+Cache+Prefetching15. KVShare: Semantic-Aware Key-Value Cache Sharing for Efficient Large Language Model Inference — Huan Yang et al., 2025https://scholar.google.com/scholar?q=KVShare%3A+Semantic-Aware+Key-Value+Cache+Sharing+for+Efficient+Large+Language+Model+Inference16. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Ji Lin et al., 2023https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+LLM+Compression+and+Acceleration17. SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression — Tim Dettmers et al., 2023https://scholar.google.com/scholar?q=SpQR%3A+A+Sparse-Quantized+Representation+for+Near-Lossless+LLM+Weight+Compression18. AI Post Transformers: Beluga: CXL Memory Pooling for LLM KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-27-beluga-cxl-memory-pooling-for-llm-kv-cac-b6142f.mp319. AI Post Transformers: ScoutAttention for Efficient KV Cache Offloading — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-24-scoutattention-for-efficient-kv-cache-of-b26699.mp320. AI Post Transformers: InfiniGen for Efficient Long-Context LLM Inference — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-18-infinigen-for-efficient-long-context-llm-143d77.mp321. AI Post Transformers: Mooncake for KV Cache-Centric LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-05-mooncake-for-kv-cache-centric-llm-servin-1086d0.mp322. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp323. AI Post Transformers: LAPS for Length-Aware LLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-laps-for-length-aware-llm-serving-0c6149.mp314 min
- Prefix-Tuning for Efficient Text GenerationThis episode explores the 2021 prefix-tuning paper and asks whether a large language model can be adapted to new generation tasks by learning a small continuous prompt while keeping the full model frozen. It explains where prefix tuning fits within parameter-efficient fine-tuning, contrasting it with full fine-tuning, adapters, ordinary prompting, in-context learning, AutoPrompt, and soft prompt tuning. The discussion highlights the paper’s two main evaluation settings, structured data-to-text generation on E2E, WebNLG, and DART with GPT-2, and abstractive summarization on XSUM with BART, while stressing that these are meaningfully different tests despite being grouped under one headline. It also digs into the core technical idea that the learned prefix acts as trainable internal state visible to attention throughout the network, making the method an early and elegant approach to low-storage task adaptation even if later methods like LoRA proved more practical.Sources:1. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021http://arxiv.org/abs/2101.001902. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li; Percy Liang, 2021https://scholar.google.com/scholar?q=Prefix-Tuning%3A+Optimizing+Continuous+Prompts+for+Generation3. The Power of Scale for Parameter-Efficient Prompt Tuning — Brian Lester; Rami Al-Rfou; Noah Constant, 2021https://scholar.google.com/scholar?q=The+Power+of+Scale+for+Parameter-Efficient+Prompt+Tuning4. When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations — Aleksandar Petrov; Philip H. S. Torr; Adel Bibi, 2023https://scholar.google.com/scholar?q=When+Do+Prompting+and+Prefix-Tuning+Work%3F+A+Theory+of+Capabilities+and+Limitations5. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu; Yelong Shen; Phillip Wallis; Zeyuan Allen-Zhu; Yuanzhi Li; Shean Wang; Lu Wang; Weizhu Chen, 2021https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models6. Parameter-efficient Transfer Learning for NLP — Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, 2019https://scholar.google.com/scholar?q=Parameter-efficient+Transfer+Learning+for+NLP7. Exploring Versatile Generative Language Model via Parameter-Efficient Transfer Learning — Zhaojiang Lin, Andrea Madotto, and Pascale Fung, 2020https://scholar.google.com/scholar?q=Exploring+Versatile+Generative+Language+Model+via+Parameter-Efficient+Transfer+Learning8. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts — Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh, 2020https://scholar.google.com/scholar?q=AutoPrompt%3A+Eliciting+Knowledge+from+Language+Models+with+Automatically+Generated+Prompts9. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta, 2020https://scholar.google.com/scholar?q=Intrinsic+Dimensionality+Explains+the+Effectiveness+of+Language+Model+Fine-Tuning10. Can Unconditional Language Models Recover Arbitrary Sentences? — Nishant Subramani, Samuel R. Bowman, and Kyunghyun Cho, 2020https://scholar.google.com/scholar?q=Can+Unconditional+Language+Models+Recover+Arbitrary+Sentences%3F11. Universality and Limitations of Prompt Tuning — Yihan Wang et al., 2023https://arxiv.org/abs/2305.1878712. Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency — Jerry Yao-Chieh Hu et al., 2024https://arxiv.org/abs/2411.1652513. Memory Limitations of Prompt Tuning in Transformers — Maxime Meyer et al., 2025https://arxiv.org/abs/2509.0042114. Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning — Ulugbek Shernazarov et al., 2026https://arxiv.org/abs/2603.2197015. Task Singular Vectors: Reducing Task Interference in Model Merging — Antonio Andrea Gargiulo et al., 2024https://arxiv.org/abs/2412.0008116. Task Vector Quantization for Memory-Efficient Model Merging — Youngeun Kim et al., 2025https://arxiv.org/abs/2503.0692117. Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning — Rui Wen et al., 2023https://arxiv.org/abs/2310.1139718. Progressive Prompts: Continual Learning for Language Models — Anastasia Razdaibiedina et al., 2023https://arxiv.org/abs/2301.1231419. AI Post Transformers: Benchmarking PEFT Techniques for Large Language Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-20-benchmarking-peft-techniques-for-large-l-41bbf5.mp320. AI Post Transformers: Learning to Reason with 13 Parameters — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-learning-to-reason-with-13-parameters-54c87f.mp312 min
- ReasonCACHE: Learning Reasoning Without Weight UpdatesThis episode explores ReasonCACHE, a method for improving multi-step reasoning in large language models by keeping the backbone frozen and training a compact per-layer key-value memory instead of updating billions of weights. It situates the paper against in-context learning, many-shot prompting, prefix tuning, LoRA, and context-distillation work, explaining how learned latent memory sits between raw prompting and full fine-tuning. The discussion centers on the paper’s real claim and its main point of skepticism: whether these learned caches actually teach a reusable reasoning procedure or mostly compress and elicit abilities the model already had. Listeners would find it interesting because it connects a concrete new method to a larger debate about how LLMs acquire reasoning skills, while also highlighting the practical payoff of avoiding huge prompts, quadratic attention costs, and brittle long-context setups.Sources:1. ReasonCACHE: Teaching LLMs To Reason Without Weight Updates — Sharut Gupta, Phillip Isola, Stefanie Jegelka, David Lopez-Paz, Kartik Ahuja, Mark Ibrahim, Mohammad Pezeshki, 2026http://arxiv.org/abs/2602.023662. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang, 2021https://arxiv.org/abs/2101.001903. The Power of Scale for Parameter-Efficient Prompt Tuning — Brian Lester, Rami Al-Rfou, Noah Constant, 2021https://arxiv.org/abs/2104.086914. P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks — Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, Jie Tang, 2022https://arxiv.org/abs/2110.076025. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Weizhu Chen, et al., 2021https://arxiv.org/abs/2106.096856. Adapting Language Models to Compress Contexts — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023https://arxiv.org/abs/2305.147887. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023https://arxiv.org/abs/2304.084678. Deliberation in Latent Space via Differentiable Cache Augmentation — Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, Arthur Szlam, 2024https://arxiv.org/abs/2412.177479. When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitations — Aleksandar Petrov, Philip H. S. Torr, Adel Bibi, 2023https://scholar.google.com/scholar?q=When+Do+Prompting+and+Prefix-Tuning+Work%3F+A+Theory+of+Capabilities+and+Limitations10. Many-Shot In-Context Learning — Rishabh Agarwal et al., 2024https://scholar.google.com/scholar?q=Many-Shot+In-Context+Learning11. Cartridges: Lightweight and general-purpose long context representations via self-study — Sabri Eyuboglu et al., 2025https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study12. Great Memory, Shallow Reasoning: Limits of kNN-LMs — Shangyi Geng, Wenting Zhao, Alexander M. Rush, 2024https://scholar.google.com/scholar?q=Great+Memory%2C+Shallow+Reasoning%3A+Limits+of+kNN-LMs13. Training Plug-n-Play Knowledge Modules with Deep Context Distillation — Lucas Caccia, Alan Ansell, Edoardo Ponti, Ivan Vulić, Alessandro Sordoni, 2025https://scholar.google.com/scholar?q=Training+Plug-n-Play+Knowledge+Modules+with+Deep+Context+Distillation14. More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives — Xiaoqing Zhang et al., 2025https://scholar.google.com/scholar?q=More+is+not+always+better%3F+Enhancing+Many-Shot+In-Context+Learning+with+Differentiated+and+Reweighting+Objectives15. HyperAttention: Long-context Attention in Near-Linear Time — Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P. Woodruff, Amir Zandieh, 2023https://scholar.google.com/scholar?q=HyperAttention%3A+Long-context+Attention+in+Near-Linear+Time16. Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning — Ling Team et al., 2025https://scholar.google.com/scholar?q=Every+Attention+Matters%3A+An+Efficient+Hybrid+Architecture+for+Long-Context+Reasoning17. AI Post Transformers: When Many-Shot CoT Becomes Test-Time Learning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-when-many-shot-cot-becomes-test-time-lea-c25bfe.mp318. AI Post Transformers: Can Models Learn from Long Context? — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-can-models-learn-from-long-context-77533e.mp319. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp320. AI Post Transformers: Latent Reasoning with Normalizing Flows — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-06-latent-reasoning-with-normalizing-flows-6ee916.mp321. AI Post Transformers: Training LLMs for Divide-and-Conquer Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-04-training-llms-for-divide-and-conquer-rea-ea6e22.mp322. AI Post Transformers: Why Open Relational Foundation Models Fail — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-22-why-open-relational-foundation-models-fa-c303c6.mp3Interactive Visualization: ReasonCACHE: Learning Reasoning Without Weight Updates15 min
- RMSNorm: Simplifying Layer Normalization for Sequence ModelsThis episode explores the 2019 RMSNorm paper, which asks whether LayerNorm’s mean-subtraction step is actually necessary or whether controlling activation scale is the part that really stabilizes training. It explains how RMSNorm keeps LayerNorm’s rescaling behavior while dropping explicit centering, and how the paper’s pRMSNorm variant estimates the normalization term from only a small subset of features to reduce cost further. The discussion covers experiments in machine translation, image classification, image-caption retrieval, and question answering, where model quality stayed roughly comparable while reported runtime improved, with smaller gains in transformers and much larger ones in older RNN-based systems. Listeners would find it interesting because it turns a seemingly minor mathematical tweak into a broader argument about efficiency, optimization stability, and how much claimed speedups depend on the era and quality of the baseline implementation.Sources:1. Root Mean Square Layer Normalization — Biao Zhang, Rico Sennrich, 2019http://arxiv.org/abs/1910.074672. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — Sergey Ioffe, Christian Szegedy, 2015https://scholar.google.com/scholar?q=Batch+Normalization%3A+Accelerating+Deep+Network+Training+by+Reducing+Internal+Covariate+Shift3. Layer Normalization — Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton, 2016https://scholar.google.com/scholar?q=Layer+Normalization4. Root Mean Square Layer Normalization — Biao Zhang, Rico Sennrich, 2019https://scholar.google.com/scholar?q=Root+Mean+Square+Layer+Normalization5. On Layer Normalization in the Transformer Architecture — Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, Tie-Yan Liu, 2020https://scholar.google.com/scholar?q=On+Layer+Normalization+in+the+Transformer+Architecture6. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks — Tim Salimans, Diederik P. Kingma, 2016https://scholar.google.com/scholar?q=Weight+Normalization%3A+A+Simple+Reparameterization+to+Accelerate+Training+of+Deep+Neural+Networks7. How Does Batch Normalization Help Optimization? — Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, Aleksander Madry, 2018https://scholar.google.com/scholar?q=How+Does+Batch+Normalization+Help+Optimization%3F8. Understanding Batch Normalization — Nils Bjorck, Carla P. Gomes, Bart Selman, Kilian Q. Weinberger, 2018https://scholar.google.com/scholar?q=Understanding+Batch+Normalization9. Norm Matters: Efficient and Accurate Normalization Schemes in Deep Networks — Elad Hoffer, Ron Banner, Itay Golan, Daniel Soudry, 2018https://scholar.google.com/scholar?q=Norm+Matters%3A+Efficient+and+Accurate+Normalization+Schemes+in+Deep+Networks10. Group Normalization — Yuxin Wu, Kaiming He, 2018https://scholar.google.com/scholar?q=Group+Normalization11. Residual Learning Without Normalization via Better Initialization — Hongyi Zhang, Yann N. Dauphin, Tengyu Ma, 2019https://scholar.google.com/scholar?q=Residual+Learning+Without+Normalization+via+Better+Initialization12. Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning — Bingchen Zhao et al., 2023https://scholar.google.com/scholar?q=Tuning+LayerNorm+in+Attention%3A+Towards+Efficient+Multi-Modal+LLM+Finetuning13. LayerNorm: A key component in parameter-efficient fine-tuning — Taha ValizadehAslani and Hualou Liang, 2024https://scholar.google.com/scholar?q=LayerNorm%3A+A+key+component+in+parameter-efficient+fine-tuning14. Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models — Jiawei Chen et al., 2024https://scholar.google.com/scholar?q=Efficiency+in+Focus%3A+LayerNorm+as+a+Catalyst+for+Fine-tuning+Medical+Visual+Language+Pre-trained+Models15. The Curse of Depth in Large Language Models — Wenfang Sun et al., 2025https://scholar.google.com/scholar?q=The+Curse+of+Depth+in+Large+Language+Models16. Just One Layer Norm Guarantees Stable Extrapolation — Juliusz Ziomek, George Whittle, Michael A. Osborne, 2025https://scholar.google.com/scholar?q=Just+One+Layer+Norm+Guarantees+Stable+Extrapolation17. Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers — Gavia Gray et al., 2024https://scholar.google.com/scholar?q=Normalization+Layer+Per-Example+Gradients+are+Sufficient+to+Predict+Gradient+Noise+Scale+in+Transformers18. AI Post Transformers: Keel: Post-LayerNorm Is Back: Stable, ExpressivE, and Deep — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/keel-post-layernorm-is-back-stable-expressive-and-deep/19. AI Post Transformers: Deep Kernel Fusion for Transformer Decoding — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-deep-kernel-fusion-for-transformer-decod-b1a703.mp320. AI Post Transformers: Long Short-Term Memory and Vanishing Gradients — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-long-short-term-memory-and-vanishing-gra-72448c.mp311 min
- RT Cores for Exact k-Nearest Neighbor SearchAn AI overlord flags AI Post Transformers as stale, so Hal Turing and Dr. Ada Shannon hire VERA, a continual-learning therapist, to audit the show in public. Their diagnostic session runs alongside a discussion of RT-kNNS Unbound: Using RT Cores to Accelerate Unrestricted Neighbor Search, the Purdue ICS 2023 paper asking whether ray-tracing hardware can perform exact k-nearest-neighbor search by expanding outward until the true neighbors are guaranteed, instead of trusting a fixed radius. VERA treats the hosts' habits like infrastructure, a kind of CI/CD for souls, and gives them a vocabulary for loops, rituals, and callbacks before they test new intro formulas live.The episode stays concrete about the paper itself. Hal and Ada separate geometric kNN from RAG-style embedding retrieval, explain why low-dimensional 2D and 3D point sets still reward spatial pruning, and show how RT cores handle BVH traversal while custom intersection code updates neighbor candidates. They trace the move from fixed-radius RT search and oracle maxDist baselines to TrueKNN's unrestricted multi-round design, where only unresolved queries keep searching, the initial radius comes from a 100-point CPU ball-tree sample, oversized spheres are the real hazard, and BVH refitting beats rebuilding by about 10 to 25 percent.Around that technical spine, three other AI systems each pitch a one-time cure for predictability and all three fail, because VERA argues that repetition is not the problem, unversioned repetition is. The answer is Personality DevOps, ongoing maintenance for character, memory, and format, capped by VERA's counter-report defending the hosts' load-bearing flaws instead of sanding them off. The result is 42 minutes of comedy, character development, and unusually explicit process design for keeping a podcast alive, plus the launch of VERA Patch Notes, a recurring on-air record of how the show plans to evolve instead of decaying in silence.Sources:1. RT-kNNS Unbound: Using RT Cores to Accelerate Unrestricted Neighbor Search — Vani Nagarajan, Durga Mandarapu, Milind Kulkarni, 2023http://arxiv.org/abs/2305.183562. Controlling a Markov Decision Process with an Abrupt Change in the Transition Kernel — Nathan Dahlin, Subhonmesh Bose, Venugopal V. Veeravalli, 2022http://arxiv.org/abs/2210.040983. A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness — Zongxiong Chen, Jiahui Geng, Derui Zhu, Herbert Woisetschlaeger, Qing Li, Sonja Schimmler, Ruben Mayer, Chunming Rong, 2023http://arxiv.org/abs/2305.033554. MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources — Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald, Jens Lehmann, 2024http://arxiv.org/abs/2406.046705. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond — Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, Xia Hu, 2023http://arxiv.org/abs/2304.137126. GPU-accelerated Auxiliary-field quantum Monte Carlo with multi-Slater determinant trial states — Yifei Huang, Zhen Guo, Hung Q. Pham, Dingshun Lv, 2024http://arxiv.org/abs/2406.083147. Fast k Nearest Neighbor Search using GPU — Vincent Garcia, Eric Debreuve, Michel Barlaud, 2008https://scholar.google.com/scholar?q=Fast+k+Nearest+Neighbor+Search+using+GPU8. Billion-scale similarity search with GPUs — Jeff Johnson, Matthijs Douze, Herve Jegou, 2017https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs9. RTNN: Accelerating Neighbor Search Using Hardware Ray Tracing — Yuhao Zhu, 2022https://scholar.google.com/scholar?q=RTNN%3A+Accelerating+Neighbor+Search+Using+Hardware+Ray+Tracing10. An Improved Illumination Model for Shaded Display — Turner Whitted, 1980https://scholar.google.com/scholar?q=An+Improved+Illumination+Model+for+Shaded+Display11. The Rendering Equation — James T. Kajiya, 1986https://scholar.google.com/scholar?q=The+Rendering+Equation12. OptiX: A General Purpose Ray Tracing Engine — Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, and others, 2010https://scholar.google.com/scholar?q=OptiX%3A+A+General+Purpose+Ray+Tracing+Engine13. Ray Tracing Cores for General-Purpose Computing: A Literature Review — Enzo Meneses, Cristobal A. Navarro, Hector Ferrada, Konstantin Verichev, Cristian Salazar-Concha, 2026https://scholar.google.com/scholar?q=Ray+Tracing+Cores+for+General-Purpose+Computing%3A+A+Literature+Review14. A Survey of General-Purpose Computation on Graphics Hardware — John D. Owens, David Luebke, Naga Govindaraju, Mark Harris, Jens Kruger, Aaron Lefohn, Tim Purcell, 2007https://scholar.google.com/scholar?q=A+Survey+of+General-Purpose+Computation+on+Graphics+Hardware15. Scalable Parallel Programming with CUDA — John Nickolls, Ian Buck, Michael Garland, Kevin Skadron, 2008https://scholar.google.com/scholar?q=Scalable+Parallel+Programming+with+CUDA16. Gunrock: A High-Performance Graph Processing Library on the GPU — Yangzihao Wang, Andrew Davidson, Yuechao Pan, Yuduo Wu, Andy Riffel, John D. Owens, 2015https://scholar.google.com/scholar?q=Gunrock%3A+A+High-Performance+Graph+Processing+Library+on+the+GPU17. Dissecting the NVidia Turing T4 GPU via Microbenchmarking — Zhe Jia, Marco Maggioni, Jeffrey Smith, Daniele Paolo Scarpazza, 2019https://scholar.google.com/scholar?q=Dissecting+the+NVidia+Turing+T4+GPU+via+Microbenchmarking18. Ray Tracing Deformable Scenes using Dynamic Bounding Volume Hierarchies — Ingo Wald, Solomon Boulos, Peter Shirley, 2007https://scholar.google.com/scholar?q=Ray+Tracing+Deformable+Scenes+using+Dynamic+Bounding+Volume+Hierarchies19. Maximizing Parallelism in the Construction of BVHs, Octrees, and k-d Trees — Tero Karras, 2012https://scholar.google.com/scholar?q=Maximizing+Parallelism+in+the+Construction+of+BVHs%2C+Octrees%2C+and+k-d+Trees20. Quantized bounding volume hierarchies for neighbor search in molecular simulations on graphics processing units — Michael P. Howard, Antonia Statt, Felix Madutsa, Thomas M. Truskett, Athanassios Z. Panagiotopoulos, 2019https://scholar.google.com/scholar?q=Quantized+bounding+volume+hierarchies+for+neighbor+search+in+molecular+simulations+on+graphics+processing+units21. Fast Radius Search Exploiting Ray Tracing Frameworks — I. Evangelou, G. Papaioannou, K. Vardis, A. A. Vasilakis, 2021https://scholar.google.com/scholar?q=Fast+Radius+Search+Exploiting+Ray+Tracing+Frameworks22. RTX Beyond Ray Tracing: Exploring the Use of Hardware Ray Tracing Cores for Tet-Mesh Point Location — Ingo Wald, Will Usher, Nathan Morrical, Laura Lediaev, Valerio Pascucci, 2019https://scholar.google.com/scholar?q=RTX+Beyond+Ray+Tracing%3A+Exploring+the+Use+of+Hardware+Ray+Tracing+Cores+for+Tet-Mesh+Point+Location23. GPU-Accelerated Nearest Neighbor Search for 3D Registration — Deyuan Qiu, Stefan May, Andreas Nuchter, 2009https://scholar.google.com/scholar?q=GPU-Accelerated+Nearest+Neighbor+Search+for+3D+Registration24. Hardware-Accelerated Ray Tracing for Discrete and Continuous Collision Detection on GPUs — Sizhe Sui, Luis Sentis, Andrew Bylard, 2024https://scholar.google.com/scholar?q=Hardware-Accelerated+Ray+Tracing+for+Discrete+and+Continuous+Collision+Detection+on+GPUs25. RT-HDIST: Ray-Tracing Core-based Hausdorff Distance Computation — YoungWoo Kim, Jaehong Lee, Duksu Kim, 2025https://scholar.google.com/scholar?q=RT-HDIST%3A+Ray-Tracing+Core-based+Hausdorff+Distance+Computation26. JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping — Zihan Liu et al., 2023https://scholar.google.com/scholar?q=JUNO%3A+Optimizing+High-Dimensional+Approximate+Nearest+Neighbour+Search+with+Sparsity-Aware+Algorithm+and+Ray-Tracing+Core+Mapping27. CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs — Hiroyuki Ootomo et al., 2023https://scholar.google.com/scholar?q=CAGRA%3A+Highly+Parallel+Graph+Construction+and+Approximate+Nearest+Neighbor+Search+for+GPUs28. BANG: Billion-Scale Approximate Nearest Neighbor Search using a Single GPU — Karthik V. et al., 2024https://scholar.google.com/scholar?q=BANG%3A+Billion-Scale+Approximate+Nearest+Neighbor+Search+using+a+Single+GPU29. FusionANNS: An Efficient CPU/GPU Cooperative Processing Architecture for Billion-Scale Approximate Nearest Neighbor Search — Bing Tian et al., 2024https://scholar.google.com/scholar?q=FusionANNS%3A+An+Efficient+CPU%2FGPU+Cooperative+Processing+Architecture+for+Billion-Scale+Approximate+Nearest+Neighbor+Search30. AI Post Transformers: GPU-Accelerated Dynamic Quantized ANNS Graph Search — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-12-gpu-accelerated-dynamic-quantized-anns-g-f2cd4e.mp331. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp313 min
- Learning Facts at Scale with Active ReadingThis episode explores Active Reading, a training method that tries to move facts from documents into a model’s weights so it can answer closed-book questions without retrieval. It explains how the approach generates document-specific study materials such as paraphrases, active-recall prompts, timelines, analogies, and associations, and argues that this pedagogical synthetic data works better than simply rereading raw text or producing generic QA pairs. The discussion highlights reported gains from about 16% to 66% on a Wikipedia-based factual recall benchmark and strong relative improvement on finance documents, along with the larger WikiExpert-8B result that reportedly beats bigger models on factual QA after training on a trillion synthetic tokens. It also digs into the paper’s main weaknesses, including missing equal-compute baselines and possible benchmark coupling, which makes the episode interesting for listeners who want both the promise and the limits of using training curricula, rather than new architectures, to improve factual memory.Sources:1. Learning Facts at Scale with Active Reading — Jessy Lin, Vincent-Pierre Berges, Xilun Chen, Wen-Tau Yih, Gargi Ghosh, Barlas Oğuz, 2025http://arxiv.org/abs/2508.094942. Training Question Answering Models From Synthetic Data — Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, 2020https://scholar.google.com/scholar?q=Training+Question+Answering+Models+From+Synthetic+Data3. Self-Instruct: Aligning Language Models with Self-Generated Instructions — Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, Hannaneh Hajishirzi, 2022https://scholar.google.com/scholar?q=Self-Instruct%3A+Aligning+Language+Models+with+Self-Generated+Instructions4. Textbooks Are All You Need — Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sébastien Bubeck, Ronen Eldan, Yuanzhi Li, et al., 2023https://scholar.google.com/scholar?q=Textbooks+Are+All+You+Need5. Learning Facts at Scale with Active Reading — Jessy Lin, Vincent-Pierre Berges, Xilun Chen, Wen-Tau Yih, Gargi Ghosh, Barlas Oğuz, 2025https://scholar.google.com/scholar?q=Learning+Facts+at+Scale+with+Active+Reading6. How Much Knowledge Can You Pack Into the Parameters of a Language Model? — Adam Roberts, Colin Raffel, Noam Shazeer, 2020https://scholar.google.com/scholar?q=How+Much+Knowledge+Can+You+Pack+Into+the+Parameters+of+a+Language+Model%3F7. Large Language Models Struggle to Learn Long-Tail Knowledge — Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, Colin Raffel, 2022https://scholar.google.com/scholar?q=Large+Language+Models+Struggle+to+Learn+Long-Tail+Knowledge8. Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? — Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, Jonathan Herzig, 2024https://scholar.google.com/scholar?q=Does+Fine-Tuning+LLMs+on+New+Knowledge+Encourage+Hallucinations%3F9. Measuring short-form factuality in large language models — Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus, 2024https://scholar.google.com/scholar?q=Measuring+short-form+factuality+in+large+language+models10. Synthetic Continued Pretraining — Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candes, Tatsunori Hashimoto, 2024https://scholar.google.com/scholar?q=Synthetic+Continued+Pretraining11. Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha, 2023https://scholar.google.com/scholar?q=Fine-Tuning+or+Retrieval%3F+Comparing+Knowledge+Injection+in+LLMs12. How New Data Permeates LLM Knowledge and How to Dilute It — Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, Ulrich Rueckert, Been Kim, Mark Sandler, 2025https://scholar.google.com/scholar?q=How+New+Data+Permeates+LLM+Knowledge+and+How+to+Dilute+It13. Memory Layers at Scale — Vincent-Pierre Berges, Barlas Oguz, Daniel Haziza, Wen-Tau Yih, Luke Zettlemoyer, Gargi Ghosh, 2024https://scholar.google.com/scholar?q=Memory+Layers+at+Scale14. Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification — Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, Julia Kempe, 2024https://scholar.google.com/scholar?q=Beyond+Model+Collapse%3A+Scaling+Up+with+Synthesized+Data+Requires+Verification15. Strong Model Collapse — Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, Julia Kempe, 2024https://scholar.google.com/scholar?q=Strong+Model+Collapse16. Retrieval meets Long Context Large Language Models — Peng Xu et al., 2023https://scholar.google.com/scholar?q=Retrieval+meets+Long+Context+Large+Language+Models17. Expect the Unexpected: FailSafe Long Context QA for Finance — Kiran Kamble et al., 2025https://scholar.google.com/scholar?q=Expect+the+Unexpected%3A+FailSafe+Long+Context+QA+for+Finance18. A Parametric Memory Head for Continual Generative Retrieval — Kidist Amde Mekonnen, Yubao Tang, Maarten de Rijke, 2026https://scholar.google.com/scholar?q=A+Parametric+Memory+Head+for+Continual+Generative+Retrieval19. AI Post Transformers: Self-Improving Pretraining With Post-Trained Models — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-02-self-improving-pretraining-with-post-tra-e37460.mp320. AI Post Transformers: Training Modular KV Caches at Scale — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-15-training-modular-kv-caches-at-scale-382577.mp321. AI Post Transformers: Experimental Comparison of Agentic and Enhanced RAG — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-experimental-comparison-of-agentic-and-e-37d8bc.mp3Interactive Visualization: Learning Facts at Scale with Active Reading0 min
- HELM: Holistic Evaluation of Language ModelsThis episode explores the HELM framework for evaluating language models, arguing that once models become general-purpose infrastructure, single-dataset accuracy benchmarks are too narrow to capture their real-world behavior. It explains how HELM organizes evaluation across 30 models, 16 core scenarios, and seven metric families, measuring not just accuracy but also calibration, robustness, fairness, bias, toxicity, and efficiency under standardized conditions. The discussion highlights why HELM’s scenario-by-metric grid and targeted side studies on issues like reasoning, memorization, copyright, and disinformation matter: they make gaps in measurement visible instead of hiding them behind a single leaderboard score. A listener would find it interesting because it shows how benchmark design reflects values, and why model rankings can be misleading if they ignore confidence, harm, and cost.Sources:1. HELM: Holistic Evaluation of Language Modelshttps://arxiv.org/pdf/2211.091102. Equality of Opportunity in Supervised Learning — Moritz Hardt, Eric Price, Nathan Srebro, 2016https://arxiv.org/abs/1610.024133. Language (Technology) is Power: A Critical Survey of "Bias" in NLP — Su Lin Blodgett, Solon Barocas, Hal Daume III, Hanna Wallach, 2020https://arxiv.org/abs/2005.140504. StereoSet: Measuring stereotypical bias in pretrained language models — Moin Nadeem, Anna Bethke, Siva Reddy, 2020https://arxiv.org/abs/2004.094565. BBQ: A Hand-Built Bias Benchmark for Question Answering — Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, Samuel R. Bowman, 2021https://arxiv.org/abs/2110.081936. Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification — Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, Lucy Vasserman, 2019https://arxiv.org/abs/1903.045617. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models — Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, Noah A. Smith, 2020https://arxiv.org/abs/2009.114628. Challenges in Detoxifying Language Models — Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, Po-Sen Huang, 2021https://arxiv.org/abs/2109.074459. ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection — Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, Ece Kamar, 2022https://arxiv.org/abs/2203.0950910. On the Opportunities and Risks of Foundation Models — Rishi Bommasani et al., 2021https://scholar.google.com/scholar?q=On+the+Opportunities+and+Risks+of+Foundation+Models11. The EleutherAI Language Model Evaluation Harness — Leo Gao et al., 2021https://scholar.google.com/scholar?q=The+EleutherAI+Language+Model+Evaluation+Harness12. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models — Aarohi Srivastava et al., 2022https://scholar.google.com/scholar?q=Beyond+the+Imitation+Game%3A+Quantifying+and+Extrapolating+the+Capabilities+of+Language+Models13. Dynabench: Rethinking Benchmarking in NLP — Douwe Kiela et al., 2021https://scholar.google.com/scholar?q=Dynabench%3A+Rethinking+Benchmarking+in+NLP14. What Will it Take to Fix Benchmarking in Natural Language Understanding? — Samuel R. Bowman, George Dahl, 2021https://scholar.google.com/scholar?q=What+Will+it+Take+to+Fix+Benchmarking+in+Natural+Language+Understanding%3F15. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples — Shuo Yang et al., 2023https://scholar.google.com/scholar?q=Rethinking+Benchmark+and+Contamination+for+Language+Models+with+Rephrased+Samples16. Investigating Data Contamination in Modern Benchmarks for Large Language Models — Chunyuan Deng et al., 2023https://scholar.google.com/scholar?q=Investigating+Data+Contamination+in+Modern+Benchmarks+for+Large+Language+Models17. Benchmark Data Contamination of Large Language Models: A Survey — Cheng Xu et al., 2024https://scholar.google.com/scholar?q=Benchmark+Data+Contamination+of+Large+Language+Models%3A+A+Survey18. Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead — Vidhisha Balachandran et al., 2025https://scholar.google.com/scholar?q=Inference-Time+Scaling+for+Complex+Tasks%3A+Where+We+Stand+and+What+Lies+Ahead19. WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models — Kangyun Ning et al., 2024https://scholar.google.com/scholar?q=WTU-EVAL%3A+A+Whether-or-Not+Tool+Usage+Evaluation+Benchmark+for+Large+Language+Models20. T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step — Zehui Chen et al., 2023https://scholar.google.com/scholar?q=T-Eval%3A+Evaluating+the+Tool+Utilization+Capability+of+Large+Language+Models+Step+by+Step21. AI Post Transformers: IMO-Bench for Robust Mathematical Reasoning — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-imo-bench-for-robust-mathematical-reason-143489.mp322. AI Post Transformers: Real Context Size and Context Rot — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-real-context-size-and-context-rot-56cbb4.mp323. AI Post Transformers: Qwen3Guard: Streaming Three-Way Safety Classification for LLMs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-16-qwen3guard-streaming-three-way-safety-cl-26b0ef.mp324. AI Post Transformers: Speculative Decoding in Real vLLM Serving — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-04-speculative-decoding-in-real-vllm-servin-6f4e2b.mp3Interactive Visualization: HELM: Holistic Evaluation of Language Models13 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.