Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE InferenceThis episode examines PagedMoE, a system for running a huge mixture-of-experts model (Qwen3.5-397B-A17B) on a single PC with expert weights stored on an SSD. Cache misses leave decoding stalled on storage nearly half the time, and giving each token a different bit-width per expert would normally make that worse. Treating every expert-and-precision pair as its own cache object drops the hit rate from 57% to 35%. The paper's fix is to store each expert as nested refinement units, so a resident low-precision copy can be topped up with only the missing bits, or reused with no I/O at all. The discussion separates this idea from the prior work it borrows from (Matryoshka Quantization, Any-Precision LLM, SliceMoE, HOBBIT). It also splits the reported 1.56x speedup into the part from requesting fewer bits and the part from sharing cached bits across precisions. Listeners get a clear walk through the binary-tree codebook construction, the quantization-error numbers, and the limits of the evidence, such as a single-layer headroom study and no head-to-head comparison with SliceMoE.Sources:1. PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference — Yutong Lin, Yingdi Shan, Shaoyuan Chen, Yuanyong Chen, Yongwei Wu, Mingxing Zhang, 2026http://arxiv.org/abs/2611.031732. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev, Denis Mazur, 2023https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading3. MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache — Leyang Xue, Yao Fu, Zhan Lu, Luo Mai, Mahesh Marina, 2024 (arXiv; later revisions)https://scholar.google.com/scholar?q=MoE-Infinity%3A+Efficient+MoE+Inference+on+Personal+Machines+with+Sparsity-Aware+Expert+Cache4. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices — Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu, 2023https://scholar.google.com/scholar?q=EdgeMoE%3A+Empowering+Sparse+Large+Language+Models+on+Mobile+Devices5. LLM in a Flash: Efficient Large Language Model Inference with Limited Memory — Keivan Alizadeh, Iman Mirzadeh, et al. (Apple), 2023https://scholar.google.com/scholar?q=LLM+in+a+Flash%3A+Efficient+Large+Language+Model+Inference+with+Limited+Memory6. HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference — Peng Tang, Jiacheng Liu, Xiaofeng Hou, et al., 2024https://scholar.google.com/scholar?q=HOBBIT%3A+A+Mixed+Precision+Expert+Offloading+System+for+Fast+MoE+Inference7. Mixture Compressor for Mixture-of-Experts LLMs Gains More (MC) — Wei Huang, Yue Liao, Jianhui Liu, et al., 2024https://scholar.google.com/scholar?q=Mixture+Compressor+for+Mixture-of-Experts+LLMs+Gains+More+%28MC%298. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers9. Microscaling Data Formats for Deep Learning (OCP MX specification) — Bita Darvish Rouhani et al. (Microsoft, AMD, Arm, Intel, Meta, NVIDIA, Qualcomm), 2023https://scholar.google.com/scholar?q=Microscaling+Data+Formats+for+Deep+Learning+%28OCP+MX+specification%2910. Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs — Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, Jae W. Lee, 2024https://scholar.google.com/scholar?q=Any-Precision+LLM%3A+Low-Cost+Deployment+of+Multiple%2C+Different-Sized+LLMs11. Matryoshka Quantization — Pranav Nair, Puranjay Datta, Jeff Dean, Prateek Jain, Aditya Kusupati, 2025https://scholar.google.com/scholar?q=Matryoshka+Quantization12. QTIP: Quantization with Trellises and Incoherence Processing — Albert Tseng, Qingyao Sun, David Hou, Christopher De Sa, 2024https://scholar.google.com/scholar?q=QTIP%3A+Quantization+with+Trellises+and+Incoherence+Processing13. Stripes: Bit-Serial Deep Neural Network Computing — Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor Aamodt, Andreas Moshovos, 2016https://scholar.google.com/scholar?q=Stripes%3A+Bit-Serial+Deep+Neural+Network+Computing14. SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference — Yuseon Choi, Sangjin Kim, Jungjun Oh, Gwangtae Park, Byeongcheol Kim, Hoi-Jun Yoo, 2026https://scholar.google.com/scholar?q=SliceMoE%3A+Bit-Sliced+Expert+Caching+under+Miss-Rate+Constraints+for+Efficient+MoE+Inference15. D²MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving — Haodong Wang, Qihua Zhou, Zicong Hong, Song Guo, 2025https://scholar.google.com/scholar?q=D%C2%B2MoE%3A+Dual+Routing+and+Dynamic+Scheduling+for+Efficient+On-Device+MoE-based+LLM+Serving16. DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge — Yuegui Huang, Zhiyuan Fang, Weiqi Luo, Ruoyu Wu, Wuhui Chen, Zibin Zheng, 2026https://scholar.google.com/scholar?q=DyMoE%3A+Dynamic+Expert+Orchestration+with+Mixed-Precision+Quantization+for+Efficient+MoE+Inference+on+Edge17. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design — Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, Dahua Lin, 2025https://scholar.google.com/scholar?q=MxMoE%3A+Mixed-precision+Quantization+for+MoE+with+Accuracy+and+Performance+Co-Design18. GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs — Jianing Deng, Song Wang, Dongwei Wang, Zijie Liu, Tianlong Chen, Huanrui Yang, Jingtong Hu, 2026https://scholar.google.com/scholar?q=GEMQ%3A+Global+Expert-Level+Mixed-Precision+Quantization+for+MoE+LLMs19. KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models — Hongtao Chen, Weiyu Xie, Boxin Zhang, et al., 2025https://scholar.google.com/scholar?q=KTransformers%3A+Unleashing+the+Full+Potential+of+CPU%2FGPU+Hybrid+Inference+for+MoE+Models20. T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, Mao Yang, 2025https://scholar.google.com/scholar?q=T-MAC%3A+CPU+Renaissance+via+Table+Lookup+for+Low-Bit+LLM+Deployment+on+Edge21. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng et al., 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU22. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention23. Speculative MoE / expert prefetching works (e.g., Pre-gated MoE, ProMoE, AdapMoE) — Various, 2024–2025https://scholar.google.com/scholar?q=Speculative+MoE+%2F+expert+prefetching+works+%28e.g.%2C+Pre-gated+MoE%2C+ProMoE%2C+AdapMoE%29Interactive Visualization: PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE Inference17 min
- Intelligence per Watt: Measuring Local AI Efficiency Against the CloudThis episode examines "Intelligence per Watt," a Stanford and Together AI paper that asks whether small local models on laptop-class hardware can take real query load off the cloud. It walks through the paper's definitions. "Local" means at most 20 billion active parameters per token, which includes mixture-of-experts models like gpt-oss-120B. The proposed metric is average task accuracy divided by average power draw, and the paper also reports an energy-per-query (joules) variant. The discussion points out that watts alone ignore latency, so a low-power device that runs slowly can still cost more energy per query. It also flags the measurement caveats: power is read from the accelerator only, not the wall; queries run at batch size 1; chat accuracy is judged against a single reference model's answers; and "best-of-local" coverage is an oracle router that couldn't be deployed. The study spans about a million single-turn queries, more than twenty models and eight accelerators, and the episode places it against earlier LLM energy work such as "From Words to Watts" and MLPerf Power. Listeners get a careful guide to what the metric does and doesn't capture before the headline efficiency numbers are discussed.Sources:1. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré, 2025http://arxiv.org/abs/2511.078852. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, et al. (Stanford, Together AI; senior authors incl. John Hennessy, Azalia Mirhoseini, Christopher Ré), 2025 (v7 revised Oct 2026)https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI3. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Arya Tschand, Vijay Janapa Reddi, et al. (MLCommons), 2024https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts+for+Sustainable+AI4. Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model — Alexandra Sasha Luccioni, Sylvain Viguier, Anne-Laure Ligozat, 2022https://scholar.google.com/scholar?q=Estimating+the+Carbon+Footprint+of+BLOOM%2C+a+176B+Parameter+Language+Model5. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, et al. (MIT Lincoln Laboratory), 2023https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference6. Small Language Models are the Future of Agentic AI — Peter Belcak, Greg Heinrich, Shizhe Diao, et al. (NVIDIA Research), 2025https://scholar.google.com/scholar?q=Small+Language+Models+are+the+Future+of+Agentic+AI7. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention8. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration — Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Song Han, et al., 2023 (MLSys 2024)https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+On-Device+LLM+Compression+and+Acceleration9. Energy Considerations of Large Language Model Inference and Efficiency Optimizations — Jared Fernandez, Clara Na, Vashisth Tiwari, Yonatan Bisk, Sasha Luccioni, Emma Strubell, 2025https://scholar.google.com/scholar?q=Energy+Considerations+of+Large+Language+Model+Inference+and+Efficiency+Optimizations10. Part-time Power Measurements: nvidia-smi's Lack of Attention — Zeyu Yang, Karel Adamek, Wesley Armour, 2023https://scholar.google.com/scholar?q=Part-time+Power+Measurements%3A+nvidia-smi%27s+Lack+of+Attention11. Energy Use of AI Inference: Efficiency Pathways and Test-Time Compute — Felipe Oviedo et al., 2025https://scholar.google.com/scholar?q=Energy+Use+of+AI+Inference%3A+Efficiency+Pathways+and+Test-Time+Compute12. Minions: Cost-Efficient Collaboration Between On-Device and Cloud Language Models — Avanika Narayan, Dan Biderman, Sabri Eyuboglu, Avner May, Scott Linderman, James Zou, Christopher Ré, 2025https://scholar.google.com/scholar?q=Minions%3A+Cost-Efficient+Collaboration+Between+On-Device+and+Cloud+Language+Models13. RouteLLM: Learning to Route LLMs with Preference Data — Isaac Ong et al., 2025https://scholar.google.com/scholar?q=RouteLLM%3A+Learning+to+Route+LLMs+with+Preference+Data14. PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU — Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen, 2024https://scholar.google.com/scholar?q=PowerInfer%3A+Fast+Large+Language+Model+Serving+with+a+Consumer-Grade+GPU15. Green AI — Roy Schwartz, Jesse Dodge, Noah A. Smith, Oren Etzioni, 2020https://scholar.google.com/scholar?q=Green+AI16. Implications of Historical Trends in the Electrical Efficiency of Computing — Jonathan Koomey, Stephen Berard, Marla Sanchez, Henry Wong, 2010https://scholar.google.com/scholar?q=Implications+of+Historical+Trends+in+the+Electrical+Efficiency+of+ComputingInteractive Visualization: Intelligence per Watt: Measuring Local AI Efficiency Against the Cloud18 min
- Joules-to-GDP: Economic Value Produced Per Unit of AI EnergyThis episode examines "Joules-to-GDP," a Stanford-led paper that asks how much economic value AI systems produce per joule of energy when doing real professional work. It lays out the setup: 220 public GDPval tasks across 44 occupations, expert completion time multiplied by Bureau of Labor Statistics median wages as the dollar value, and a small LLM judge deciding whether the AI's deliverable beats the expert's. The central metrics are Net Value per Joule and Time Saving per Joule. Both are measured against an expert-only baseline and a tougher expert-with-AI baseline. The discussion also covers how energy is metered across a whole batch and split among concurrent tasks, and how running a small model across eight GPUs can waste power. It flags a confound: one model ran on H200s rather than H100s, so differences between models may reflect hardware rather than capability. Listeners get a sceptical look at whether ranking AI by energy efficiency gives a different answer from ranking by capability, cost or speed.Sources:1. Joules-to-GDP: Economic Value Produced Per Unit of AI Energyhttps://arxiv.org/pdf/2610.308582. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, et al. (OpenAI), 2025https://scholar.google.com/scholar?q=GDPval%3A+Evaluating+AI+Model+Performance+on+Real-World+Economically+Valuable+Tasks3. The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — Jae-Won Chung, Jeff J. Ma, Ruofan Wu, Jiachen Liu, Oh Jun Kweon, Yuxuan Xia, Zhiyu Wu, Mosharaf Chowdhury, 2025https://scholar.google.com/scholar?q=The+ML.ENERGY+Benchmark%3A+Toward+Automated+Inference+Energy+Measurement+and+Optimization4. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, Vijay Gadepally, 2023https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference5. Energy and Policy Considerations for Deep Learning in NLP — Emma Strubell, Ananya Ganesh, Andrew McCallum, 2019https://scholar.google.com/scholar?q=Energy+and+Policy+Considerations+for+Deep+Learning+in+NLP6. The Simple Macroeconomics of AI — Daron Acemoglu, 2024https://scholar.google.com/scholar?q=The+Simple+Macroeconomics+of+AI7. Cost-of-Pass: An Economic Framework for Evaluating Language Models — Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou, 2026https://scholar.google.com/scholar?q=Cost-of-Pass%3A+An+Economic+Framework+for+Evaluating+Language+Models8. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, et al., 2025https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI9. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — Arya Tschand et al., 2024https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts10. Economic Evaluations of Language Models — Alexander Wan, Stephane Hatgis-Kessell, Tomás Aguirre, Percy Liang, Rishi Bommasani, 2026https://scholar.google.com/scholar?q=Economic+Evaluations+of+Language+Models11. Remote Labor Index: Measuring AI Automation of Remote Work — Mantas Mazeika et al., 2025https://scholar.google.com/scholar?q=Remote+Labor+Index%3A+Measuring+AI+Automation+of+Remote+Work12. Power Hungry Processing: Watts Driving the Cost of AI Deployment? — Alexandra Sasha Luccioni, Yacine Jernite, Emma Strubell, 2024https://scholar.google.com/scholar?q=Power+Hungry+Processing%3A+Watts+Driving+the+Cost+of+AI+Deployment%3F13. AI Agents That Matter — Sayash Kapoor et al., 2024https://scholar.google.com/scholar?q=AI+Agents+That+Matter14. Generative AI at Work — Erik Brynjolfsson, Danielle Li, Lindsey R. Raymond, 2025https://scholar.google.com/scholar?q=Generative+AI+at+Work15. Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms — Prabhu Vellaisamy et al., 2026https://scholar.google.com/scholar?q=Characterization+of+Request+and+Token+Energy+Costs+for+LLM+Inference+Workloads+on+GPU+PlatformsInteractive Visualization: Joules-to-GDP: Economic Value Produced Per Unit of AI Energy17 min
- ECHO: Lossless KV Cache Offloading for Sparse Attention LLMsThis episode examines ECHO, a system from Shanghai Jiao Tong University and collaborators (OSDI 2026) that offloads the KV cache to host memory for serving sparse-attention models like DeepSeek-V3.2. It explains why sparse attention cuts compute but not storage. Cache size still grows linearly with context, so GPU memory limits batch size and leaves the sparse kernels underused. It then covers why naive recall over PCIe is slow: the top-k token set is known only after the lightning indexer has scored every earlier token. Prior prefetchers such as InfiniGen and FreeKV guess which tokens to fetch, which is lossy. ECHO instead guesses only when to fetch and then guarantees a recall of any missed token. To make this work under CUDA graph execution, it keeps its cache-management state in fixed-size integer tensors on the GPU. The discussion also stresses that the predictability ECHO relies on comes from one trained indexer, not from attention in general. It ends by questioning whether "lossless" is only a design property or something the paper actually measures.Sources:1. ECHO: Lossless KV Cache Offloading for Sparse Attention LLMshttps://www.usenix.org/system/files/osdi26-liu-guangda.pdf2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention3. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management4. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, et al., 2024https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, et al., 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU6. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré, 2022https://scholar.google.com/scholar?q=FlashAttention%3A+Fast+and+Memory-Efficient+Exact+Attention+with+IO-Awareness7. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision8. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention9. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, et al. (DeepSeek-AI and collaborators), 2025https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention10. FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference — Liu et al., 2025https://scholar.google.com/scholar?q=FreeKV%3A+Boosting+KV+Cache+Retrieval+for+Efficient+LLM+Inference11. ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction — Chen et al., 2024https://scholar.google.com/scholar?q=ArkVale%3A+Efficient+Generative+LLM+Inference+with+Recallable+Key-Value+Eviction12. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving13. ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp — Chen et al., 2025https://scholar.google.com/scholar?q=ESS%3A+An+Offload-Centric+Latent-Cache+Management+Architecture+for+DeepSeek-V3.2-Exp14. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Tang et al., 2024https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference15. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (NSA) — Yuan et al. (DeepSeek-AI), 2025https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention+%28NSA%2916. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Cheng et al., 2025https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference17. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+InferenceInteractive Visualization: ECHO: Lossless KV Cache Offloading for Sparse Attention LLMs17 min
- LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM ServingThis episode examines LeanSparse, a serving system that cuts the GPU memory needed to hold many long-context conversations on models with native sparse attention, such as DeepSeek-V4-Flash. It explains why sparse attention reduces reads but not storage. Even with KV offloaded to host memory, the lightning indexer's keys must stay on the GPU, and at 512K tokens they take about 7.2 times the space of the hot KV buffer. It also covers why agent-style workloads, where context grows every turn, make this memory limit bite, and why prefill needs a large temporary workspace. That workspace forces a bad choice between reserving memory that sits idle and making new requests queue behind decoders, which hurts time to first token. The proposed fixes are to stream each layer's history through two alternating GPU buffers during prefill, and to search compact indexer keys during decode, fetching only a short list of originals to rank exactly. All of this runs on a shared instance, avoiding the duplicated weights of prefill-decode disaggregation. Listeners get a clear walkthrough of where memory goes in modern sparse-attention serving, plus the reasoning behind why the obvious fix of offloading the indexer keys fails.Sources:1. LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs — Zaifeng Pan, Chris Wu, Xinwei Qiang, Zhengding Hu, Zhongkai Yu, Yufei Ding, 2026http://arxiv.org/abs/2611.046712. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — Jingyang Yuan, Huazuo Gao, Damai Dai, et al. (DeepSeek-AI, Peking University), 2025https://scholar.google.com/scholar?q=Native+Sparse+Attention%3A+Hardware-Aligned+and+Natively+Trainable+Sparse+Attention3. DeepSeek-V3.2-Exp technical report (DeepSeek Sparse Attention, DSA) — DeepSeek-AI, 2025https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp+technical+report+%28DeepSeek+Sparse+Attention%2C+DSA%294. MoBA: Mixture of Block Attention for Long-Context LLMs — Enzhe Lu, Zhejun Jiang, Jingyuan Liu, et al. (Moonshot AI), 2025https://scholar.google.com/scholar?q=MoBA%3A+Mixture+of+Block+Attention+for+Long-Context+LLMs5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference6. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management — Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim, 2024https://scholar.google.com/scholar?q=InfiniGen%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Dynamic+KV+Cache+Management7. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference — Hanshi Sun, Li-Wen Chang, Wenlei Bao, et al., 2024https://scholar.google.com/scholar?q=ShadowKV%3A+KV+Cache+in+Shadows+for+High-Throughput+Long-Context+LLM+Inference8. RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval — Di Liu, Meng Chen, Baotong Lu, et al., 2024https://scholar.google.com/scholar?q=RetrievalAttention%3A+Accelerating+Long-Context+LLM+Inference+via+Vector+Retrieval9. Billion-scale similarity search with GPUs (FAISS) — Jeff Johnson, Matthijs Douze, Hervé Jégou, 2017https://scholar.google.com/scholar?q=Billion-scale+similarity+search+with+GPUs+%28FAISS%2910. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI, Tsinghua), 2024https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving11. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, et al., 2024https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving12. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, et al., 2024https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve13. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, et al., 2023https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU14. PQCache: Product Quantization-based KVCache for Long Context LLM Inference — Hailin Zhang et al., 2025https://scholar.google.com/scholar?q=PQCache%3A+Product+Quantization-based+KVCache+for+Long+Context+LLM+Inference15. DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (DeepSeek Sparse Attention) — DeepSeek-AI, 2025https://scholar.google.com/scholar?q=DeepSeek-V3.2%3A+Pushing+the+Frontier+of+Open+Large+Language+Models+%28DeepSeek+Sparse+Attention%2916. CachedAttention (AttentionStore): Cost-Efficient LLM Serving for Multi-Turn Conversations — Bin Gao et al., 2024https://scholar.google.com/scholar?q=CachedAttention+%28AttentionStore%29%3A+Cost-Efficient+LLM+Serving+for+Multi-Turn+Conversations17. SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving — Zhou et al., 2025https://scholar.google.com/scholar?q=SparseServe%3A+Unlocking+Parallelism+for+Dynamic+Sparse+Attention+in+Long-Context+LLM+Serving18. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference — Agrawal et al., 2024https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference19. Product Quantization for Nearest Neighbor Search — Hervé Jégou, Matthijs Douze, Cordelia Schmid, 2011https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+SearchInteractive Visualization: LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM Serving0 min
- How to Loop Your Transformer: Scaling, Specialization, Parallel InferenceThis episode examines how to loop a transformer's layers, meaning reuse the same weights several times per token, depending on whether model size, training compute, or decoding speed is the scarce resource. It compares Full-Stack, Middle-Stack, and Per-Layer looping. Full-Stack gives the best quality at fixed parameters, Middle-Stack gives the best return at fixed training FLOPs, and gains flatten or turn unstable past about 16 loops. The 32-loop evidence comes only from the 70M and 100M models. Small per-loop low-rank adapters add only 1 to 4 percent more parameters and help more as loop counts grow, but at comparable FLOPs, doubling true depth still reaches lower loss. The discussion then turns to the decoding-latency problem, where each loop adds a sequential pass. It covers the Parallel Loop Transformer, which uses the previous token's state to run loops in parallel, and why that shift compounds prediction errors and causes magnitude mismatches. It also covers a proposed fix that re-injects the current token's embedding through normalization and a gate. Listeners get a practical guide to matching a looping design to a real resource constraint, with caveats about how far the evidence scales.Sources:1. How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference — Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale, Mike Lewis, Maryam Fazel-Zarandi, Shashank Rajput, Sujan Kumar Gonugondla, 2026http://arxiv.org/abs/2610.40325v12. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser, 2018https://scholar.google.com/scholar?q=Universal+Transformers3. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, et al., 2025https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach4. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, Sashank Reddi, 2025https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts%3A+On+the+Power+of+Looped+Transformers5. Parallel Loop Transformer (PLT) — Wu et al. (full title and author list not verified; cited in the paper as Wu et al., 2025), 2025https://scholar.google.com/scholar?q=Parallel+Loop+Transformer+%28PLT%296. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, 2021https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models7. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA — Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, Tal Schuster, 2024/2025 (cited as Bae et al., 2025 in the paper; verify the exact venue year)https://scholar.google.com/scholar?q=Relaxed+Recursive+Transformers%3A+Effective+Parameter+Sharing+with+Layer-wise+LoRA8. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2023https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding9. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention10. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads — Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Tri Dao, 2024https://scholar.google.com/scholar?q=Medusa%3A+Simple+LLM+Inference+Acceleration+Framework+with+Multiple+Decoding+Heads11. Parallel Loop Transformer for Efficient Test-Time Computation Scaling — Bohong Wu, Mengzhao Chen, Xiang Luo, et al., 2025https://scholar.google.com/scholar?q=Parallel+Loop+Transformer+for+Efficient+Test-Time+Computation+Scaling12. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — Sangmin Bae, Yujin Kim, Reza Bayat, et al., 2025/2026https://scholar.google.com/scholar?q=Mixture-of-Recursions%3A+Learning+Dynamic+Recursive+Depths+for+Adaptive+Token-Level+Computation13. Parcae: Scaling Laws for Stable Looped Language Models — Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, Daniel Y. Fu, 2026https://scholar.google.com/scholar?q=Parcae%3A+Scaling+Laws+for+Stable+Looped+Language+Models14. How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models — Kristian Schwethelm, Daniel Rueckert, Georgios Kaissis, 2026https://scholar.google.com/scholar?q=How+Much+Is+One+Recurrence+Worth%3F+Iso-Depth+Scaling+Laws+for+Looped+Language+Models15. LoopCoder-v2: Only Loop Once for Efficient Test-Time Computation Scaling — Jian Yang, Shawn Guo, Wei Zhang, et al., 2026https://scholar.google.com/scholar?q=LoopCoder-v2%3A+Only+Loop+Once+for+Efficient+Test-Time+Computation+Scaling16. Hyperloop Transformers — Abbas Zeitoun, Lucas Torroba-Hennigen, Yoon Kim, 2026https://scholar.google.com/scholar?q=Hyperloop+Transformers17. T2MLR: Transformer with Temporal Middle-Layer Recurrence — Ziyang Cai, Xingyu Zhu, Yihe Dong, Yinghui He, Sanjeev Arora, 2026https://scholar.google.com/scholar?q=T2MLR%3A+Transformer+with+Temporal+Middle-Layer+Recurrence18. Scaling Latent Reasoning via Looped Language Models (Ouro) — Rui-Jie Zhu, Zixuan Wang, Kai Hua, et al., 2026https://scholar.google.com/scholar?q=Scaling+Latent+Reasoning+via+Looped+Language+Models+%28Ouro%2919. Staircase Attention for Recurrent Processing of Sequences — Da Ju, Stephen Roller, Sainbayar Sukhbaatar, Jason E. Weston, 2022https://scholar.google.com/scholar?q=Staircase+Attention+for+Recurrent+Processing+of+SequencesInteractive Visualization: How to Loop Your Transformer: Scaling, Specialization, Parallel Inference14 min
- Test-Time Training via Context Distillation: Learning What to RememberThis episode examines "Learning What to Remember: Test-Time Training via Context Distillation" (TTCD), which asks what a model should write into a fixed-size memory when it can't keep everything. It starts with why linear-attention and recurrent models like DeltaNet and Gated DeltaNet need lossy bounded state. It then traces the progression of test-time training from the delta rule through TTT layers, Titans, LaCT, and In-Place TTT, which reuses a transformer's MLP down-projection as fast weights. The paper's method runs the same frozen backbone twice, once with a long sliding window as teacher and once with a short window as student. It writes the residual between their MLP outputs into the fast weights through an outer-product update, so the information outlives the short window. The discussion stays skeptical. It asks whether the residual really captures remote history or just a modest window extension, whether the distillation loss actually shapes the writes, and why the closest next-token competitor, TTT-E2E, is left out of the comparisons.Sources:1. Learning What to Remember: Test-Time Training via Context Distillation — Zixuan Wang, Xingyu Dang, Rui-Jie Zhu, Zixin Wen, Hengyu Fu, Wenhao Chai, Jason D. Lee, 2026http://arxiv.org/abs/2608.016722. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, 2024 (ICML 2025 version also circulated)https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States3. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2024 (ICLR 2025)https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule4. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time5. A General Method for Test-Time Training / Learning by Distilling Context — Snell, Klein, Zhong (Learning by Distilling Context); Askell et al. (A General Language Assistant as a Laboratory for Alignment), 2022 (Snell et al.); 2021 (Askell et al.)https://scholar.google.com/scholar?q=A+General+Method+for+Test-Time+Training+%2F+Learning+by+Distilling+Context6. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning — Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas, 2024https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+Test-Time+Training+for+Few-Shot+Learning7. End-to-End Test-Time Training for Long Context (TTT-E2E) — Arnuv Tandon, Karan Dalal, Xinhao Li, et al., 2025https://scholar.google.com/scholar?q=End-to-End+Test-Time+Training+for+Long+Context+%28TTT-E2E%298. In-Place Test-Time Training — Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, Tianle Cai, 2026https://scholar.google.com/scholar?q=In-Place+Test-Time+Training9. Test-Time Training Done Right (LaCT) — Tianyuan Zhang, Sai Bi, Yicong Hong, et al., 2025https://scholar.google.com/scholar?q=Test-Time+Training+Done+Right+%28LaCT%2910. Learning by Distilling Context — Charlie Snell, Dan Klein, Ruiqi Zhong, 2022https://scholar.google.com/scholar?q=Learning+by+Distilling+Context11. Cartridges: Lightweight and general-purpose long context representations via self-study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, et al., 2025https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study12. Kimi Linear: An Expressive, Efficient Attention Architecture — Kimi Team, Yu Zhang, et al., 2025https://scholar.google.com/scholar?q=Kimi+Linear%3A+An+Expressive%2C+Efficient+Attention+Architecture13. Linear Transformers Are Secretly Fast Weight Programmers — Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber, 2021https://scholar.google.com/scholar?q=Linear+Transformers+Are+Secretly+Fast+Weight+Programmers14. Infini-attention: Leave No Context Behind — Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal, 2024https://scholar.google.com/scholar?q=Infini-attention%3A+Leave+No+Context+Behind15. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, et al., 2024https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F16. Mechanistic Design and Scaling of Hybrid Architectures (MAD) — Michael Poli, et al., 2024https://scholar.google.com/scholar?q=Mechanistic+Design+and+Scaling+of+Hybrid+Architectures+%28MAD%29Interactive Visualization: Test-Time Training via Context Distillation: Learning What to Remember17 min
- RaBitQ-SSD: Split Codes and Pipelined I/O for SSD Vector SearchThis episode examines RaBitQ-SSD, a system for approximate nearest neighbor search over vectors stored on SSD, which the paper says can serve ten billion vectors on one drive at over 3,000 queries per second. It starts with the basics: why billion-scale embeddings are kept on SSD instead of DRAM, how recall at k is measured, how IVF partitions vectors into inverted lists, and why graph indexes such as DiskANN and PipeANN pay a disk read for every uncached hop. It then covers how RaBitQ's one-bit-per-dimension codes give each candidate a distance estimate with a probabilistic lower bound. That bound lets the system skip individual disk reads, and the paper reports about 38 pages read per query against roughly 870 for SPANN on Wikipedia embeddings. The hosts weigh the headline claims against a simple throughput model: once the SSD saturates at its measured 1.63 million reads per second, throughput depends on pages read per query. They also explain how the Split-RaBitQ design keeps only a prefix of each binary code in memory, rescaled so the estimate stays unbiased. Listeners get a clear account of the memory and blocking-I/O problems that make a direct port to SSD impractical.Sources:1. RaBitQ-SSD: Split Codes and Pipelined I/O for SSD-Resident Vector Search — Yuexuan Xu, Jianyang Gao, Michael Norris, Alibek Zhakubayev, Pankaj Singh, Junjie Qi, Matthijs Douze, Cheng Long, 2026http://arxiv.org/abs/2610.026522. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search — Jianyang Gao, Cheng Long, 2024https://scholar.google.com/scholar?q=RaBitQ%3A+Quantizing+High-Dimensional+Vectors+with+a+Theoretical+Error+Bound+for+Approximate+Nearest+Neighbor+Search3. Practical and Asymptotically Optimal Quantization of High-Dimensional Vectors in Euclidean Space for Approximate Nearest Neighbor Search (Extended RaBitQ) — Jianyang Gao, Yutong Gou, Yuexuan Xu, Yongyi Yang, Cheng Long, Raymond Chi-Wing Wong, 2025https://scholar.google.com/scholar?q=Practical+and+Asymptotically+Optimal+Quantization+of+High-Dimensional+Vectors+in+Euclidean+Space+for+Approximate+Nearest+Neighbor+Search+%28Extended+RaBitQ%294. DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node — Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnaswamy, Rohan Kadekodi, 2019https://scholar.google.com/scholar?q=DiskANN%3A+Fast+Accurate+Billion-point+Nearest+Neighbor+Search+on+a+Single+Node5. SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search — Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, Jingdong Wang, 2021https://scholar.google.com/scholar?q=SPANN%3A+Highly-efficient+Billion-scale+Approximate+Nearest+Neighbor+Search6. PipeANN: Low-Latency and High-Throughput Billion-Scale Vector Search via Pipelined I/O and Compute (title as I recall it; verify) — Not confirmed from memory; verify against the citation in the paper (reference [22]), 2025https://scholar.google.com/scholar?q=PipeANN%3A+Low-Latency+and+High-Throughput+Billion-Scale+Vector+Search+via+Pipelined+I%2FO+and+Compute+%28title+as+I+recall+it%3B+verify%297. PipeANN: Low-Latency Billion-Scale Vector Search via Speculative I/O Pipelining — Hao Guo, Youyou Lu, 2025https://scholar.google.com/scholar?q=PipeANN%3A+Low-Latency+Billion-Scale+Vector+Search+via+Speculative+I%2FO+Pipelining8. Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment — Mengzhao Wang et al., 2024https://scholar.google.com/scholar?q=Starling%3A+An+I%2FO-Efficient+Disk-Resident+Graph+Index+Framework+for+High-Dimensional+Vector+Similarity+Search+on+Data+Segment9. Product Quantization for Nearest Neighbor Search / Quicker ADC (FastScan) — Hervé Jégou, Matthijs Douze, Cordelia Schmid; André et al., 2011 / 2015https://scholar.google.com/scholar?q=Product+Quantization+for+Nearest+Neighbor+Search+%2F+Quicker+ADC+%28FastScan%2910. HELMSMAN (learned list selection with an SPDK I/O path for IVF on SSD) — As cited in the paper, 2025–2026https://scholar.google.com/scholar?q=HELMSMAN+%28learned+list+selection+with+an+SPDK+I%2FO+path+for+IVF+on+SSD%2911. AlayaLaser: Efficient Index Layout and Search Strategy for Large-scale High-dimensional Vector Similarity Search — As cited in the paper (ref [10]), 2025https://scholar.google.com/scholar?q=AlayaLaser%3A+Efficient+Index+Layout+and+Search+Strategy+for+Large-scale+High-dimensional+Vector+Similarity+Search12. VeloANN and SkipDisk — As cited in the paper (refs [73], [52]), 2025–2026https://scholar.google.com/scholar?q=VeloANN+and+SkipDisk13. Accelerating Large-Scale Inference with Anisotropic Vector Quantization (ScaNN) — Ruiqi Guo et al., 2020https://scholar.google.com/scholar?q=Accelerating+Large-Scale+Inference+with+Anisotropic+Vector+Quantization+%28ScaNN%2914. Filtered-DiskANN / FreshDiskANN — Gollapudi et al.; Singh et al., 2023 / 2021https://scholar.google.com/scholar?q=Filtered-DiskANN+%2F+FreshDiskANNInteractive Visualization: RaBitQ-SSD: Split Codes and Pipelined I/O for SSD Vector Search16 min
- Investor's Last Exam: Benchmarking AI Equity Research Without Future LeakageThis episode examines a paper that builds a benchmark for testing whether AI agents can do equity investment research without seeing the future. It centers on a Tesla task whose search used a "before March 24, 2026" date filter. The filter still returned a July page reporting realized Q2 deliveries of 480,126, because pages can keep an old publication date after their content is updated. The benchmark's fix is version-level availability: every document revision is stored separately and is eligible only if it predates the decision time. The database holds about 3.4 million versions, 1.52 TB, and every tool call is checked against it. The discussion explains why finance lacks a pass/fail oracle like SWE-bench. It describes the expert rubrics and dependency graphs that grade the chain from evidence to assumptions to judgment. It also covers the three yardsticks of answer quality, forecast outcome and simulated return, which the paper says rank models differently. The hosts separate retrieval leakage from "parametric leakage," meaning outcomes already in a model's pretraining weights, and tie that to prior work on lookahead bias in finance. The episode closes by introducing the 86 buy-side-style tasks, the 11 agent systems and the shared six-tool setup.Sources:1. Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future? — Huangfei Jiang, Anning Hu, Kai Sheng, Yuchen Cai, Rui Ge, Xiangyu Kong, Jianghao Lin, Shuzi Chen, Jiyi Weng, Junchi Yan, 2026http://arxiv.org/abs/2610.383422. FinanceBench: A New Benchmark for Financial Question Answering — Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, Bertie Vidgen, 2023https://scholar.google.com/scholar?q=FinanceBench%3A+A+New+Benchmark+for+Financial+Question+Answering3. BloombergGPT: A Large Language Model for Finance — Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, Gideon Mann, 2023https://scholar.google.com/scholar?q=BloombergGPT%3A+A+Large+Language+Model+for+Finance4. TradingAgents: Multi-Agents LLM Financial Trading Framework — Yijia Xiao, Edward Sun, Di Luo, Wei Wang, 2024https://scholar.google.com/scholar?q=TradingAgents%3A+Multi-Agents+LLM+Financial+Trading+Framework5. Financial Statement Analysis with Large Language Models — Alex G. Kim, Maximilian Muhn, Valeri V. Nikolaev, 2024https://scholar.google.com/scholar?q=Financial+Statement+Analysis+with+Large+Language+Models6. Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis — Paul Glasserman, Caden Lin, 2023https://scholar.google.com/scholar?q=Assessing+Look-Ahead+Bias+in+Stock+Return+Predictions+Generated+By+GPT+Sentiment+Analysis7. Lookahead Bias in Pretrained Language Models — Suproteem Sarkar, Keyon Vafa, 2024https://scholar.google.com/scholar?q=Lookahead+Bias+in+Pretrained+Language+Models8. Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance — David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, Qiji Jim Zhu, 2014https://scholar.google.com/scholar?q=Pseudo-Mathematics+and+Financial+Charlatanism%3A+The+Effects+of+Backtest+Overfitting+on+Out-of-Sample+Performance9. Advances in Financial Machine Learning — Marcos López de Prado, 2018https://scholar.google.com/scholar?q=Advances+in+Financial+Machine+Learning10. HealthBench: Evaluating Large Language Models Towards Improved Human Health — Rahul K. Arora et al. (OpenAI), 2025https://scholar.google.com/scholar?q=HealthBench%3A+Evaluating+Large+Language+Models+Towards+Improved+Human+Health11. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al., 2023https://scholar.google.com/scholar?q=Judging+LLM-as-a-Judge+with+MT-Bench+and+Chatbot+Arena12. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains — Anisha Gunjal et al., 2025https://scholar.google.com/scholar?q=Rubrics+as+Rewards%3A+Reinforcement+Learning+Beyond+Verifiable+Domains13. Humanity's Last Exam — Long Phan et al. (Center for AI Safety and Scale AI), 2025https://scholar.google.com/scholar?q=Humanity%27s+Last+Exam14. Approaching Human-Level Forecasting with Language Models — Danny Halawi, Fred Zhang, Chen Yueh-Han, Jacob Steinhardt, 2024https://scholar.google.com/scholar?q=Approaching+Human-Level+Forecasting+with+Language+Models15. ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities — Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, Philip E. Tetlock, 2024https://scholar.google.com/scholar?q=ForecastBench%3A+A+Dynamic+Benchmark+of+AI+Forecasting+Capabilities16. Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models — Alejandro Lopez-Lira, Yuehua Tang, 2023https://scholar.google.com/scholar?q=Can+ChatGPT+Forecast+Stock+Price+Movements%3F+Return+Predictability+and+Large+Language+Models17. Chronologically Consistent Large Language Models — Songrun He, Linying Lv, Asaf Manela, Jimmy Wu, 2025https://scholar.google.com/scholar?q=Chronologically+Consistent+Large+Language+Models18. Look-Ahead-Bench: A standardized benchmark of look-ahead bias in point-in-time LLMs for finance — Mostapha Benhenda, 2026https://scholar.google.com/scholar?q=Look-Ahead-Bench%3A+A+standardized+benchmark+of+look-ahead+bias+in+point-in-time+LLMs+for+finance19. Pitfalls in Evaluating Language Model Forecasters — Daniel Paleka, Shashwat Goel, Jonas Geiping, Florian Tramèr, 2025https://scholar.google.com/scholar?q=Pitfalls+in+Evaluating+Language+Model+Forecasters20. Temporal leakage in search-engine date-filtered web retrieval: A retrospective forecasting case study — Ali El Lahib, Ying-Jieh Xia, Zehan Li, Yuxuan Wang, Xinyu Pi, 2026https://scholar.google.com/scholar?q=Temporal+leakage+in+search-engine+date-filtered+web+retrieval%3A+A+retrospective+forecasting+case+study21. Bench to the Future: A Pastcasting Benchmark for Forecasting Agents — FutureSearch et al., 2025https://scholar.google.com/scholar?q=Bench+to+the+Future%3A+A+Pastcasting+Benchmark+for+Forecasting+Agents22. LLM Evaluators Recognize and Favor Their Own Generations — Arjun Panickssery, Samuel R. Bowman, Shi Feng, 2024https://scholar.google.com/scholar?q=LLM+Evaluators+Recognize+and+Favor+Their+Own+Generations23. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks — Tejal Patwardhan et al., 2025https://scholar.google.com/scholar?q=GDPval%3A+Evaluating+AI+Model+Performance+on+Real-World+Economically+Valuable+Tasks24. Advances in Financial Machine Learning (purged and embargoed cross-validation) — Marcos López de Prado, 2018https://scholar.google.com/scholar?q=Advances+in+Financial+Machine+Learning+%28purged+and+embargoed+cross-validation%2925. The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality — David H. Bailey, Marcos López de Prado, 2014https://scholar.google.com/scholar?q=The+Deflated+Sharpe+Ratio%3A+Correcting+for+Selection+Bias%2C+Backtest+Overfitting+and+Non-Normality26. StockBench: Can LLM Agents Trade Stocks Profitably in Real-World Markets? — Yanxu Chen et al., 2025https://scholar.google.com/scholar?q=StockBench%3A+Can+LLM+Agents+Trade+Stocks+Profitably+in+Real-World+Markets%3FInteractive Visualization: Investor's Last Exam: Benchmarking AI Equity Research Without Future Leakage0 min
- Agent Memory Costs: Characterizing Write-Time Overhead in Long-Horizon WorkloadsThis episode examines a paper that profiles ten agent-memory systems as systems workloads and splits their cost into construction (the write path), retrieval and generation, rather than comparing them on accuracy alone. It covers why pasting full history into a long context fails: prefill cost grows quadratically, prefix caching breaks across sessions, and recall degrades for material in the middle of long inputs. It then explains how agent memory differs from RAG and walks through a seven-stage pipeline and four construction styles, from none to deterministic chunking, LLM-mediated fact extraction and agentic control. The ten systems, including BM25, GraphRAG, Mem0, Letta and MIRIX, are placed along a gradient of increasing LLM involvement at write time. The discussion also lays out the measurement setup (phase-tagged calls aligned with GPU telemetry on a single H100 with Qwen3 models, plus remote OpenAI models) and previews the argument that construction is the hidden bill. Listeners get a concrete view of where long-lived agents spend compute and energy, and why that matters for scheduling.Sources:1. Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads — Yasmine Omri, Ziyu Gan, Zachary Broveak, Robin Geens, Zexue He, Alex Pentland, Marian Verhelst, Tsachy Weissman, Thierry Tambe, 2026http://arxiv.org/abs/2606.064482. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems3. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory — Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, Deshraj Yadav, 2025https://scholar.google.com/scholar?q=Mem0%3A+Building+Production-Ready+AI+Agents+with+Scalable+Long-Term+Memory4. Evaluating Very Long-Term Conversational Memory of LLM Agents (LoCoMo) — Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang, 2024https://scholar.google.com/scholar?q=Evaluating+Very+Long-Term+Conversational+Memory+of+LLM+Agents+%28LoCoMo%295. Sleep-time Compute: Beyond Inference Scaling at Test-time — Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, Joseph E. Gonzalez, 2025https://scholar.google.com/scholar?q=Sleep-time+Compute%3A+Beyond+Inference+Scaling+at+Test-time6. Zep: A Temporal Knowledge Graph Architecture for Agent Memory — Preston Rasmussen, Pavel Paliychuk, Travis Beauvais, Jack Ryan, Daniel Chalef, 2025https://scholar.google.com/scholar?q=Zep%3A+A+Temporal+Knowledge+Graph+Architecture+for+Agent+Memory7. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng et al., 2025https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference8. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin et al., 2024https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving9. Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference with Chunked Prefills — Amey Agrawal et al., 2024https://scholar.google.com/scholar?q=Sarathi-Serve%3A+Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Chunked+Prefills10. Zep, LightMem, and other efficiency-oriented memory systems (e.g., LightMem: Lightweight and Efficient Memory-Augmented Generation) — Various, 2025https://scholar.google.com/scholar?q=Zep%2C+LightMem%2C+and+other+efficiency-oriented+memory+systems+%28e.g.%2C+LightMem%3A+Lightweight+and+Efficient+Memory-Augmented+Generation%2911. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu et al., 2025https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study12. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM) — Woosuk Kwon et al., 2023https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention+%28vLLM%2913. Large-scale cluster management at Google with Borg — Abhishek Verma et al., 2015https://scholar.google.com/scholar?q=Large-scale+cluster+management+at+Google+with+BorgInteractive Visualization: Agent Memory Costs: Characterizing Write-Time Overhead in Long-Horizon Workloads0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.