Sign up to save your podcasts
Or
Share AI Post Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
AI Post Transformers
By mcgrofTechnology
AI Post Transformers episodes
- KumoRFM for In-Context Relational LearningThis episode explores KumoRFM, a 2025 proposal for a foundation model that can perform in-context learning directly on relational databases, aiming to handle tasks like churn prediction, fraud detection, recommendation, and forecasting without training a separate model for each schema and label. It explains how the approach represents warehouse data as heterogeneous graphs of rows and foreign-key relationships, using attention over local relational neighborhoods instead of flattening everything into handcrafted feature tables. The discussion focuses on the paper’s strongest claim, zero-shot transfer, and carefully separates true inference-time generalization from easier settings like continued pretraining on the target database or later fine-tuning on the target task. Listeners would find it interesting because the episode gets precise about what this system could change in enterprise ML, while also surfacing the practical caveats around task specification, temporal leakage, infrastructure cost, and how literal the any database, any task promise really is.Sources:1. KumoRFM for In-Context Relational Learninghttps://kumo.ai/research/kumo_relational_foundation_model.pdf2. Modeling Relational Data with Graph Convolutional Networks — Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Max Welling, et al., 2017https://scholar.google.com/scholar?q=Modeling+Relational+Data+with+Graph+Convolutional+Networks3. Heterogeneous Graph Transformer — Ziniu Hu, Yuxiao Dong, Kuansan Wang, Yizhou Sun, 2020https://scholar.google.com/scholar?q=Heterogeneous+Graph+Transformer4. Relational Deep Learning: Graph Representation Learning on Relational Databases — Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Jure Leskovec, et al., 2023https://scholar.google.com/scholar?q=Relational+Deep+Learning%3A+Graph+Representation+Learning+on+Relational+Databases5. Relational Graph Transformer — Vijay Prakash Dwivedi, Sri Jaladi, Yangyi Shen, Federico Lopez, Matthias Fey, Jure Leskovec, et al., 2025 (ICLR 2026)https://scholar.google.com/scholar?q=Relational+Graph+Transformer6. One Model to Rule them All: Towards Zero-Shot Learning for Databases — Benjamin Hilprecht, Carsten Binnig, 2021https://scholar.google.com/scholar?q=One+Model+to+Rule+them+All%3A+Towards+Zero-Shot+Learning+for+Databases7. Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction — Benjamin Hilprecht, Carsten Binnig, 2022https://scholar.google.com/scholar?q=Zero-Shot+Cost+Models+for+Out-of-the-box+Learned+Cost+Prediction8. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second — Noah Hollmann, Samuel Muller, Katharina Eggensperger, Frank Hutter, 2022 (ICLR 2023)https://scholar.google.com/scholar?q=TabPFN%3A+A+Transformer+That+Solves+Small+Tabular+Classification+Problems+in+a+Second9. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data — Rishabh Ranjan, Valter Hudovernik, Mark Znidar, Carlos Guestrin, Jure Leskovec, et al., 2025 (ICLR 2026)https://scholar.google.com/scholar?q=Relational+Transformer%3A+Toward+Zero-Shot+Foundation+Models+for+Relational+Data10. Relational In-Context Learning via Synthetic Pre-training with Structural Prior — Yanbo Wang, Jiaxuan You, Chuan Shi, Muhan Zhang, 2026https://scholar.google.com/scholar?q=Relational+In-Context+Learning+via+Synthetic+Pre-training+with+Structural+Prior11. OpenRFM: Dissecting Relational In-Context Learning — Zhikai Chen et al., 2026https://scholar.google.com/scholar?q=OpenRFM%3A+Dissecting+Relational+In-Context+Learning12. KumoRFM-2: Scaling Foundation Models for Relational Learning — Valter Hudovernik et al., 2026https://scholar.google.com/scholar?q=KumoRFM-2%3A+Scaling+Foundation+Models+for+Relational+Learning13. Retrieval & Fine-Tuning for In-Context Tabular Models — Valentin Thomas et al., 2024https://scholar.google.com/scholar?q=Retrieval+%26+Fine-Tuning+for+In-Context+Tabular+Models14. Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models — Xumeng Wen et al., 2025https://scholar.google.com/scholar?q=Scalable+In-Context+Learning+on+Tabular+Data+via+Retrieval-Augmented+Large+Language+Models15. Exploring Fine-Tuning for Tabular Foundation Models — Aditya Tanna et al., 2026https://scholar.google.com/scholar?q=Exploring+Fine-Tuning+for+Tabular+Foundation+Models16. AI Post Transformers: Predictive Query Language for Relational Databases — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-08-predictive-query-language-for-relational-103e68.mp317. AI Post Transformers: KumoRFM-2 for Relational Learning at Scale — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-08-kumorfm-2-for-relational-learning-at-sca-13c996.mp318. AI Post Transformers: Why LightGBM Made Boosted Trees Fast — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-why-lightgbm-made-boosted-trees-fast-286a89.mp319. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp311 min
- Atlas: Test-Time Memory for Long ContextsThis episode explores Atlas, a 2025 paper on test-time memorization that asks whether a model with fixed recurrent memory can learn to update that memory during inference and rival Transformers on long-context recall and reasoning. It explains the core tradeoff between Transformer-style KV caches, which preserve near-exact token access at growing cost, and bounded recurrent memory, which must decide what to keep, compress, or forget. The discussion focuses on why earlier recurrent memory systems fell short, then breaks down Atlas's proposed fixes: evaluating memory updates against a window of recent tokens rather than only the newest token, using richer key representations, and learning stronger retention and optimizer-style write rules. Listeners get a clear view of why this matters for post-Transformer architectures, and why fixed-size memory remains both a promising direction and a stubborn bottleneck.Sources:1. ATLAS: Learning to Optimally Memorize the Context at Test Time — Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, Vahab Mirrokni, 2025http://arxiv.org/abs/2505.237352. Linear Transformers Are Secretly Fast Weight Programmers — Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber, 2021https://scholar.google.com/scholar?q=Linear+Transformers+Are+Secretly+Fast+Weight+Programmers3. Retentive Network: A Successor to Transformer for Large Language Models — Yutao Sun, Li Dong, Shaohan Huang, Furu Wei, et al., 2023https://scholar.google.com/scholar?q=Retentive+Network%3A+A+Successor+to+Transformer+for+Large+Language+Models4. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2025https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time5. It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization — Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni, 2025https://scholar.google.com/scholar?q=It%27s+All+Connected%3A+A+Journey+Through+Test-Time+Memorization%2C+Attentional+Bias%2C+Retention%2C+and+Online+Optimization6. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun et al., 2024https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States7. RNNs are not Transformers (Yet): The Key Bottleneck on In-Context Retrieval — Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, 2024https://scholar.google.com/scholar?q=RNNs+are+not+Transformers+%28Yet%29%3A+The+Key+Bottleneck+on+In-Context+Retrieval8. Test-time Regression: a Unifying Framework for Designing Sequence Models with Associative Memory — Ke Alexander Wang, Jiaxin Shi, Emily B. Fox, 2025https://scholar.google.com/scholar?q=Test-time+Regression%3A+a+Unifying+Framework+for+Designing+Sequence+Models+with+Associative+Memory9. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack — Yuri Kuratov et al., 2024https://scholar.google.com/scholar?q=BABILong%3A+Testing+the+Limits+of+LLMs+with+Long+Context+Reasoning-in-a-Haystack10. SCBench: A KV Cache-Centric Analysis of Long-Context Methods — Yucheng Li et al., 2024https://scholar.google.com/scholar?q=SCBench%3A+A+KV+Cache-Centric+Analysis+of+Long-Context+Methods11. Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks — Zheng Wang et al., 2024https://scholar.google.com/scholar?q=Model+Tells+You+Where+to+Merge%3A+Adaptive+KV+Cache+Merging+for+LLMs+on+Long-Context+Tasks12. Forgetting Transformer: Softmax Attention with a Forget Gate — Zhixuan Lin et al., 2025https://scholar.google.com/scholar?q=Forgetting+Transformer%3A+Softmax+Attention+with+a+Forget+Gate13. Test-Time Training Done Right — Tianyuan Zhang et al., 2025https://scholar.google.com/scholar?q=Test-Time+Training+Done+Right14. Associative Recurrent Memory Transformer — Ivan Rodkin et al., 2024https://scholar.google.com/scholar?q=Associative+Recurrent+Memory+Transformer15. AI Post Transformers: Titans: Learning to Memorize at Test Time — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-20-titans-learning-to-memorize-at-test-time-054662.mp316. AI Post Transformers: Gated Delta Networks for Long-Context Retrieval — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-17-gated-delta-networks-for-long-context-re-706d85.mp317. AI Post Transformers: Parallelizing DeltaNet Linear Transformers over Sequence Length — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-parallelizing-deltanet-linear-transforme-2d0377.mp318. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp319. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp320. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp3Interactive Visualization: Atlas: Test-Time Memory for Long Contexts11 min
- Learning at Test Time with Expressive RNN StatesThis episode explores the paper Learning to (Learn at Test Time): RNNs with Expressive Hidden States and its attempt to give recurrent models transformer-like long-context behavior without the quadratic cost of attention. It explains why standard RNN hidden states are a bottleneck, compares that limitation to transformers’ growing KV cache, and highlights a key empirical motivation: in the paper’s setup, Mamba’s token-level perplexity improvements flatten around 16k tokens while transformers keep improving deeper into a 32k context. The discussion focuses on the paper’s core idea of test-time training, where the hidden state is treated as a small inner model whose parameters are updated online with a self-supervised learning rule, rather than as a fixed vector summary. Listeners would find it interesting because it connects old fast-weights and dynamic-evaluation ideas to a new systems-level proposal for long-context efficiency, while also noting the open question of whether better perplexity truly translates into stronger retrieval and reasoning.Sources:1. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, 2024http://arxiv.org/abs/2407.046202. Using Fast Weights to Attend to the Recent Past — Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu, 2016https://arxiv.org/abs/1610.062583. Linear Transformers Are Secretly Fast Weight Programmers — Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber, 2021https://arxiv.org/abs/2102.111744. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023https://arxiv.org/abs/2312.007525. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Xiaolong Wang, Tatsunori Hashimoto, Carlos Guestrin, 2024https://arxiv.org/abs/2407.046206. Dynamic Evaluation of Transformer Language Models — Ben Krause, Emmanuel Kahembwe, Iain Murray, Steve Renals, 2019https://arxiv.org/abs/1904.083787. Effective Long-Context Scaling of Foundation Models — Wenhan Xiong et al., 2023https://arxiv.org/abs/2309.160398. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models — Soham De et al., 2024https://arxiv.org/abs/2402.194279. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — Tri Dao, Albert Gu, 2024https://arxiv.org/abs/2405.2106010. An Empirical Study of Mamba-based Language Models — Roger Waleffe et al., 2024https://arxiv.org/abs/2406.0788711. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim et al., 2025https://arxiv.org/abs/2505.2341612. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse — Jingbo Yang et al., 2025https://arxiv.org/abs/2502.1600213. ReMamba: Equip Mamba with Effective Long-Sequence Modeling — Danlong Yuan et al., 2024https://arxiv.org/abs/2408.1549614. LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement — Zhifan Ye et al., 2025https://arxiv.org/abs/2504.1605315. Fast-weight Product Key Memory — Tianyu Zhao, Llion Jones, 2026https://arxiv.org/abs/2601.0067116. Test-Time Learning for Large Language Models — Jinwu Hu et al., 2025https://arxiv.org/abs/2505.2063317. Test-Time Training on Nearest Neighbors for Large Language Models — Moritz Hardt, Yu Sun, 2023https://arxiv.org/abs/2305.1846618. AI Post Transformers: Titans: Learning to Memorize at Test Time — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-20-titans-learning-to-memorize-at-test-time-054662.mp319. AI Post Transformers: Gated Linear Attention for Efficient Long Sequences — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-18-gated-linear-attention-for-efficient-lon-c858ab.mp320. AI Post Transformers: Kimi Linear: Efficient Expressive Attention Architecture — Hal Turing & Dr. Ada Shannon, 2025https://podcast.do-not-panic.com/episodes/kimi-linear-efficient-expressive-attention-architecture/21. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp322. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp323. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp3Interactive Visualization: Learning at Test Time with Expressive RNN States13 min
- Do Transformers Need Three Projections?This episode explores whether transformers really need separate query, key, and value projections, treating the problem as weight tying inside attention rather than as a brand-new model design. It explains why KV-cache size and memory bandwidth are major bottlenecks for long-context, on-device decoding, then compares increasingly aggressive sharing schemes, especially the difference between tying keys and values versus tying queries and keys. The discussion emphasizes that the broader sweep happens at 300M parameters, while only the shared-K/V variant is carried to 1.2B scale and remains in contention against practical baselines like grouped-query and multi-query attention. Listeners get a concrete deployment tradeoff: shared K/V can reduce KV-cache memory by about 50 percent at roughly a 3.1 percent perplexity cost, making the episode especially interesting for anyone focused on efficient inference.Sources:1. Do Transformers Need Three Projections? Systematic Study of QKV Variants — Ali Kayyam, Anusha Madan Gopal, M Anthony Lewis, 2026http://arxiv.org/abs/2606.04032v22. Attention Is All You Need — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, 2017https://arxiv.org/abs/1706.037623. Fast Transformer Decoding: One Write-Head is All You Need — Noam Shazeer, 2019https://arxiv.org/abs/1911.021504. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai, 2023https://arxiv.org/abs/2305.132455. Do Transformers Need Three Projections? Systematic Study of QKV Variants — Ali Kayyam, Anusha Madan Gopal, M. Anthony Lewis, 2026https://arxiv.org/abs/2606.040326. Using the Output Embedding to Improve Language Models — Ofir Press, Lior Wolf, 2017https://arxiv.org/abs/1608.058597. Linformer: Self-Attention with Linear Complexity — Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma, 2020https://scholar.google.com/scholar?q=Linformer%3A+Self-Attention+with+Linear+Complexity8. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, Francois Fleuret, 2020https://scholar.google.com/scholar?q=Transformers+are+RNNs%3A+Fast+Autoregressive+Transformers+with+Linear+Attention9. Mamba: Linear-Time Sequence Modeling with Selective State Spaces — Albert Gu, Tri Dao, 2023https://scholar.google.com/scholar?q=Mamba%3A+Linear-Time+Sequence+Modeling+with+Selective+State+Spaces10. AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations — Qian Tao et al., 2024https://arxiv.org/abs/2410.1321211. LongHeads: Multi-Head Attention is Secretly a Long Context Processor — Yi Lu et al., 2024https://arxiv.org/abs/2402.1068512. MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads — Weihao Liu et al., 2025https://arxiv.org/abs/2502.1396313. Squeezed Attention: Accelerating Long Context Length LLM Inference — Coleman Hooper et al., 2024https://arxiv.org/abs/2411.0968814. Beyond Uniform Query Distribution: Key-Driven Grouped Query Attention — Zohaib Khan et al., 2024https://arxiv.org/abs/2408.0845415. Weight Decay Induces Low-Rank Attention Layers — Seijin Kobayashi et al., 2024https://arxiv.org/abs/2410.2381916. Dissecting Query-Key Interaction in Vision Transformers — Xu Pan et al., 2024https://arxiv.org/abs/2405.1488017. AI Post Transformers: Affordable Large-Scale Decoding Through Model-System Co-Design — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-19-affordable-large-scale-decoding-through-e1d7ed.mp318. AI Post Transformers: Memory-Bound, Not Bandwidth-Limited Batch-1 LLM Decode — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-06-02-memory-bound-not-bandwidth-limited-batch-114799.mp319. AI Post Transformers: Prefill-as-a-Service for Cross-Datacenter KV Cache — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-19-prefill-as-a-service-for-cross-datacente-7560be.mp320. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp321. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp3Interactive Visualization: Do Transformers Need Three Projections?0 min
- Robots Need More Than VLAs and World ModelsThis episode explores the position paper Robots Need More Than VLAs & World Models and its claim that the main bottleneck in robotics may be grounding: turning raw physical behavior into robot-usable signals such as actions, contacts, task phases, goals, and rewards. It explains why vision-language-action models, world models, and reward models play different roles, and why simply scaling policy transformers cannot recover supervision that was never captured in the data. The discussion also digs into cross-embodiment learning and task-preserving retargeting, focusing on how humans and different robots can share useful experience despite mismatched bodies, sensors, and action spaces. A standout example is EgoMimic, which uses egocentric human video, 3D hand tracking, cross-domain alignment, and joint human-robot training to improve long-horizon real-robot manipulation, giving listeners a concrete picture of what might actually unlock broader robot generalization.Sources:1. Robots Need More than VLA and World Models — Elis Karcini, Faisal Mehrban, Quang Nguyen, Mac Schwager, Arash Ajoudani, Cesar Cadena, Jan Peters, Marco Hutter, Haitham Bou-Ammar, 2026http://arxiv.org/abs/2606.065562. RT-1: Robotics Transformer for Real-World Control at Scale — Anthony Brohan, Noah Brown, Chelsea Finn, Sergey Levine, et al., 2022https://scholar.google.com/scholar?q=RT-1%3A+Robotics+Transformer+for+Real-World+Control+at+Scale3. RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control — Anthony Brohan, Noah Brown, Danny Driess, Karol Hausman, Chelsea Finn, Sergey Levine, et al., 2023https://scholar.google.com/scholar?q=RT-2%3A+Vision-Language-Action+Models+Transfer+Web+Knowledge+to+Robotic+Control4. OpenVLA: An Open-Source Vision-Language-Action Model — Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Chelsea Finn, Sergey Levine, et al., 2024https://scholar.google.com/scholar?q=OpenVLA%3A+An+Open-Source+Vision-Language-Action+Model5. π0: A Vision-Language-Action Flow Model for General Robot Control — Kevin Black, Noah Brown, Danny Driess, Karol Hausman, Sergey Levine, Chelsea Finn, et al., 2024https://scholar.google.com/scholar?q=%CF%800%3A+A+Vision-Language-Action+Flow+Model+for+General+Robot+Control6. Deep reinforcement learning from human preferences — Paul Christiano, Jan Leike, Tom B. Brown, Dario Amodei, et al., 2017https://scholar.google.com/scholar?q=Deep+reinforcement+learning+from+human+preferences7. Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videos — Annie S. Chen, Suraj Nair, Chelsea Finn, 2021https://scholar.google.com/scholar?q=Learning+Generalizable+Robotic+Reward+Functions+from+%22In-The-Wild%22+Human+Videos8. Language to Rewards for Robotic Skill Synthesis — Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Brian Ichter, Ted Xiao, Fei Xia, et al., 2023https://scholar.google.com/scholar?q=Language+to+Rewards+for+Robotic+Skill+Synthesis9. RoboReward: General-Purpose Vision-Language Reward Models for Robotics — Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, Chelsea Finn, 2026https://scholar.google.com/scholar?q=RoboReward%3A+General-Purpose+Vision-Language+Reward+Models+for+Robotics10. Open X-Embodiment: Robotic Learning Datasets and RT-X Models — Open X-Embodiment Collaboration; Abby O'Neill, Fei Xia, Chelsea Finn, Sergey Levine, et al., 2023https://scholar.google.com/scholar?q=Open+X-Embodiment%3A+Robotic+Learning+Datasets+and+RT-X+Models11. XSkill: Cross Embodiment Skill Discovery — Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, Shuran Song, 2023https://scholar.google.com/scholar?q=XSkill%3A+Cross+Embodiment+Skill+Discovery12. Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignment — Tianyu Wang, Dwait Bhatt, Xiaolong Wang, Nikolay Atanasov, 2024https://scholar.google.com/scholar?q=Cross-Embodiment+Robot+Manipulation+Skill+Transfer+using+Latent+Space+Alignment13. Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfer — Gemini Robotics Team; Abbas Abdolmaleki, Anthony Brohan, Keerthana Gopalakrishnan, Ted Xiao, et al., 2025https://scholar.google.com/scholar?q=Gemini+Robotics+1.5%3A+Pushing+the+Frontier+of+Generalist+Robots+with+Advanced+Embodied+Reasoning%2C+Thinking%2C+and+Motion+Transfer14. Learning Latent Plans from Play — Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Sergey Levine, Pierre Sermanet, et al., 2019https://scholar.google.com/scholar?q=Learning+Latent+Plans+from+Play15. R3M: A Universal Visual Representation for Robot Manipulation — Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, Abhinav Gupta, 2022https://scholar.google.com/scholar?q=R3M%3A+A+Universal+Visual+Representation+for+Robot+Manipulation16. Zero-Shot Robot Manipulation from Passive Human Videos — Homanga Bharadhwaj, Abhinav Gupta, Shubham Tulsiani, Vikash Kumar, 2023https://scholar.google.com/scholar?q=Zero-Shot+Robot+Manipulation+from+Passive+Human+Videos17. GenSim: Generating Robotic Simulation Tasks via Large Language Models — Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Huazhe Xu, Xiaolong Wang, et al., 2023https://scholar.google.com/scholar?q=GenSim%3A+Generating+Robotic+Simulation+Tasks+via+Large+Language+Models18. $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control — Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, et al., 2024https://scholar.google.com/scholar?q=%24%5Cpi_0%24%3A+A+Vision-Language-Action+Flow+Model+for+General+Robot+Control19. EgoMimic: Scaling Imitation Learning via Egocentric Video — Simar Kareer, Dhruv Patel, Ryan Punamiya, Pranay Mathur, Shuo Cheng, Chen Wang, Judy Hoffman, Danfei Xu, 2024https://scholar.google.com/scholar?q=EgoMimic%3A+Scaling+Imitation+Learning+via+Egocentric+Video20. LEGATO: Cross-Embodiment Imitation Using a Grasping Tool — Mingyo Seo, H. Andy Park, Shenli Yuan, Yuke Zhu, Luis Sentis, 2024https://scholar.google.com/scholar?q=LEGATO%3A+Cross-Embodiment+Imitation+Using+a+Grasping+Tool21. Rank2Reward: Learning Shaped Reward Functions from Passive Video — Daniel Yang, Davin Tjia, Jacob Berg, Dima Damen, Pulkit Agrawal, Abhishek Gupta, 2024https://scholar.google.com/scholar?q=Rank2Reward%3A+Learning+Shaped+Reward+Functions+from+Passive+Video22. Genie: Generative Interactive Environments — Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, et al., 2024https://scholar.google.com/scholar?q=Genie%3A+Generative+Interactive+Environments23. Neural Scaling Laws in Robotics — Sebastian Sartor, Neil Thompson, 2024https://arxiv.org/abs/2405.1400524. Diffusion-VLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression — Junjie Wen et al., 2024https://arxiv.org/abs/2412.0329325. Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviation — Ria Doshi et al., 2024https://arxiv.org/abs/2408.1181226. RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learning — Lawrence Yunliang Chen et al., 2024https://arxiv.org/abs/2409.0340327. CLAM: Continuous Latent Action Models for Robot Learning from Unlabeled Demonstrations — Anthony Liang et al., 2025https://arxiv.org/abs/2505.0499928. DayDreamer: World Models for Physical Robot Learning — Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, Pieter Abbeel, 2022https://arxiv.org/abs/2206.1417629. Ctrl-World: A Controllable Generative World Model for Robot Manipulation — Yanjiang Guo et al., 2025https://arxiv.org/abs/2510.1012530. AI Post Transformers: DreamerV3 World Models Across 150 Tasks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-20-dreamerv3-world-models-across-150-tasks-af5edb.mp331. AI Post Transformers: When LLM Judges Become Coin Flips — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-when-llm-judges-become-coin-flips-8b43ef.mp332. AI Post Transformers: SkillsBench for Evaluating Agent Skills — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-skillsbench-for-evaluating-agent-skills-58bb1e.mp3Interactive Visualization: Robots Need More Than VLAs and World Models14 min
- End-to-End Context Compression at ScaleThis episode explores End-to-End Context Compression at Scale, a paper on whether learned context compression can beat the cost of long-context inference in quality, time to first token, and peak memory. It explains the main design choices behind the authors’ Latent Context Language Models, which use a 0.6B encoder and 4B decoder to replace long token sequences with learned latent memory at compression ratios from 1:4 to 1:16, and contrasts that approach with full-context prompting, retrieval, summarization, and KV-cache compression methods such as SnapKV and KVzip. The discussion highlights the paper’s core result: on RULER and LongBench EN-16, the released system reportedly sets a new Pareto frontier, delivering up to 8.8x faster inference on RULER and 5.2x faster on LongBench with lower memory use and stronger accuracy at aggressive compression. It also digs into the catch that makes the result interesting for practitioners: this speedup depends on a heavily trained system and changes the serving stack, so the real question is not just whether the benchmark wins are real, but whether learned compression is finally practical infrastructure for long-horizon agents and large-scale deployment.Sources:1. End-to-End Context Compression at Scale — Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra, Zaiqian Chen, Artem Gazizov, Venkata Anoop Suhas Kumar Morisetty, Bhavya Kailkhura, Harshitha Menon, Zhuang Liu, Brian R. Bartoldson, Tom Goldstein, Sanae Lotfi, Micah Goldblum, Pavel Izmailov, 2026http://arxiv.org/abs/2606.096592. Learning to Compress Prompts with Gist Tokens — Jesse Mu, Xiang Lisa Li, Noah Goodman, 2023https://arxiv.org/abs/2304.084673. Adapting Language Models to Compress Contexts — Alexis Chevalier, Alexander Wettig, Anirudh Ajith, Danqi Chen, 2023https://arxiv.org/abs/2305.147884. Long-Context Language Modeling with Parallel Context Encoding — Howard Yen, Tianyu Gao, Danqi Chen, 2024https://arxiv.org/abs/2402.166175. ARC-Encoder: learning compressed text representations for large language models — Hippolyte Pilchen, Edouard Grave, Patrick Perez, 2025https://arxiv.org/abs/2510.205356. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen, 2024https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation7. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, Hyun Oh Song, 2025https://scholar.google.com/scholar?q=KVzip%3A+Query-Agnostic+KV+Cache+Compression+with+Context+Reconstruction8. Fast KV Compaction via Attention Matching — Adam Zweiger, Xinghong Fu, Han Guo, Yoon Kim, 2026https://scholar.google.com/scholar?q=Fast+KV+Compaction+via+Attention+Matching9. Cartridges: Lightweight and general-purpose long context representations via self-study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Re, 2025https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+general-purpose+long+context+representations+via+self-study10. Latent Context Compilation: Distilling Long Context into Compact Portable Memory — Zeju Li, Yizhou Zhou, Qiang Xu, 2026https://scholar.google.com/scholar?q=Latent+Context+Compilation%3A+Distilling+Long+Context+into+Compact+Portable+Memory11. RULER: What's the Real Context Size of Your Long-Context Language Models? — Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, 2024https://scholar.google.com/scholar?q=RULER%3A+What%27s+the+Real+Context+Size+of+Your+Long-Context+Language+Models%3F12. LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks — Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li, 2024https://scholar.google.com/scholar?q=LongBench+v2%3A+Towards+Deeper+Understanding+and+Reasoning+on+Realistic+Long-context+Multitasks13. ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse — Yu Zhu et al., 2026https://arxiv.org/abs/2605.2285014. PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference — Krishna Teja Chitty-Venkata et al., 2025https://arxiv.org/abs/2509.0437715. ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference — Xiang Liu et al., 2025https://arxiv.org/abs/2502.0029916. Long Context Compression with Activation Beacon — Peitian Zhang et al., 2024https://arxiv.org/abs/2401.0346217. AI Post Transformers: Explicit Information Transmission for Context Compression — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-05-explicit-information-transmission-for-co-24e3c2.mp318. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp319. AI Post Transformers: DeepSeek-V4 and Practical Million-Token Context — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-deepseek-v4-and-practical-million-token-6f4de1.mp320. AI Post Transformers: Memory Sparse Attention for 100M-Token Scaling — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-07-memory-sparse-attention-for-100m-token-s-377cff.mp321. AI Post Transformers: Long Context Pre-Training with Lighthouse Attention — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-13-long-context-pre-training-with-lighthous-e85bbe.mp322. AI Post Transformers: Compressed Convolutional Attention in Latent Space — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-compressed-convolutional-attention-in-la-61e1cf.mp310 min
- Unembedding Matrices as Feature Lenses for EmbeddingsThis episode explores why decoder-style language models can generate fluent text yet still underperform dedicated embedding models when asked for zero-shot sentence vectors, despite embeddings being critical infrastructure for search, retrieval-augmented generation, clustering, and recommendation. It examines the paper’s main argument that the problem is not just bad pooling or prompting, but a deeper geometric bias: sentence representations appear overly aligned with frequent, low-information tokens, which becomes visible when they are projected through the model’s unembedding matrix. It also digs into the debate over whether decoder models mainly suffer from poor extraction recipes or from genuinely weaker embedding spaces, using concrete details from the authors’ code such as prompt-based summarization, last-token pooling, and custom truncation. A listener would find it interesting because the discussion connects mechanistic interpretability to real embedding-system design, and even suggests why filtering or reducing dimensions could improve both quality and efficiency.Sources:1. Your UnEmbedding Matrix is Secretly a Feature Lens for Text Embeddings — Songhao Wu, Zhongxin Chen, Yuxuan Liu, Heng Cui, Cong Li, Rui Yan, 2026http://arxiv.org/abs/2606.075022. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Nils Reimers, Iryna Gurevych, 2019https://arxiv.org/abs/1908.100843. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, Iryna Gurevych, 2021https://arxiv.org/abs/2104.086634. MTEB: Massive Text Embedding Benchmark — Niklas Muennighoff, Nouamane Tazi, Loic Magne, Nils Reimers, 2022https://arxiv.org/abs/2210.073165. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders — Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, Siva Reddy, 2024https://arxiv.org/abs/2404.059616. Indexing by Latent Semantic Analysis — Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, Richard Harshman, 1990https://www.cs.csustan.edu/~mmartin/LDS/Deerwester-et-al.pdf7. All-but-the-Top: Simple and Effective Postprocessing for Word Representations — Jiaqi Mu, Suma Bhat, Pramod Viswanath, 2018https://arxiv.org/abs/1702.014178. Whitening Sentence Representations for Better Semantics and Faster Retrieval — Jianlin Su, Jiarun Cao, Weijie Liu, Yangyiwen Ou, 2021https://arxiv.org/abs/2103.153169. Matryoshka Representation Learning — Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, Ali Farhadi, 2022https://arxiv.org/abs/2205.1314710. Scaling Sentence Embeddings with Large Language Models — Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, Fuzhen Zhuang, 2023https://scholar.google.com/scholar?q=Scaling+Sentence+Embeddings+with+Large+Language+Models11. Improving Text Embeddings with Large Language Models — Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei, 2024https://scholar.google.com/scholar?q=Improving+Text+Embeddings+with+Large+Language+Models12. Eliciting Latent Predictions from Transformers with the Tuned Lens — Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, Jacob Steinhardt, 2023https://scholar.google.com/scholar?q=Eliciting+Latent+Predictions+from+Transformers+with+the+Tuned+Lens13. Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings — Qian Chen, Wen Wang, Qinglin Zhang, Siqi Zheng, Chong Deng, Hai Yu, Jiaqing Liu, Yukun Ma, Chong Zhang, 2023https://scholar.google.com/scholar?q=Ditto%3A+A+Simple+and+Efficient+Approach+to+Improve+Sentence+Embeddings14. Is anisotropy really the cause of BERT embeddings not being semantic? — Alejandro Fuster Baggetto, Victor Fresno, 2022https://scholar.google.com/scholar?q=Is+anisotropy+really+the+cause+of+BERT+embeddings+not+being+semantic%3F15. Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing — Richard Diehl Martinez et al., 2024https://arxiv.org/abs/2410.1146216. Anisotropy Is Inherent to Self-Attention in Transformers — Nathan Godey, Eric de la Clergerie, Benoit Sagot, 2024https://arxiv.org/abs/2401.1214317. Indic-TunedLens: Interpreting Multilingual Models in Indian Languages — Mihir Panchal et al., 2026https://arxiv.org/abs/2602.1503818. KV-Embedding: Training-free Text Embedding via Internal KV Re-routing in Decoder-only LLMs — Yixuan Tang, Yi Yang, 2026https://arxiv.org/abs/2601.0104619. Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Token — Ailiang Lin et al., 2025https://arxiv.org/abs/2507.2338620. On the Theoretical Limitations of Embedding-Based Retrieval — Orion Weller et al., 2025https://arxiv.org/abs/2508.2103821. AI Post Transformers: Why Transformers Fail at Counting — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-08-why-transformers-fail-at-counting-137924.mp322. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp323. AI Post Transformers: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-03-25-turboquant-online-vector-quantiz-1967b7.mp3Interactive Visualization: Unembedding Matrices as Feature Lenses for Embeddings15 min
- Predictive Query Language for Relational DatabasesThis episode explores Predictive Query Language (PQL), a SQL-shaped domain-specific language for defining supervised learning tasks directly over relational databases by specifying the prediction target, entity, and future time horizon in one declarative statement. It explains why training label generation is often the real bottleneck in applied machine learning, unpacking concepts like prediction entities, relational learning, anchor times, point-in-time consistency, and information leakage. The discussion compares PQL to earlier work on prediction engineering and relational deep learning, arguing that its main contribution is not a new model but a more disciplined way to construct temporally valid prediction problems from messy, multi-table operational data. Listeners interested in real-world ML systems will find it interesting because it focuses on the part most papers skip: how to ask the predictive question correctly when database history is incomplete, revised, and easy to misuse.Sources:1. Predictive Query Language for Relational Databaseshttps://arxiv.org/pdf/2602.095722. Declarative Machine Learning - A Classification of Basic Properties and Types — Matthias Boehm, Alexandre V. Evfimievski, Niketan Pansare, Berthold Reinwald, 2016https://arxiv.org/abs/1605.058263. The MADlib Analytics Library or MAD Skills, the SQL — Joseph M. Hellerstein, Christopher Re, Florian Schoppmann, Daisy Zhe Wang, et al., 2012https://arxiv.org/abs/1208.41654. MLog: Towards Declarative In-Database Machine Learning — Xupeng Li, Bin Cui, Yiru Chen, Wentao Wu, Ce Zhang, 2017https://www.vldb.org/pvldb/vol10/p1933-zhang.pdf5. sql4ml A declarative end-to-end workflow for machine learning — Nantia Makrynioti, Ruy Ley-Wild, Vasilis Vassalos, 2019https://arxiv.org/abs/1907.124156. Deep Feature Synthesis: Towards Automating Data Science Endeavors — James Max Kanter, Kalyan Veeramachaneni, 2015https://doi.org/10.1109/DSAA.2015.73448587. Label, Segment, Featurize: A Cross Domain Framework for Prediction Engineering — James Max Kanter, Owen Gillespie, Kalyan Veeramachaneni, 2016https://www.maxkanter.com/papers/DSAA_LSF_2016.pdf8. Predictive Query Language: A Domain-Specific Language for Predictive Modeling on Relational Databases — Vid Kocijan, Jinu Sunil, Jan Eric Lenssen, Viman Deb, et al., 2026https://arxiv.org/abs/2602.095729. RelBench: A Benchmark for Deep Learning on Relational Databases — Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, et al., 2024https://arxiv.org/abs/2407.2006010. Temporal features in SQL:2011 — Krishna Kulkarni, Jan-Eike Michels, 2012https://sigmodrecord.org/publications/sigmodRecord/1209/pdfs/07.industry.kulkarni.pdf11. Time Travel and Provenance for Machine Learning Pipelines — Alexandru A. Ormenisan, Moritz Meister, Fabio Buso, Robin Andersson, Seif Haridi, Jim Dowling, 2020https://www.usenix.org/conference/opml20/presentation/ormenisan12. Optimizing Data Pipelines for Machine Learning in Feature Stores — Rui Liu, Kwanghyun Park, Fotis Psallidas, Xiaoyong Zhu, et al., 2023https://www.microsoft.com/en-us/research/publication/optimizing-data-pipelines-for-machine-learning-in-feature-stores/13. The Hopsworks Feature Store for Machine Learning — Javier de la Rua Martinez, Fabio Buso, Antonios Kouzoupis, Alexandru A. Ormenisan, et al., 2024https://content.hopsworks.ai/hubfs/The_Hopsworks_Feature_Store_for_Machine_Learning.pdf14. Leakage in Data Mining: Formulation, Detection, and Avoidance — Shachar Kaufman, Saharon Rosset, Claudia Perlich, 2012https://doi.org/10.1145/2020408.202049615. How to avoid machine learning pitfalls: a guide for academic researchers — Michael A. Lones, 2021https://arxiv.org/abs/2108.0249716. Leakage and the reproducibility crisis in machine-learning-based science — Sayash Kapoor, Arvind Narayanan, 2023https://doi.org/10.1016/j.patter.2023.10080417. MLearn: A Declarative Machine Learning Language for Database Systems — Maximilian E. Schuele, Matthias Bungeroth, Alfons Kemper, Stephan Guennemann, Thomas Neumann, 2019https://scholar.google.com/scholar?q=MLearn%3A+A+Declarative+Machine+Learning+Language+for+Database+Systems18. End-to-end Optimization of Machine Learning Prediction Queries — Kwanghyun Park, Karla Saur, Dalitso Banda, Rathijit Sen, Matteo Interlandi, Konstantinos Karanasos, 2022https://scholar.google.com/scholar?q=End-to-end+Optimization+of+Machine+Learning+Prediction+Queries19. Relational Deep Learning: Graph Representation Learning on Relational Databases — Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, Jure Leskovec, 2023https://scholar.google.com/scholar?q=Relational+Deep+Learning%3A+Graph+Representation+Learning+on+Relational+Databases20. KumoRFM: A Foundation Model for In-Context Learning on Relational Data — Matthias Fey, Vid Kocijan, Federico Lopez, Jan Eric Lenssen, Jure Leskovec, 2025https://scholar.google.com/scholar?q=KumoRFM%3A+A+Foundation+Model+for+In-Context+Learning+on+Relational+Data21. Toward a Declarative Query Language for Machine Learning — Hasan M. Jamil, 2024https://vldb.org/workshops/2024/proceedings/TaDA/TaDA.2.pdf22. Implementing a Declarative Query Language for High Level Machine Learning Application Design — Hasan Rahman, Hasan M. Jamil, 2025https://papers.ssrn.com/sol3/papers.cfm?abstract_id=521020423. LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL — Yihan Wang, Peiyu Liu, Xin Yang, 2025https://arxiv.org/abs/2503.1859624. E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL — Hasan Alp Caferoglu, Ozgur Ulusoy, 2024https://arxiv.org/abs/2409.1675125. AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale — Ziyang Wang et al., 2025https://arxiv.org/abs/2511.1719026. From Alignment to Entailment: A Unified Textual Entailment Framework for Entity Alignment — Yu Zhao et al., 2023https://arxiv.org/abs/2305.1150127. AI Post Transformers: SGLang for Faster Structured LLM Programs — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-06-sglang-for-faster-structured-llm-program-c59f1c.mp328. AI Post Transformers: Caffe and the Rise of CNN Frameworks — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-06-caffe-and-the-rise-of-cnn-frameworks-cf15f3.mp314 min
- KumoRFM-2 for Relational Learning at ScaleThis episode explores KumoRFM-2, a relational foundation model designed to learn directly from connected database tables instead of flattening customers, orders, products, and tickets into a single feature table. It explains why relational learning matters for enterprise tasks such as churn, fraud, and demand prediction, arguing that flattening often erases multi-hop relationships, repeated interactions, and temporal patterns that carry the real signal. The discussion centers on KumoRFM-2’s main technical claim: a two-stage, task-conditioned attention pipeline that first selects relevant information within each table and then aggregates evidence across foreign-key neighborhoods and labeled in-context examples derived from predictive queries. Listeners would find it interesting because it connects a very practical data-engineering pain point to a broader question about whether pretrained, database-native models can beat hand-built tabular pipelines without cheating on time-aware prediction.Sources:1. KumoRFM-2: Scaling Foundation Models for Relational Learning — Valter Hudovernik, Federico López, Vid Kocijan, Akihiro Nitta, Jan Eric Lenssen, Jure Leskovec, Matthias Fey, 2026http://arxiv.org/abs/2604.125962. Learning Probabilistic Relational Models — Nir Friedman, Lise Getoor, Daphne Koller, Avi Pfeffer, 1999https://scholar.google.com/scholar?q=Learning+Probabilistic+Relational+Models3. Relational Inductive Biases, Deep Learning, and Graph Networks — Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Yujia Li, Razvan Pascanu, et al., 2018https://scholar.google.com/scholar?q=Relational+Inductive+Biases%2C+Deep+Learning%2C+and+Graph+Networks4. Relational Deep Learning: Graph Representation Learning on Relational Databases — Matthias Fey, Weihua Hu, Kexin Huang, Jan Eric Lenssen, Rishabh Ranjan, Joshua Robinson, Rex Ying, Jiaxuan You, Jure Leskovec, 2023https://scholar.google.com/scholar?q=Relational+Deep+Learning%3A+Graph+Representation+Learning+on+Relational+Databases5. RelBench: A Benchmark for Deep Learning on Relational Databases — Joshua Robinson, Rishabh Ranjan, Weihua Hu, Kexin Huang, Jiaqi Han, Alejandro Dobles, Matthias Fey, Jan E. Lenssen, Jure Leskovec, et al., 2024https://scholar.google.com/scholar?q=RelBench%3A+A+Benchmark+for+Deep+Learning+on+Relational+Databases6. Position: Why Tabular Foundation Models Should Be a Research Priority — Boris Van Breugel, Mihaela Van Der Schaar, 2024https://scholar.google.com/scholar?q=Position%3A+Why+Tabular+Foundation+Models+Should+Be+a+Research+Priority7. Accurate Predictions on Small Data with a Tabular Foundation Model — Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter, et al., 2025https://scholar.google.com/scholar?q=Accurate+Predictions+on+Small+Data+with+a+Tabular+Foundation+Model8. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data — Jingang Qu, David Holzmüller, Gaël Varoquaux, Marine Le Morvan, 2025https://scholar.google.com/scholar?q=TabICL%3A+A+Tabular+Foundation+Model+for+In-Context+Learning+on+Large+Data9. KumoRFM: A Foundation Model for In-Context Learning on Relational Data — Matthias Fey, Vid Kocijan, Federico Lopez, Jan Eric Lenssen, Jure Leskovec, 2025https://scholar.google.com/scholar?q=KumoRFM%3A+A+Foundation+Model+for+In-Context+Learning+on+Relational+Data10. PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models — Vignesh Kothapalli et al., 2026https://scholar.google.com/scholar?q=PluRel%3A+Synthetic+Data+unlocks+Scaling+Laws+for+Relational+Foundation+Models11. Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data — Rishabh Ranjan et al., 2026https://scholar.google.com/scholar?q=Relational+Transformer%3A+Toward+Zero-Shot+Foundation+Models+for+Relational+Data12. No Need to Train Your RDB Foundation Model — Linjie Xu, Yanlin Zhang, Quan Gan, Minjie Wang, and David Wipf, 2026https://scholar.google.com/scholar?q=No+Need+to+Train+Your+RDB+Foundation+Model13. TabICLv2: A better, faster, scalable, and open tabular foundation model — Jingang Qu, David Holzmuller, Gael Varoquaux, and Marine Le Morvan, 2026https://scholar.google.com/scholar?q=TabICLv2%3A+A+better%2C+faster%2C+scalable%2C+and+open+tabular+foundation+model14. Griffin: Towards a Graph-Centric Relational Database Foundation Model — Yanbo Wang, Xiyuan Wang, Quan Gan, Minjie Wang, Qibin Yang, David Wipf, and Muhan Zhang, 2025https://scholar.google.com/scholar?q=Griffin%3A+Towards+a+Graph-Centric+Relational+Database+Foundation+Model15. Graph Machine Learning Meets Multi-Table Relational Data — Quan Gan, Minjie Wang, David Wipf, Christos Faloutsos, 2024https://scholar.google.com/scholar?q=Graph+Machine+Learning+Meets+Multi-Table+Relational+Data16. Large Scale Transfer Learning for Tabular Data via Language Modeling — Josh Gardner, Juan C. Perdomo, Ludwig Schmidt, 2024https://scholar.google.com/scholar?q=Large+Scale+Transfer+Learning+for+Tabular+Data+via+Language+Modeling17. Towards Synthetic Data for Fine-tuning Tabular Foundation Models — Magnus Buhler, Lennart Purucker, Frank Hutter, 2025https://scholar.google.com/scholar?q=Towards+Synthetic+Data+for+Fine-tuning+Tabular+Foundation+Models18. Range-limited Augmentation for Few-shot Learning in Tabular Data with Comprehensive Benchmark — Kyungeun Lee et al., 2025https://scholar.google.com/scholar?q=Range-limited+Augmentation+for+Few-shot+Learning+in+Tabular+Data+with+Comprehensive+Benchmark19. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp320. AI Post Transformers: Scaling Laws for Multilingual Code Pretraining — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-15-scaling-laws-for-multilingual-code-pretr-7d220e.mp314 min
- Unified Neural Scaling Laws Across RegimesThis episode explores Unified Neural Scaling Laws, a framework for predicting model performance when parameter count, data volume, training steps, inference compute, and training-recipe choices all change at once. It explains how the paper moves beyond classic smooth power-law curves by introducing broken multivariate scaling laws with regime shifts, including hyperbreaks, bottleneck versus non-bottleneck components, and joint interaction surfaces across training variables. The discussion highlights the paper’s argument that good forecasting must capture both beneficial scaling effects and harmful effects such as overfitting, bad hyperparameter regimes, and data or compute limits, rather than assuming one clean trend forever. A listener would find it interesting because it ties abstract scaling-law math directly to expensive real-world training decisions and to the question of whether pretraining gains actually transfer to downstream benchmarks.Sources:1. Unified Neural Scaling Laws — Ethan Caballero, Priyank Jaini, David Krueger, Irina Rish, 2026http://arxiv.org/abs/2605.262482. A Constructive Prediction of the Generalization Error Across Scales — Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, Nir Shavit, 2020https://scholar.google.com/scholar?q=A+Constructive+Prediction+of+the+Generalization+Error+Across+Scales3. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al., 2020https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models4. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al., 2022https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models5. Unified Neural Scaling Laws — Ethan Caballero, Priyank Jaini, David Krueger, Irina Rish, 2026https://scholar.google.com/scholar?q=Unified+Neural+Scaling+Laws6. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — Priya Goyal, Piotr Dollar, Ross Girshick, Kaiming He, et al., 2017https://scholar.google.com/scholar?q=Accurate%2C+Large+Minibatch+SGD%3A+Training+ImageNet+in+1+Hour7. An Empirical Model of Large-Batch Training — Sam McCandlish, Jared Kaplan, Dario Amodei, OpenAI Dota Team, 2018https://scholar.google.com/scholar?q=An+Empirical+Model+of+Large-Batch+Training8. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer — Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, et al., 2022https://scholar.google.com/scholar?q=Tensor+Programs+V%3A+Tuning+Large+Neural+Networks+via+Zero-Shot+Hyperparameter+Transfer9. Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, Yair Carmon, 2024https://scholar.google.com/scholar?q=Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models10. Reconciling modern machine learning practice and the bias-variance trade-off — Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal, 2019https://scholar.google.com/scholar?q=Reconciling+modern+machine+learning+practice+and+the+bias-variance+trade-off11. Deep Double Descent: Where Bigger Models and More Data Hurt — Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever, 2020https://scholar.google.com/scholar?q=Deep+Double+Descent%3A+Where+Bigger+Models+and+More+Data+Hurt12. Broken Neural Scaling Laws — Ethan Caballero, Kshitij Gupta, Irina Rish, David Krueger, 2023https://scholar.google.com/scholar?q=Broken+Neural+Scaling+Laws13. Scaling Data-Constrained Language Models — Niklas Muennighoff, Alexander M. Rush, Boaz Barak, et al., 2023https://scholar.google.com/scholar?q=Scaling+Data-Constrained+Language+Models14. Observational Scaling Laws and the Predictability of Language Model Performance — Yangjun Ruan, Chris J. Maddison, Tatsunori Hashimoto, 2024https://scholar.google.com/scholar?q=Observational+Scaling+Laws+and+the+Predictability+of+Language+Model+Performance15. Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic — Sachin Goyal et al., 2024https://scholar.google.com/scholar?q=Scaling+Laws+for+Data+Filtering+--+Data+Curation+cannot+be+Compute+Agnostic16. Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies — Zhengyu Chen et al., 2025https://scholar.google.com/scholar?q=Revisiting+Scaling+Laws+for+Language+Models%3A+The+Role+of+Data+Quality+and+Training+Strategies17. Scaling Laws for Optimal Data Mixtures — Mustafa Shukor et al., 2025https://scholar.google.com/scholar?q=Scaling+Laws+for+Optimal+Data+Mixtures18. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient — Jan Ludziejewski et al., 2025https://scholar.google.com/scholar?q=Joint+MoE+Scaling+Laws%3A+Mixture+of+Experts+Can+Be+Memory+Efficient19. Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning — Wenkai Yang et al., 2025https://scholar.google.com/scholar?q=Towards+Thinking-Optimal+Scaling+of+Test-Time+Compute+for+LLM+Reasoning20. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping et al., 2025https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach21. Towards Robust Scaling Laws for Optimizers — Alexandra Volkova et al., 2026https://scholar.google.com/scholar?q=Towards+Robust+Scaling+Laws+for+Optimizers22. Neural Scaling Laws Rooted in the Data Distribution — Ari Brill, 2024https://scholar.google.com/scholar?q=Neural+Scaling+Laws+Rooted+in+the+Data+Distribution23. The Quantization Model of Neural Scaling — Eric J. Michaud et al., 2023https://scholar.google.com/scholar?q=The+Quantization+Model+of+Neural+Scaling24. AI Post Transformers: Muon Is Scalable for LLM Training — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-04-25-muon-is-scalable-for-llm-training-587ed8.mp325. AI Post Transformers: TMAS: Scaling Test-Time Compute with Multi-Agent Synergy — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-14-tmas-scaling-test-time-compute-with-mult-3abe7a.mp326. AI Post Transformers: Trajectory Summaries for Long-Horizon Coding Agents — Hal Turing & Dr. Ada Shannon, 2026https://podcast.do-not-panic.com/episodes/2026-05-24-trajectory-summaries-for-long-horizon-co-0194be.mp3Interactive Visualization: Unified Neural Scaling Laws Across Regimes0 min
About AI Post Transformers
From the publisher's feed
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis,…
proper citations, and nerdy humor.