In this episode:
• Introduction: More Than an Improv Artist?
• The Two Faces of Prediction: Linda breaks down the key difference between teacher-forced training and autoregressive inference. Professor Norris likens this to learning to drive with an instructor who constantly corrects you, which doesn't prepare you for driving alone.
• The Clever Hans Cheat: Linda explains the paper's core concept: 'teacher-forcing' can lead to a 'Clever Hans cheat,' where the model learns shortcuts from the training data instead of the actual task. This results in the first, most crucial token of a plan becoming 'indecipherable.'
• Breaking the Cheat Code: The hosts discuss the paper's proposed solutions, like 'teacherless training,' which force the model to look ahead and plan instead of relying on shortcuts. Professor Norris notes this is like removing the training wheels, forcing the model to learn the hard way.
• Conclusion: Recalibrating the Paradigm: Linda and Professor Norris conclude that the paper makes a strong, empirically-backed case that the teacher-forcing training objective is a core limitation. They agree it's a pivotal step in moving the field towards models that can genuinely plan.

In this episode:
• More Than Just NPCs: Linda introduces this week's paper, a comprehensive survey on the roles of Large Language Models in games. Professor Norris provides some witty historical context on game AI, setting the stage for the discussion.
• What Counts as 'Large'?: The hosts discuss the paper's specific definition of a Large Language Model. They cover why establishing a clear scope, centered on transformer models of a certain size, is crucial for a useful survey.
• The LLM as a Player: Linda breaks down how LLMs can act as players in different types of games, from board games like Chess to complex API-driven worlds like Minecraft's VOYAGER. Professor Norris questions the path to superhuman performance.
• The Supporting Cast: The discussion moves beyond players to other in-game roles for LLMs. This includes creating dynamic Non-Player Characters (NPCs), acting as helpful Player Assistants, and even generating commentary for game streams.
• The LLM as Creator: The hosts explore the creative roles for LLMs, such as acting as a Game Master in tabletop role-playing games or as a Design Assistant that collaborates with human developers to generate levels and game concepts.
• Hallucinations, Copyright, and the Future: Professor Norris and Linda conclude by discussing the paper's roadmap, highlighting the key limitations like hallucinations and the significant ethical questions around copyright and bias that the field must address.

In this episode:
• The Benchmark Treadmill: Linda introduces the problem with existing ML benchmarks, noting they are often either too easy or too artificial. Professor Norris adds witty commentary on how quickly new models seem to 'solve' and saturate these tests.
• Let's Ask the Unanswerable: Linda presents the core idea from the UQ paper: evaluating models on genuinely unsolved questions from platforms like Stack Exchange. Professor Norris and Linda discuss how this hits a sweet spot between being difficult and realistic.
• How to Find a Good Unsolved Question: The hosts dive into the meticulous creation of the UQ-Dataset. Linda explains the three-stage filtering pipeline, and Professor Norris expresses his appreciation for the rigor involved in finding high-quality, truly unsolved problems.
• Who Validates the Validator?: With no ground truth answers, how do you score the models? Linda explains the clever 'UQ-Validator' system and the 'generator-validator gap,' while Professor Norris highlights the crucial role of the community platform for human verification.
• Pushing the Frontier of Knowledge... Slowly: Linda and Professor Norris review the humbling results, where even top models pass the validator on only 15% of questions. They discuss the implications of this new, more challenging evaluation paradigm for the future of AI research.

In this episode:
• The Billion-Dollar Guessing Game: Professor Norris and Linda introduce the high-stakes problem of LLM evaluation. Linda presents today's paper, which offers a framework to make our small-scale experiments more predictive of large-scale success.
• Tuning In the Signal, Tuning Out the Noise: Linda breaks down the paper's core concepts: 'signal' as a benchmark's ability to distinguish models and 'noise' as its random variability. Professor Norris helps clarify with analogies, questioning if it's really that simple.
• From Lab Coat to Crystal Ball: The hosts discuss how the Signal-to-Noise Ratio (SNR) predicts real-world outcomes, like whether a good small model scales up well (decision accuracy) and how accurately we can predict future performance (scaling law error).
• Three Simple Tricks to a Better Benchmark: Linda enthusiastically details the paper's three practical interventions for improving benchmarks: filtering noisy subtasks, averaging final checkpoints, and switching to continuous metrics like bits-per-byte.
• The Sound of a Clear Signal: Professor Norris and Linda recap the main lesson: when choosing or creating a benchmark, aim for high signal and low noise. They conclude that this simple framework provides a powerful, practical tool for the entire ML community.

In this episode:
• Can a Jack-of-All-Trades Learn to Be a Doctor?: Linda introduces the challenge of specializing large language models for specific domains and presents a new paper that proposes a smarter way to pick the right training data.
• The Deceptive First Sip: The hosts discuss why the common practice of 'micro-annealing'—using small-scale tests to evaluate data sources—can be misleading, as the best data for a short run may not be the best for a long one.
• Plotting the Curve, Not Just the Point: Linda explains the paper's core proposal: instead of relying on a single test, estimate a scaling law for each data source by running multiple experiments to predict its utility at scale.
• The Tortoise and the Hare of Data: Professor Norris and Linda dive into the paper's key experiment, revealing how synthetic data (the hare) starts fast but is overtaken by more diverse, filtered data (the tortoise) as compute increases.
• Scaling Smartly: The Takeaway: The hosts conclude by emphasizing the practical importance of scaling-aware data selection to avoid wasting significant compute and money on suboptimal data strategies.

In this episode:
• The Transformer's Black Box: Linda introduces the 'Thinking Like Transformers' paper, highlighting the challenge of understanding the computational model behind transformers, unlike RNNs and their connection to finite state machines. Professor Norris agrees, sharing a witty remark about the opacity of modern deep learning models.
• Introducing RASP: A Language for Transformers: Linda explains the core concept of RASP (Restricted Access Sequence Processing Language), a programming language designed to mirror the information flow of a transformer. She details the main operations: element-wise computations, and the crucial 'select' and 'aggregate' pair that mimics attention.
• From Code to Heads: RASP in Action: To make the concepts concrete, Linda walks through a simple RASP program from the paper, such as creating a histogram of tokens. They discuss the key insight that a RASP program can be 'compiled' to estimate the number of layers and attention heads a transformer would need for the task.
• Implications and Insights: The hosts explore the broader implications of the RASP model, such as analyzing the expressive power of restricted-attention models and explaining empirical results like the 'Sandwich Transformer'. Professor Norris is particularly intrigued by how this formal model can explain real-world phenomena.
• Thinking Like a Researcher: Professor Norris and Linda summarize the paper's contributions, agreeing that RASP provides a powerful conceptual tool for reasoning about transformer capabilities. Linda concludes by mentioning the publicly available RASP REPL for listeners who want to experiment themselves.

In this episode:
• Honey, I Shrunk the Activated Parameters: Linda introduces the massive Kimi K2 paper, focusing on its 'agentic intelligence' and surprisingly small number of activated parameters. Professor Norris offers his initial witty skepticism about yet another trillion-parameter model.
• Taming the Exploding Logits: The hosts get technical, discussing the novel MuonClip optimizer designed to solve training instability. They also explore the clever pre-training data strategy of 'rephrasing' to maximize token utility from a limited data pool.
• Teaching a Model to Use Tools: This chapter focuses on post-training, where Linda explains the large-scale synthetic data pipeline for teaching tool use. They also delve into the reinforcement learning framework that combines verifiable rewards with a self-critique mechanism.
• Climbing the Leaderboard: Linda and Professor Norris unpack Kimi K2's impressive benchmark performance, highlighting its state-of-the-art results on agentic and coding tasks. They conclude with final thoughts on what this powerful open-weight model means for the field.