In this episode:
• Introduction: Less is More in Optimization?: Professor Norris and Linda introduce the "Drop-Muon" paper, which challenges the fundamental assumption that all neural network layers must be updated at every training step. They set the stage by questioning if selectively updating layers could lead to faster convergence.
• A Refresher on the Muon Family: Linda provides a high-level overview of modern non-Euclidean optimizers like Muon, Scion, and Gluon. They discuss how these methods use layer-specific geometry to improve training, which provides the foundation for the Drop-Muon approach.
• The Drop-Muon Algorithm: Randomized Progressive Training: Linda explains the core mechanism of Drop-Muon, focusing on how it samples a random subset of layers to update at each iteration. Professor Norris probes the practicalities of this approach, especially the concept of 'Randomized Progressive Training' and its computational cost.
• The Theoretical Justification: When is Full-Network Update Optimal?: The hosts delve into the paper's theoretical contributions, highlighting the key finding that full-network updates are only optimal under a very restrictive and unlikely condition on layer smoothness constants. They discuss the implications of the cost model, which accounts for backpropagation and parameter update costs.
• Empirical Results and Final Thoughts: Linda presents the experimental results, which show Drop-Muon achieving the same accuracy as standard Muon up to 1.4x faster in wall-clock time. They conclude by discussing the practical impact of this 'update less, converge faster' strategy for training large models.

In this episode:
• The Finicky Diet of Large Language Models: Linda introduces a paper about how LLMs learn from mixtures of web data and high-quality data. Professor Norris expresses his initial intuition that more data is always better, setting the stage for the paper's surprising findings.
• It's Not a Slope, It's a Cliff: Unveiling Phase Transitions: The hosts discuss the paper's core finding: knowledge acquisition isn't gradual but exhibits sudden 'phase transitions'. Linda explains how, below a critical model size or data mixing ratio, models learn almost nothing from specialized datasets, a result Professor Norris finds both fascinating and counter-intuitive.
• The Knapsack Theory of Knowledge: To explain the 'why', Linda and Professor Norris explore the paper's theoretical model of 'capacity allocation'. They use a knapsack analogy to describe how a model with finite capacity strategically decides which data is 'worth' learning to minimize overall loss.
• Learning More by Training on Less?: Linda and Professor Norris discuss the practical implications, including the paradoxical strategy of throwing away data to improve learning. They cover the paper's proposed solutions, like random subsampling and Compact Knowledge Mixing, and what this means for data curation.
• Final Thoughts and Critical Points: The hosts summarize the paper's key insight: data mixing recipes are not one-size-fits-all, and the relationship between model size, data, and knowledge is sharp and discontinuous. They wrap up by emphasizing the importance of understanding these dynamics for efficient model training.

In this episode:
• Another Week, Another 'Open' Model?: Linda introduces the Apertus paper, framing it as a response to the systemic shortcomings of current open models. Professor Norris questions what makes this one different from the countless other 'open' releases.
• Data Compliance and the Goldfish in the Machine: The hosts dive into Apertus's strict data compliance, including its novel retroactive application of robots.txt and the use of the 'Goldfish' training objective to prevent the model from memorizing its training data.
• More Than Just English: A Truly Global LLM: Linda gets excited about the model's vast multilingual capabilities, trained on over 1800 languages. They discuss the implications for low-resource languages and the significance of a 40% non-English training data mix.
• The Swiss AI Charter and Other Training Secrets: The discussion turns to the technical details of training Apertus, including its unique optimizer and its novel approach to safety alignment using a 'Swiss AI Charter' for controversial topics.
• Final Thoughts: A New Standard for Openness?: Professor Norris and Linda summarize Apertus's contributions, concluding that its commitment to compliance, multilingualism, and full transparency sets a powerful new benchmark for the entire field.

In this episode:
• The Forgetful Student: Professor Norris and Linda introduce the central problem: Large Language Models often struggle to reliably learn and recall facts. They set the stage for this week's paper, which proposes a solution inspired by how humans study.
• Learning by Self-Teaching: Linda explains the core concept of 'Active Reading,' where a model generates its own diverse study materials like timelines, summaries, and associations to internalize knowledge from a given text.
• From 16% to 66% Accuracy: The hosts dive into the stunning results, where Active Reading drastically outperforms methods like simple repetition or standard data augmentation on expert QA benchmarks, showing massive gains in factual recall.
• A Trillion Tokens of Homework: The discussion turns to scaling this method to the entire Wikipedia, creating an 8-billion parameter 'WikiExpert' model that punches far above its weight, and the surprising training tweaks needed to make it work.
• The Self-Taught Model: Professor Norris and Linda wrap up by reflecting on the key insight that models learn best when they teach themselves. They discuss the implications for building more reliable and factual AI systems.

In this episode:
• The Optimizer Royal Rumble: Professor Norris and Linda introduce the chaotic landscape of LLM optimizers, where everyone claims to beat the reigning champion, AdamW. They introduce today's paper, which aims to be the referee in this messy fight.
• The Art of the Unfair Comparison: Linda explains the paper's core thesis: many new optimizers seem fast only because they are compared against poorly tuned baselines. Professor Norris agrees, highlighting the critical importance of fair hyperparameter tuning.
• Diminishing Returns and Shifting Allegiances: The hosts dive into the paper's main findings, discussing how the speedup of new optimizers shrinks with model size and how the 'best' optimizer can change depending on the amount of training data.
• So... Do We Ditch AdamW?: Norris and Linda synthesize the practical takeaways for practitioners. They conclude that while AdamW's dominance is challenged, the victory of its rivals is not as clear-cut as claimed, praising the paper for its methodological rigor.

In this episode:
• Beyond Adam: The Great Optimizer Bake-Off: Linda introduces a paper questioning the decade-long reign of the AdamW optimizer for training large language models. Professor Norris expresses his healthy skepticism about the endless stream of 'new and improved' optimizers.
• Adam's Kingdom and Its Challengers: The hosts discuss why AdamW became the default and the paper's motivation: the lack of systematic, fair comparisons between the many new optimizers claiming to be better. Professor Norris recalls past optimizer fads.
• Creating a Level Playing Field: Linda details the paper's rigorous experimental setup, covering the 11 optimizers tested and the massive hyperparameter tuning effort required for a fair fight. Professor Norris is impressed by the scale of the benchmark.
• And the Winner Is... It's Complicated: Linda reveals the main results, highlighting that AdEMAMix and MARS are the new frontrunners, especially at scale. They break down the results from the paper's many graphs, discussing where different optimizers shine.
• Actionable Advice for the Practitioner: Professor Norris and Linda distill the paper's 'takeaways' into practical advice for listeners. They discuss the critical and often overlooked role of weight decay, learning rate schedules, and warmup.
• The Optimization Frontier: The hosts conclude that while AdamW's dominance is over, the best optimizer is context-dependent. They wrap up by discussing the paper's impact and the future of optimization research.

In this episode:
• Opening Up a New Chapter: Apertus: Linda introduces the Apertus paper, highlighting its focus on data compliance and extreme multilingualism. Professor Norris is intrigued but skeptical about what 'fully open' and 'compliant' truly mean in practice.
• Clean Data, Clear Conscience?: The hosts discuss Apertus's novel approach to data compliance, including retroactively honoring robots.txt opt-outs. They debate the ethical implications and the performance trade-offs of training on a more restricted, 'cleaner' dataset.
• Speaking Over 1800 Languages: Linda explains the massive scale of Apertus's multilingual training, with 40% of its data being non-English. Professor Norris questions the depth versus breadth of language understanding, especially for the thousands of low-resource languages included.
• Forgetting for a Better Future: The Goldfish Loss: The conversation turns to the technical recipe, focusing on the 'Goldfish objective' designed to prevent memorization. Professor Norris finds the name amusing and probes whether this technique genuinely reduces copyright and privacy risks without harming the model's capabilities.
• The Verdict on Apertus: Linda and Professor Norris wrap up by evaluating Apertus's position in the LLM landscape. They conclude that its commitment to full transparency—releasing code, data scripts, and checkpoints—sets a new, important standard for the open-source community.