In this episode:
• Introduction to State-Prediction Separation: Professor Norris and Linda introduce the paper and the core problem of standard Transformers conflating next-token prediction and future state representation.
• The SPS Mechanism: Linda explains how the authors introduce a dummy predict token to separate the input state stream from the prediction stream.
• Experimental Results and Baselines: The hosts discuss the massive data efficiency gains and how clever ablations like Delayed State prove that separation, not just compute, is the key.
• Gradient Analysis and Inference: A deep dive into how gradients are routed during training and why the inference overhead is practically negligible.
• Conclusion and Implications: Professor Norris is convinced by the arguments, and the hosts wrap up by discussing what this means for the future of LLM architectures.

In this episode:
• Introduction to Monolithic MoEs: Linda introduces EMO and Professor Norris discusses the memory bottlenecks of standard monolithic Large Language Models.
• The Document-Level Routing Constraint: Linda explains EMOs core mechanism of restricting tokens in a document to a shared pool of experts.
• Overcoming Training Instability: The hosts discuss the conflict with local load balancing and how global load balancing and dynamic pool sizing solved it.
• Semantic Specialization and Extreme Pruning: Linda reveals the performance numbers, showing how EMO can drop over 85 percent of its experts without losing accuracy.
• Future Implications for Interpretability: Professor Norris realizes the potential for model auditing and deploying specialized sub-networks on edge devices.

In this episode:
• Introduction & The Edge Deployment Problem: Professor Norris and Linda introduce the episode's paper, focusing on the difficulty of running large language models on edge devices with limited memory and compute.
• The Multi-Teacher Distillation Pipeline: Linda explains the dual-teacher approach, using Llama 4 Scout for synthetic data generation and Llama 3.3 70B for logit-based knowledge distillation to a 3B parameter student.
• Hyperparameter Surprises with Optuna: The hosts discuss the surprising finding that Bayesian optimization consistently drops the cross-entropy loss in favor of pure Kullback-Leibler divergence from the teacher.
• The Muon Optimizer and Quantization Robustness: Linda introduces the Muon optimizer and explains how fine-tuning with it dramatically reduces accuracy decay when the model is compressed to 4-bit weights using GPTQ.
• Results & Conclusion: A summary of the final throughput and latency gains, proving that highly specialized, edge-ready LLMs can achieve double the speed and half the memory without sacrificing accuracy.

In this episode:
• Welcome & The Parametric Memory Bottleneck: Professor Norris and Linda introduce the episode's topic, discussing the fundamental flaws of storing facts in model weights.
• From Relational to Continuous Queries: Linda explains the limitations of prior Limited Memory Language Models (LMLMs) and introduces Co-LMLM's continuous-query approach.
• Under the Hood: Training and The Token: The hosts dive into the technical details of how the model uses hidden states as retrieval queries via a special token.
• Scaling Up: The Annotation Pipeline: Discussion on how the researchers scaled their training data to 100 billion tokens using a clever distillation pipeline.
• Results, Unlearning, and Wrap Up: Reviewing the model's performance on factuality benchmarks and its elegant solution to machine unlearning.

In this episode:
• Introduction to Quantization and Outliers: Norris and Linda introduce the episode's paper, QuaRot, and discuss the main hurdle in LLM inference: the memory bottleneck and the pesky outlier features in activations.
• The Magic of Hadamard Rotations: Linda explains the core mechanism of QuaRot, using randomized Hadamard transformations to eliminate outliers through computational invariance.
• Taming the Attention Mechanism and KV Cache: The hosts dive into the complexities of quantizing the attention module, discussing how QuaRot rotates keys and values to achieve a fully 4-bit KV cache.
• Performance, Results, and Hardware: An analysis of the empirical results on the LLaMA-2 models, highlighting the impressive memory savings, speedups, and maintained zero-shot performance.
• Conclusion and Takeaways: Norris and Linda wrap up the discussion, reflecting on what end-to-end 4-bit inference means for the future of deploying massive language models locally.

In this episode:
• Introduction to Mechanical Dreams and the AdamW Scaling Problem: Professor Norris and Linda introduce the podcast and discuss the costly problem of hyperparameter tuning for large-scale machine learning models.
• Reframing AdamW as an Exponential Moving Average: Linda reveals the paper's core insight that AdamW can be mathematically understood as an EMA of recent weight updates.
• Scaling Weight Decay with Dataset Size: The hosts discuss the concept of timescale measured in epochs and how optimal weight decay should actually decrease as dataset size grows.
• Fixing muP for AdamW Across Model Sizes: Professor Norris and Linda explore how this new framework fixes the breakdown of Maximal Update Parametrization by scaling weight decay with model width.
• Conclusion and Practical Takeaways: The hosts summarize the scaling rules and reflect on how this impacts the future of training large language models.

In this episode:
• Introduction and The Premise: Professor Norris and Linda introduce the episode's focus: a new paper showing how weight decay impacts language model plasticity.
• Challenging the Validation Loss Paradigm: Linda explains how optimizing for pretraining validation loss can actually hurt downstream fine-tuning performance.
• The Mechanics of Plasticity: A deep dive into why weight decay helps, covering linearly separable representations, attention matrix rank reduction, and the prevention of overfitting.
• Trade-offs and Practical Implications: The hosts discuss the stability-plasticity dilemma and how these findings should change the way industry teams tune hyperparameters.
• Conclusion: Professor Norris concedes his skepticism and the hosts wrap up the episode.

In this episode:
• The Data Wall: Introduction to the problem of running out of fresh high-quality pretraining data and the limitations of classical Chinchilla scaling.
• The Compute-Data Scaling Law: Linda introduces the new CD-scaling law, focusing on the token effectiveness function that bridges compute-optimal and data-optimal scaling.
• Eta and the Saturation Ceiling: A deep dive into how data effectiveness decays with model size and data availability, culminating in the R-star saturation ceiling.
• Paraphrasing vs Repetition: Discussion on practical data expansion strategies, revealing why simple multi-epoch repetition surprisingly beats paraphrasing for large models.
• Three Operational Regimes: Summarizing the compute-bound, data-bound, and model-bound regimes, providing a map for future LLM pretraining resource allocation.

In this episode:
• Introduction to Higher-Order Optimizers: Linda introduces the NVIDIA paper on SOAP and Muon, and Professor Norris expresses his initial skepticism about dethroning AdamW.
• The Batch Size Wall and MoEs: The hosts discuss why AdamW struggles at massive batch sizes, particularly for the dense layers in Mixture of Experts models.
• Muon and Spectral Descent: Linda explains how Muon orthogonalizes momentum using Newton-Schulz iterations to maintain stability at scale.
• Fixing SOAP's Slingshot Instability: The conversation shifts to SOAP, diagnosing the stale preconditioner problem and how real-time eigenbasis updates and KL-Shampoo fix it.
• Systems Engineering and Final Verdict: Professor Norris and Linda discuss the layer-wise distributed optimizer built for Megatron-LM and conclude which optimizer wins.

In this episode:
• Introduction: The Mystery of Scaling: Professor Norris and Linda introduce the core question of why larger models learn tasks that smaller ones cannot, challenging the standard sample-efficiency dogma.
• The Statistical Bottleneck and Task Utility: Linda explains the paper's synthetic regression setup, detailing how models prioritize tasks based on a utility metric derived from task frequency and complexity.
• Gradient Interference and The Update-and-Forget Loop: The hosts dive into the learning dynamics, discussing how frequent tasks overwrite the gradients of rare tasks in small models, and how width solves this interference.
• Real-World Validation with OLMo: Linda brings the theoretical findings into the realm of real LLMs, discussing the pre-training of OLMo models and the injection of modular addition and comparison tasks.
• Representational Evidence and Final Takeaways: They discuss the mechanistic evidence, including Distributed Alignment Search and Fourier modes, before concluding with practical implications for data curation.