Sign up to save your podcasts
Or
Share Cautious Optimizers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Cautious Optimizers
In this episode:
• Introduction to Cautious Optimizers: Linda introduces the paper and its bold claim of improving optimizers with just one line of code, while Norris expresses his initial skepticism.
• The Inertia Problem in Momentum: The hosts discuss how standard momentum-based optimizers like AdamW can overshoot due to inertia, temporarily increasing the loss function.
• The One-Line Fix and Scaling: Linda breaks down the PyTorch implementation of the cautious mask, explaining how it zeros out conflicting directions and scales the remaining updates.
• Hamiltonian Dynamics and Convergence: Norris and Linda explore the theoretical guarantees of the paper, discussing how the method preserves Hamiltonian descent and ensures monotonic loss reduction.
• Empirical Triumphs and Overhead: The conversation shifts to the experimental results on LLaMA pretraining and Vision Transformers, noting the impressive performance and minimal 3 percent computational overhead.
• Conclusion: Norris admits he is fully convinced by the elegant simplicity of the paper, and Linda signs off for the episode.
• Introduction to Cautious Optimizers: Linda introduces the paper and its bold claim of improving optimizers with just one line of code, while Norris expresses his initial skepticism.
• The Inertia Problem in Momentum: The hosts discuss how standard momentum-based optimizers like AdamW can overshoot due to inertia, temporarily increasing the loss function.
• The One-Line Fix and Scaling: Linda breaks down the PyTorch implementation of the cautious mask, explaining how it zeros out conflicting directions and scales the remaining updates.
• Hamiltonian Dynamics and Convergence: Norris and Linda explore the theoretical guarantees of the paper, discussing how the method preserves Hamiltonian descent and ensures monotonic loss reduction.
• Empirical Triumphs and Overhead: The conversation shifts to the experimental results on LLaMA pretraining and Vision Transformers, noting the impressive performance and minimal 3 percent computational overhead.
• Conclusion: Norris admits he is fully convinced by the elegant simplicity of the paper, and Linda signs off for the episode.
View all episodes
By Mechanical DirkIn this episode:
• Introduction to Cautious Optimizers: Linda introduces the paper and its bold claim of improving optimizers with just one line of code, while Norris expresses his initial skepticism.
• The Inertia Problem in Momentum: The hosts discuss how standard momentum-based optimizers like AdamW can overshoot due to inertia, temporarily increasing the loss function.
• The One-Line Fix and Scaling: Linda breaks down the PyTorch implementation of the cautious mask, explaining how it zeros out conflicting directions and scales the remaining updates.
• Hamiltonian Dynamics and Convergence: Norris and Linda explore the theoretical guarantees of the paper, discussing how the method preserves Hamiltonian descent and ensures monotonic loss reduction.
• Empirical Triumphs and Overhead: The conversation shifts to the experimental results on LLaMA pretraining and Vision Transformers, noting the impressive performance and minimal 3 percent computational overhead.
• Conclusion: Norris admits he is fully convinced by the elegant simplicity of the paper, and Linda signs off for the episode.
• Introduction to Cautious Optimizers: Linda introduces the paper and its bold claim of improving optimizers with just one line of code, while Norris expresses his initial skepticism.
• The Inertia Problem in Momentum: The hosts discuss how standard momentum-based optimizers like AdamW can overshoot due to inertia, temporarily increasing the loss function.
• The One-Line Fix and Scaling: Linda breaks down the PyTorch implementation of the cautious mask, explaining how it zeros out conflicting directions and scales the remaining updates.
• Hamiltonian Dynamics and Convergence: Norris and Linda explore the theoretical guarantees of the paper, discussing how the method preserves Hamiltonian descent and ensures monotonic loss reduction.
• Empirical Triumphs and Overhead: The conversation shifts to the experimental results on LLaMA pretraining and Vision Transformers, noting the impressive performance and minimal 3 percent computational overhead.
• Conclusion: Norris admits he is fully convinced by the elegant simplicity of the paper, and Linda signs off for the episode.