Sign up to save your podcasts
Or
Share Why Warmup the Learning Rate
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Why Warmup the Learning Rate
In this episode:
• Introduction: The Mystery of Warmup: Linda introduces a new NeurIPS 2024 paper that questions the true purpose of learning rate warmup. Professor Norris shares the conventional, yet incomplete, wisdom behind the practice.
• Tolerating Larger Learning Rates and the Sharpness Factor: The hosts discuss the paper's central claim that warmup's main benefit is allowing models to tolerate larger target learning rates by moving them to flatter regions of the loss landscape.
• Catapults and the Edge of Stability: Linda dives into the technical details of loss catapults and how progressive sharpening and natural sharpness reduction guide the network during early training stages.
• Adam, Pre-conditioned Sharpness, and Training Failures: Professor Norris and Linda explore how these mechanisms apply to adaptive optimizers like Adam, and why Adam experiences catastrophic training failures instead of standard divergence.
• GI-Adam and Better Initialization Strategies: The hosts review the paper's practical improvements, including Gradient Initialized Adam and strategies for estimating the initial learning rate to save compute time.
• Conclusion and Final Thoughts: Professor Norris concedes the brilliance of the paper's arguments, and the hosts wrap up the episode with key takeaways for deep learning practitioners.
• Introduction: The Mystery of Warmup: Linda introduces a new NeurIPS 2024 paper that questions the true purpose of learning rate warmup. Professor Norris shares the conventional, yet incomplete, wisdom behind the practice.
• Tolerating Larger Learning Rates and the Sharpness Factor: The hosts discuss the paper's central claim that warmup's main benefit is allowing models to tolerate larger target learning rates by moving them to flatter regions of the loss landscape.
• Catapults and the Edge of Stability: Linda dives into the technical details of loss catapults and how progressive sharpening and natural sharpness reduction guide the network during early training stages.
• Adam, Pre-conditioned Sharpness, and Training Failures: Professor Norris and Linda explore how these mechanisms apply to adaptive optimizers like Adam, and why Adam experiences catastrophic training failures instead of standard divergence.
• GI-Adam and Better Initialization Strategies: The hosts review the paper's practical improvements, including Gradient Initialized Adam and strategies for estimating the initial learning rate to save compute time.
• Conclusion and Final Thoughts: Professor Norris concedes the brilliance of the paper's arguments, and the hosts wrap up the episode with key takeaways for deep learning practitioners.
View all episodes
By Mechanical DirkIn this episode:
• Introduction: The Mystery of Warmup: Linda introduces a new NeurIPS 2024 paper that questions the true purpose of learning rate warmup. Professor Norris shares the conventional, yet incomplete, wisdom behind the practice.
• Tolerating Larger Learning Rates and the Sharpness Factor: The hosts discuss the paper's central claim that warmup's main benefit is allowing models to tolerate larger target learning rates by moving them to flatter regions of the loss landscape.
• Catapults and the Edge of Stability: Linda dives into the technical details of loss catapults and how progressive sharpening and natural sharpness reduction guide the network during early training stages.
• Adam, Pre-conditioned Sharpness, and Training Failures: Professor Norris and Linda explore how these mechanisms apply to adaptive optimizers like Adam, and why Adam experiences catastrophic training failures instead of standard divergence.
• GI-Adam and Better Initialization Strategies: The hosts review the paper's practical improvements, including Gradient Initialized Adam and strategies for estimating the initial learning rate to save compute time.
• Conclusion and Final Thoughts: Professor Norris concedes the brilliance of the paper's arguments, and the hosts wrap up the episode with key takeaways for deep learning practitioners.
• Introduction: The Mystery of Warmup: Linda introduces a new NeurIPS 2024 paper that questions the true purpose of learning rate warmup. Professor Norris shares the conventional, yet incomplete, wisdom behind the practice.
• Tolerating Larger Learning Rates and the Sharpness Factor: The hosts discuss the paper's central claim that warmup's main benefit is allowing models to tolerate larger target learning rates by moving them to flatter regions of the loss landscape.
• Catapults and the Edge of Stability: Linda dives into the technical details of loss catapults and how progressive sharpening and natural sharpness reduction guide the network during early training stages.
• Adam, Pre-conditioned Sharpness, and Training Failures: Professor Norris and Linda explore how these mechanisms apply to adaptive optimizers like Adam, and why Adam experiences catastrophic training failures instead of standard divergence.
• GI-Adam and Better Initialization Strategies: The hosts review the paper's practical improvements, including Gradient Initialized Adam and strategies for estimating the initial learning rate to save compute time.
• Conclusion and Final Thoughts: Professor Norris concedes the brilliance of the paper's arguments, and the hosts wrap up the episode with key takeaways for deep learning practitioners.