Sign up to save your podcasts
Or
Share Learning Rates Regulate Catastrophic Overtraining
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Learning Rates Regulate Catastrophic Overtraining
In this episode:
• Introduction to Catastrophic Overtraining: Linda and Professor Norris introduce the paper and the counterintuitive phenomenon where better pretraining leads to worse catastrophic forgetting.
• Feature Drift and Optimization Regimes: The hosts discuss how the supervised finetuning learning rate acts as an implicit regularizer, introducing the Mean Principal Angle to measure feature drift.
• Sharpness and the Edge of Stability: Linda connects the mystery of overtraining to pretraining learning rate decay, explaining how model sharpness amplifies the finetuning learning rate.
• Practical Takeaways for LLM Training: Professor Norris and Linda summarize the actionable advice from the paper, including lowering SFT learning rates and rethinking pretraining schedules.
• Introduction to Catastrophic Overtraining: Linda and Professor Norris introduce the paper and the counterintuitive phenomenon where better pretraining leads to worse catastrophic forgetting.
• Feature Drift and Optimization Regimes: The hosts discuss how the supervised finetuning learning rate acts as an implicit regularizer, introducing the Mean Principal Angle to measure feature drift.
• Sharpness and the Edge of Stability: Linda connects the mystery of overtraining to pretraining learning rate decay, explaining how model sharpness amplifies the finetuning learning rate.
• Practical Takeaways for LLM Training: Professor Norris and Linda summarize the actionable advice from the paper, including lowering SFT learning rates and rethinking pretraining schedules.
View all episodes
By Mechanical DirkIn this episode:
• Introduction to Catastrophic Overtraining: Linda and Professor Norris introduce the paper and the counterintuitive phenomenon where better pretraining leads to worse catastrophic forgetting.
• Feature Drift and Optimization Regimes: The hosts discuss how the supervised finetuning learning rate acts as an implicit regularizer, introducing the Mean Principal Angle to measure feature drift.
• Sharpness and the Edge of Stability: Linda connects the mystery of overtraining to pretraining learning rate decay, explaining how model sharpness amplifies the finetuning learning rate.
• Practical Takeaways for LLM Training: Professor Norris and Linda summarize the actionable advice from the paper, including lowering SFT learning rates and rethinking pretraining schedules.
• Introduction to Catastrophic Overtraining: Linda and Professor Norris introduce the paper and the counterintuitive phenomenon where better pretraining leads to worse catastrophic forgetting.
• Feature Drift and Optimization Regimes: The hosts discuss how the supervised finetuning learning rate acts as an implicit regularizer, introducing the Mean Principal Angle to measure feature drift.
• Sharpness and the Edge of Stability: Linda connects the mystery of overtraining to pretraining learning rate decay, explaining how model sharpness amplifies the finetuning learning rate.
• Practical Takeaways for LLM Training: Professor Norris and Linda summarize the actionable advice from the paper, including lowering SFT learning rates and rethinking pretraining schedules.