Sign up to save your podcasts
Or
Share One Learning Rate Doesn't Fit All: Layerwise Spectral Scheduling for Transformers
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
One Learning Rate Doesn't Fit All: Layerwise Spectral Scheduling for Transformers
Shows that modern transformers are highly heterogeneous across layers and proposes layerwise learning rates based on weight spectrum shape, yielding up to 1.5× training speedup on LLaMA/GPT-style models.
View all episodes
By Shaoqing TanShows that modern transformers are highly heterogeneous across layers and proposes layerwise learning rates based on weight spectrum shape, yielding up to 1.5× training speedup on LLaMA/GPT-style models.