Sign up to save your podcasts
Or
Share Small-scale proxies for large-scale Transformer training instabilities
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
Small-scale proxies for large-scale Transformer training instabilities
The paper investigates training instabilities in large Transformer-based models and explores ways to reproduce and study these instabilities at smaller scales. It examines sources of instability, explores the impact of learning rate and other interventions, and studies cases where instabilities can be predicted.
https://arxiv.org/abs//2309.14322
YouTube: https://www.youtube.com/@ArxivPapers
PODCASTS:
Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016
Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers
View all episodes
By Igor Melnyk
5
33 ratings
The paper investigates training instabilities in large Transformer-based models and explores ways to reproduce and study these instabilities at smaller scales. It examines sources of instability, explores the impact of learning rate and other interventions, and studies cases where instabilities can be predicted.
https://arxiv.org/abs//2309.14322
YouTube: https://www.youtube.com/@ArxivPapers
PODCASTS:
Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016
Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers
More shows like Arxiv Papers
View all
Exchanges
956 Listeners
Odd Lots
1,976 Listeners
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
438 Listeners
The Daily
112,847 Listeners
All-In with Chamath, Jason, Sacks & Friedberg
10,064 Listeners
Hard Fork
5,532 Listeners
UnHerd with Freddie Sayers
213 Listeners
Unsupervised Learning with Jacob Effron
51 Listeners
Latent Space: The AI Engineer Podcast
98 Listeners
BG2Pod with Brad Gerstner and Bill Gurley
473 Listeners