The Lottery Ticket Hypothesis
Recent research into neural networks reveals that sometimes, not all parts of the neural net are equally responsible for the performance of the network overall. Instead, it seems like (in some neural nets, at least) there are smaller subnetworks present where most of the predictive power resides. The fascinating thing is that, for some of these subnetworks (so-called “winning lottery tickets”), it’s not the training process that makes them good at their classification or regression tasks: they just happened to be initialized in a way that was very effective. This changes the way we think about what training might be doing, in a pretty fundamental way. Sometimes, instead of crafting a good fit from wholecloth, training might be finding the parts of the network that always had predictive power to begin with, and isolating and strengthening them. This research is pretty recent, having only come to prominence in the last year, but nonetheless challenges our notions about what it means to train a machine learning model.
View all episodes
4.8
350350 ratings
Recent research into neural networks reveals that sometimes, not all parts of the neural net are equally responsible for the performance of the network overall. Instead, it seems like (in some neural nets, at least) there are smaller subnetworks present where most of the predictive power resides. The fascinating thing is that, for some of these subnetworks (so-called “winning lottery tickets”), it’s not the training process that makes them good at their classification or regression tasks: they just happened to be initialized in a way that was very effective. This changes the way we think about what training might be doing, in a pretty fundamental way. Sometimes, instead of crafting a good fit from wholecloth, training might be finding the parts of the network that always had predictive power to begin with, and isolating and strengthening them. This research is pretty recent, having only come to prominence in the last year, but nonetheless challenges our notions about what it means to train a machine learning model.
More shows like Linear Digressions
View allNot So Standard Deviations
198 Listeners
My Favorite Murder with Karen Kilgariff and Georgia Hardstark
168,508 Listeners
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
444 Listeners
Super Data Science: ML & AI Podcast with Jon Krohn
285 Listeners
The Daily
110,301 Listeners
Curbside Consults
88 Listeners
2 Bears, 1 Cave with Tom Segura & Bert Kreischer
23,476 Listeners
Quantitude
191 Listeners
Gradient Dissent: Conversations on AI
64 Listeners
The Modi Raj from The Economist
48 Listeners