This 2014 journal article introduces "Dropout", a novel technique designed to combat overfitting in deep neural networks, which are powerful but prone to memorizing training data. The core concept involves randomly deactivating a subset of neurons and their connections during the training phase, which prevents hidden units from overly relying on each other. This process effectively trains an exponential number of "thinned" networks, improving the model's robustness and generalization to new data. The authors demonstrate that dropout significantly enhances performance across diverse applications, including image recognition, speech processing, and document classification, often achieving state-of-the-art results by producing more meaningful and sparse features. The paper also compares dropout to other regularization methods, explores its impact on network behavior, and discusses its extension to Restricted Boltzmann Machines, highlighting its general applicability as a method for model averaging.

Source:

https://arxiv.org/pdf/1207.0580