Sign up to save your podcasts
Or
Share EP008: RoBERTa Proves BERT Was Just Undertrained
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
EP008: RoBERTa Proves BERT Was Just Undertrained
The paper "RoBERTa: A Robustly Optimized BERT Pretraining Approach" presents a replication study of BERT which finds that the original model was significantly undertrained.
To address this, the authors introduce RoBERTa, an improved training recipe that modifies BERT in the following ways:
• Training Methodology: It utilizes dynamic masking rather than static masking, removes the next sentence prediction (NSP) objective, and trains on longer sequences.
• Scale: The model is trained for longer periods with larger mini-batches and learning rates.
• Data: It uses a significantly larger dataset totaling over 160GB of uncompressed text, including a new dataset collected by the authors called CC-NEWS.
By implementing these design choices, RoBERTa matches or exceeds the performance of all post-BERT methods published at the time, achieving state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks. The authors conclude that with these optimizations, BERT's original masked language modeling objective is competitive with newer alternatives.
View all episodes
By Yun WuThe paper "RoBERTa: A Robustly Optimized BERT Pretraining Approach" presents a replication study of BERT which finds that the original model was significantly undertrained.
To address this, the authors introduce RoBERTa, an improved training recipe that modifies BERT in the following ways:
• Training Methodology: It utilizes dynamic masking rather than static masking, removes the next sentence prediction (NSP) objective, and trains on longer sequences.
• Scale: The model is trained for longer periods with larger mini-batches and learning rates.
• Data: It uses a significantly larger dataset totaling over 160GB of uncompressed text, including a new dataset collected by the authors called CC-NEWS.
By implementing these design choices, RoBERTa matches or exceeds the performance of all post-BERT methods published at the time, achieving state-of-the-art results on the GLUE, RACE, and SQuAD benchmarks. The authors conclude that with these optimizations, BERT's original masked language modeling objective is competitive with newer alternatives.