The July 2019 paper introduces RoBERTa, a robustly optimized BERT pretraining approach, which is a refined version of the original BERT model. The authors conduct a replication study of BERT pretraining to assess the impact of various hyperparameters, finding that BERT was significantly undertrained and could be improved by simple modifications like training longer with bigger batches, removing the Next Sentence Prediction objective, and using dynamic masking. RoBERTa, built upon these changes and trained on a larger dataset including the novel CC-NEWS corpus, achieves state-of-the-art results on major natural language understanding benchmarks like GLUE, RACE, and SQuAD. The findings emphasize that design choices and training duration are highly significant and question whether recent performance gains in post-BERT models are due more to these factors than to architectural or objective changes. Source: https://arxiv.org/pdf/1907.11692