The paper, "Training Compute-Optimal Large Language Models" by DeepMind, investigates how to optimally balance the size of a Large Language Model (LLM) and its amount of training data within a fixed computational budget.

Here is a short summary of its core contributions:

• Equal Scaling of Size and Data: Contrary to previous widely accepted research (like Kaplan et al., 2020) which suggested that model size should increase significantly faster than training data, this study finds that model size and the number of training tokens should be scaled in equal proportions.
• Current Models are Under-trained: The authors conclude that many recent mega-models are significantly under-trained because the field has historically focused on scaling parameter size while keeping the amount of training data relatively constant (typically around 300 billion tokens).
• Extensive Testing Methodology: These conclusions were reached by training over 400 language models—ranging from 70 million to over 16 billion parameters—and analyzing their optimal loss using three different predictive approaches to establish a "compute-optimal frontier".
• Introducing Chinchilla: To validate their findings, the researchers trained Chinchilla, a 70-billion parameter model trained on 1.4 trillion tokens. Even though it used the exact same compute budget as the much larger 280-billion parameter Gopher model, Chinchilla significantly outperformed Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG across a vast array of downstream evaluations (including MMLU, BIG-bench, reading comprehension, and common sense reasoning).
• Efficiency Benefits: In addition to achieving state-of-the-art accuracy, Chinchilla's 4x smaller parameter footprint means that it uses substantially less compute and memory for downstream fine-tuning and inference, making it far more practical for everyday use.