This 2000 paper, titled "Scaling Laws for Neural Language Models," explores the empirical relationships between the performance of neural language models (specifically Transformers) and various scaling factors: model size (parameters), dataset size (tokens), and computational budget (compute used for training). The authors demonstrate that model performance follows predictable power-law scalings across a wide range, often spanning multiple orders of magnitude. A key finding is that larger models are more sample-efficient, meaning they can achieve similar performance with less data and fewer training steps, suggesting that optimal compute-efficient training involves very large models that are stopped before full convergence. The research also notes that architectural details beyond these core scaling factors have minimal impact on performance.