Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Report on Frontier Model Training, published by YafahEdelman on August 31, 2023 on LessWrong.

Understanding what drives the rising capabilities of AI is important for those who work to forecast, regulate, or ensure the safety of AI. Regulations on the export of powerful GPUs need to be informed by understanding of how these GPUs are used, forecasts need to be informed by bottlenecks, and safety needs to be informed by an understanding of how the models of the future might be trained. A clearer understanding would enable policy makers to target regulations in such a way that they are difficult for companies to circumvent with only technically compliant GPUs, forecasters to avoid focus on unreliable metrics, and technical research working on mitigating the downsides of AI to understand what data models might be trained on.

This doc is built from a collection of smaller docs I wrote on a bunch of different aspects of frontier model training I consider important. I hope for people to be able to use this document as a collection of resources, to draw from it the information they find important and inform their own models.

I do not expect this doc to have a substantial impact on any serious AI labs capabilities efforts - I think my conclusions are largely discoverable in the process of attempting to scale AIs or for substantially less money than a serious such attempt would cost. Additionally I expect major labs already know many of the things in this report.

Acknowledgements

I'd like to thank the following people for their feedback, advice, and discussion:

James Bradbury, Software Engineer, Google DeepMind

Benjamin Edelman, Ph.D. Candidate, Harvard University

Horace He, Software Engineer, PyTorch/Meta

Lukas Finnveden, Research Analyst, Open Philanthropy Project

Joanna Morningstar, Chief Scientific Officer, Nanotronics

Keller Scholl, Ph.D. Candidate, Pardee RAND Graduate School

Jaime Sevilla, Director, Epoch

Cody Wild, Research Engineer, Google

Index

Cost Breakdown of ML Training

Estimates the costs of training a frontier (state of the art) model, drawing on leaks and analysis. Power usage is a small portion of the cost, GPUs are likely a slim majority.

Why ML GPUs Cost So Much

ML GPUs are expensive largely because of their communication and memory capabilities - not because of their processing power. NVIDIA's best gaming GPU provides greater ML processing power than the GPU used to train GPT-4, for only a tenth the price. Note that NVIDIA's near monopoly plausibly explains some of the price differential.

Contra FLOPs

Argues that the most common metric of ML computing power - floating point operations - is flawed, due to the rise of different types of floating point numbers making standardization difficult and the cost of processing power representing a small portion of the cost of ML.

ML Parallelism

An overview of ML parallelism techniques, showing how the common notion that "ML is embarrassingly parallel" is simplistic and breaks down at large scales - where any simple method of parallelizing a model starts to hit bottlenecks as the capabilities of individual devices become bottlenecks regardless of the number of devices involved.

We (Probably) Won't Run Out of Data

There are many routes toward preventing data from becoming a major bottleneck to ML scaling, though it's not certain any of them enable scaling as fast as has occurred historically.

AI Energy Use and Heat Signatures

ML energy usage may become important in the near future, even if it's a relatively minor concern for frontier model training right now. If current trends continue, energy usage could limit scaling, determine major engineering challenges, and provide a novel approach to surveillance of training runs using satellites and multispectral photography.

Cost Breakdown of ML Training

This section is an att...