Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Efficiency and resource use scaling parity, published by Ege Erdil on August 21, 2023 on LessWrong.

An interesting pattern I've now noticed across many different domains is that if we try to do an attribution of improvements in outcomes or performance in a domain to the two categories "we're using more resources now than in the past" and "we're making more efficient use of resources now than in the past", there is usually an even split in how much improvement can be attributed to each category.

Some examples:

In computer vision, Erdil and Besiroglu (2022) (my own paper) estimates that 40% of performance improvements in computer vision from 2012 to 2022 have been due to better algortihms, and 60% due to the scaling of compute and data.

In computer chess, a similar pattern seems to hold: roughly half of the progress in chess engine performance from Deep Blue to 2015 has been from the scaling of compute, and half from better algorithms. Stockfish 8 running on consumer hardware in 1997 could achieve an Elo rating of ~ 3000, compared to ~ 2500 for contemporary hardware; and Stockfish 8 on 2015 hardware could go up to ~ 3400.

In rapidly growing economies, accounting for growth in output per worker by dividing it into capital per worker (resource scaling) and TFP (efficiency scaling, roughly speaking) often gives an even split: see Bosworth and Collins (2008) for data on China and India specifically.

More pessimistic estimates of the growth performance of China compared to official data put this split at 75% to 25% (see this post for details) but the two effects are still at least comparable.

A toy model

A speculative explanation is the following: if we imagine that performance in some domain is measured by a multiplicative index P which can be decomposed as the product of individual contributing factors F1,F2,.,Fn so that P∝∏ni=1Fi, in general we'll have

gP=1PdPdt=n∑i=11FidFidt=n∑i=1gFi

thanks to the product rule. Note that gX denotes the growth rate of the variable X.

I now want to use a law of motion from Jones (1995) for Fi: we assume they evolve over time according to

gFi=1FidFidt=θiF-βiiIλii

where θi,βi,λi>0 are parameters and Ii is a measure of "investment input" into factor i. This general specification can capture diminishing returns on investment as we make progress or scale up resources thanks to β, and can capture returns to scale to spending more resources on investment at a given time thanks to λ.

Substituting this into the growth expression for P gives

gP=1PdPdt=n∑i=1θiF-βiiIλii

Now, suppose we have a fixed budget I at any given time to allocate across all investments Ii, and our total budget I grows over time at a rate g. To maximize the rate of progress at a given time, the marginal returns to investment across all factors should be equal, i.e. we should have

∂∂Ii(1FidFidt)=∂∂Ij(1FjdFjdt)

for all pairs i,j. Substitution gives

θiλiF-βiiIλi-1i=θjλjF-βjjIλj-1j

and upon simplification, we recover

λigFiIi=λjgFiIj

In an equilibrium where all quantities grow exponentially, the ratios Ii/Ij must therefore remain constant, i.e. all of the Ii must also grow at the aggregate rate of input growth g. Then, it's easy to see that the Jones law of motion implies gFi=gλi/βi for each factor i, from which we get the important conclusion

gFi∝λiβi=ri

that must hold in an exponential growth equilibrium. The parameter ri is often called the returns to investment, so this relation says that distinct factors account for growth in P proportional to their returns to investment parameter.

How do we interpret the data in light of the toy model?

If we simplify the setup and make it about two factors, one measuring resource use and the other measuring efficiency, then the fact that the two factors account for comparable fractions in overall progress should mean that their associated retu...