My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

What really gets AI optimists excited isn’t the prospect of automating customer service departments or human resources. Imagine, rather, what might happen to the pace of scientific progress if AI becomes a super research assistant. Tom Davidson’s new paper, How Quick and Big Would a Software Intelligence Explosion Be?, explores that very scenario.

Today on Faster, Please! — The Podcast, I talk with Davidson about what it would mean for automated AI researchers to rapidly improve their own algorithms, thus creating a self-reinforcing loop of innovation. We talk about the economic effects of self-improving AI research and how close we are to that reality.

Davidson is a senior research fellow at Forethought, where he explores AI and explosive growth. He was previously a senior research fellow at Open Philanthropy and a research scientist at the UK government’s AI Security Institute.

In This Episode

* Making human minds (1:43)

* Theory to reality (6:45)

* The world with automated research (10:59)

* Considering constraints (16:30)

* Worries and what-ifs (19:07)

Below is a lightly edited transcript of our conversation.

Making human minds (1:43)

. . . you don’t have to build any more computer chips, you don’t have to build any more fabs . . . In fact, you don’t have to do anything at all in the physical world.

Pethokoukis: A few years ago, you wrote a paper called “Could Advanced AI Drive Explosive Economic Growth?,” which argued that growth could accelerate dramatically if AI would start generating ideas the way human researchers once did. In your view, population growth historically powered kind of an ideas feedback loop. More people meant more researchers meant more ideas, rising incomes, but that loop broke after the demographic transition in the late-19th century but you suggest that AI could restart it: more ideas, more output, more AI, more ideas. Does this new paper in a way build upon that paper? “How quick and big would a software intelligence explosion be?”

The first paper you referred to is about the biggest-picture dynamic of economic growth. As you said, throughout the long run history, when we produced more food, the population increased. That additional output transferred itself into more people, more workers. These days that doesn’t happen. When GDP goes up, that doesn’t mean people have more kids. In fact, the demographic transition, the richer people get, the fewer kids they have. So now we’ve got more output, we’re getting even fewer people as a result, so that’s been blocked.

This first paper is basically saying, look, if we can manufacture human minds or human-equivalent minds in any way, be it by building more computer chips, or making better computer chips, or any way at all, then that feedback loop gets going again. Because if we can manufacture more human minds, then we can spend output again to create more workers. That’s the first paper.

The second paper double clicks on one specific way that we can use output to create more human minds. It’s actually, in a way, the scariest way because it’s the way of creating human minds which can happen the quickest. So this is the way where you don’t have to build any more computer chips, you don’t have to build any more fabs, as they’re called, these big factories that make computer chips. In fact, you don’t have to do anything at all in the physical world.

It seems like most of the conversation has been about how much investment is going to go into building how many new data centers, and that seems like that is almost the entire conversation, in a way, at the moment. But you’re not looking at compute, you’re looking at software.

Exactly, software. So the idea is you don’t have to build anything. You’ve already got loads of computer chips and you just make the algorithms that run the AIs on those computer chips more efficient. This is already happening, but it isn’t yet a big deal because AI isn’t that capable. But already, one year out, Epoch, this AI forecasting organization, estimates that just in one year, it becomes 10 times to 1000 times cheaper to run the same AI system. Just wait 12 months, and suddenly, for the same budget, you are able to run 10 times as many AI systems, or maybe even 1000 times as many for their most aggressive estimate. As I said, not a big deal today, but if we then develop an AI system which is better than any human at doing research, then now, in 10 months, you haven’t built anything, but you’ve got 10 times as many researchers that you can set to work or even more than that. So then we get this feedback loop where you make some research progress, you improve your algorithms, now you’ve got loads more researchers, you set them all to work again, finding even more algorithmic improvements. So today we’ve got maybe a few hundred people that are advancing state-of-the-art AI algorithms.

I think they’re all getting paid a billion dollars a person, too.

Exactly. But maybe we can 10x that initially by having them replaced by AI researchers that do the same thing. But then those AI researchers improve their own algorithms. Now you have 10x as many again, you have them building more computer chips, you’re just running them more efficiently, and then the cycle continues. You’re throwing more and more of these AI researchers at AI progress itself, and the algorithms are improving in what might be a very powerful feedback loop.

In this case, it seems me that you’re not necessarily talking about artificial general intelligence. This is certainly a powerful intelligence, but it’s narrow. It doesn’t have to do everything, it doesn’t have to play chess, it just has to be able to do research.

It’s certainly not fully general. You don’t need it to be able to control a robot body. You don’t need it to be able to solve the Riemann hypothesis. You don’t need it to be able to even be very persuasive or charismatic to a human. It’s not narrow, I wouldn’t say, it has to be able to do literally anything that AI researchers do, and that’s a wide range of tasks: They’re coding, they’re communicating with each other, they’re managing people, they are planning out what to work on, they are thinking about reviewing the literature. There’s a fairly wide range of stuff. It’s extremely challenging. It’s some of the hardest work in the world to do, so I wouldn’t say it’s now, but it’s not everything. It’s some kind of intermediate level of generality in between a mere chess algorithm that just does chess and the kind of AGI that can literally do anything.

Theory to reality (6:45)

I think it’s a much smaller gap for AI research than it is for many other parts of the economy.

I think people who are cautiously optimistic about AI will say something like, “Yeah, I could see the kind of intelligence you’re referring to coming about within a decade, but it’s going to take a couple of big breakthroughs to get there.” Is that true, or are we actually getting pretty close?

Famously, predicting the future of technology is very, very difficult. Just a few years before people invented the nuclear bomb, famous, very well-respected physicists were saying, “It’s impossible, this will never happen.” So my best guess is that we do need a couple of fairly non-trivial breakthroughs. So we had the start of RL training a couple of years ago, became a big deal within the language model paradigm. I think we’ll probably need another couple of breakthroughs of that kind of size.

We’re not talking a completely new approach, throw everything out, but we’re talking like, okay, we need to extend the current approach in a meaningfully different way. It’s going to take some inventiveness, it’s going to take some creativity, we’re going to have to try out a few things. I think, probably, we’ll need that to get to the researcher that can fully automate OpenAI, is a nice way of putting it — OpenAI doesn’t employ any humans anymore, they’ve just got AIs there.

There’s a difference between what a model can do on some benchmark versus becoming actually productive in the real world. That’s why, while all the benchmark stuff is interesting, the thing I pay attention to is: How are businesses beginning to use this technology? Because that’s the leap. What is that gap like, in your scenario, versus an AI model that can do a theoretical version of the lab to actually be incorporated in a real laboratory?

It’s definitely a gap. I think it’s a pretty big gap. I think it’s a much smaller gap for AI research than it is for many other parts of the economy. Let’s say we are talking about car manufacturing and you’re trying to get an AI to do everything that happens there. Man, it’s such a messy process. There’s a million different parts of the supply chain. There’s all this tacit knowledge and all the human workers’ minds. It’s going to be really tough. There’s going to be a very big gap going from those benchmarks to actually fully automating the supply chain for cars.

For automating what OpenAI does, there’s still a gap, but it’s much smaller, because firstly, all of the work is virtual. Everyone at OpenAI could, in principle, work remotely. Their top research scientists, they’re just on a computer all day. They’re not picking up bricks and doing stuff like that. So also that already means it’s a lot less messy. You get a lot less of that kind of messy world reality stuff slowing down adoption. And also, a lot of it is coding, and coding is almost uniquely clean in that, for many coding tasks, you can define clearly defined metrics for success, and so that makes AI much better. You can just have a go. Did AI succeed in the test? If not, try something else or do a gradient set update.

That said, there’s still a lot of messiness here, as any coder will know, when you’re writing good code, it’s not just about whether it does the function that you’ve asked it to do, it needs to be well-designed, it needs to be modular, it needs to be maintainable. These things are much harder to evaluate, and so AIs often pass our benchmarks because they can do the function that you asked it to do, the code runs, but they kind of write really spaghetti code — code that no one wants to look at, that no one can understand, and so no company would want to use that.

So there’s still going to be a pretty big benchmark-to-reality gap, even for OpenAI, and I think that’s one of the big uncertainties in terms of, will this happen in three years versus will this happen in 10 years, or even 15 years?

Since you brought up the timeline, what’s your guess? I didn’t know whether to open with that question or conclude with that question — we’ll stick it right in the middle of our chat.

Great. Honestly, my best guess about this does change more often than I would like it to, which I think tells us, look, there’s still a state of flux. This is just really something that’s very hard to know about. Predicting the future is hard. My current best guess is it’s about even odds that we’re able to fully automate OpenAI within the next 10 years. So maybe that’s a 50-50.

The world with AI research automation (10:59)

. . . I’m talking about 30 percent growth every year. I think it gets faster than that. If you want to know how fast it eventually gets, you can think about the question of how fast can a kind of self-replicating system double itself?

So then what really would be the impact of that kind of AI research automation? How would you go about quantifying that kind of acceleration? What does the world look like?

Yeah, so many possibilities, but I think what strikes me is that there is a plausible world where it is just way, way faster than almost everyone is expecting it to be. So that’s the world where you fully automate OpenAI, and then we get that feedback loop that I was talking about earlier where AIs make their algorithms way more efficient, now you’ve got way more of them, then they make their algorithms way more efficient again, now they’re way smarter. Now they’re thinking a hundred times faster. The feedback loop continues and maybe within six months you now have a billion superintelligent AIs running on this OpenAI data center. The combined cognitive abilities of all these AIs outstrips the whole of the United States, outstrips anything we’ve seen from any kind of company or entity before, and they can all potentially be put towards any goal that OpenAI wants to. And then there’s, of course, the risk that OpenAI’s lost control of these systems, often discussed, in which case these systems could all be working together to pursue a particular goal. And so what we’re talking about here is really a huge amount of power. It’s a threat to national security for any government in which this happens, potentially. It is a threat to everyone if we lose control of these systems, or if the company that develops them uses them for some kind of malicious end. And, in terms of economic impacts, I personally think that that again could happen much more quickly than people think, and we can get into that.

In the first paper we mentioned, it was kind of a thought experiment, but you were really talking about moving the decimal point in GDP growth, instead of talking about two and three percent, 20 and 30 percent. Is that the kind of world we’re talking about?

I speak to economists a lot, and —

They hate those kinds of predictions, by the way.

Obviously, they think I’m crazy. Not all of them. There are economists that take it very seriously. I think it’s taken more seriously than everyone else realizes. It’s like it’s a bit embarrassing, at the moment, to admit that you take it seriously, but there are a few really senior economists who absolutely know their stuff. They’re like, “Yep, this checks out. I think that’s what’s going to happen.” And I’ve had conversation with them where they’re like, “Yeah, I think this is going to happen.” But the really loud, dominant view where I think people are a little bit scared to speak out against is they’re like, “Obviously this is sci-fi.”

One analogy I like to give to people who are very, very confident that this is all sci-fi and it’s rubbish is to imagine that we were sitting there in the year 1400, imagine we had an economics professor who’d been studying the rate of economic growth, and they’ve been like, “Yeah, we’ve always had 0.1 percent growth every single year throughout history. We’ve never seen anything higher.” And then there was some kind of futurist economist rogue that said, “Actually, I think that if I extrapolate the curves in this way and we get this kind of technology, maybe we could have one percent growth.” And then all the other economists laugh at them, tell them they’re insane – that’s what happened. In 1400, we’d never had growth that was at all fast, and then a few hundred years later, we developed industrial technology, we started that feedback loop, we were investing more and more resources in scientific progress and in physical capital, and we did see much faster growth.

So I think it can be useful to try and challenge economists and say, “Okay, I know it sounds crazy, but history was crazy. This crazy thing happened where growth just got way, way faster. No one would’ve predicted it. You would not have predicted it.” And I think being in that mindset can encourage people to be like, “Yeah, okay. You know what? Maybe if we do get AI that’s really that powerful, it can really do everything, and maybe it is possible.”

But to answer your question, yeah, I’m talking about 30 percent growth every year. I think it gets faster than that. If you want to know how fast it eventually gets, you can think about the question of how fast can a kind of self-replicating system double itself? So ultimately, what the economy is going to be like is it’s going to have robots and factories that are able to fully create new versions of themselves. Everything you need: the roads, the electricity, the robots, the buildings, all of that will be replicated. And so you can look at actually biology and say, do we have any examples of systems which fully replicate themselves? How long does it take? And if you look at rats, for example, they’re able to double the number of rats by grabbing resources from the environment, and giving birth, and whatnot. The doubling time is about six weeks for some types of rats. So that’s an example of here’s a physical system — ultimately, everything’s made of physics — a physical system that has some intelligence that’s able to go out into the world, gather resources, replicate itself. The doubling time is six weeks.

Now, who knows how long it’ll take us to get to AI that’s that good? But when we do, you could see the whole physical economy, maybe a part that humans aren’t involved with, a whole automated city without any humans just doubling itself every few weeks. If that happens, and the amount of stuff we’re able to reduce as a civilization is doubling again on the order of weeks. And, in fact, there are some animals that double faster still, in days, but that’s the kind of level of craziness. Now we’re talking about 1000 percent growth, at that point. We don’t know how crazy it could get, but I think we should take even the really crazy possibilities, we shouldn’t fully rule them out.

Considering constraints (16:30)

I really hope people work less. If we get this good future, and the benefits are shared between all . . . no one should work. But that doesn’t stop growth . . .

There’s this great AI forecast chart put out by the Federal Reserve Bank of Dallas, and I think its main forecast — the one most economists would probably agree with — has a line showing AI improving GDP by maybe two tenths of a percent. And then there are two other lines: one is more or less straight up, and the other one is straight down, because in the first, AI created a utopia, and in the second, AI gets out of control and starts killing us, and whatever. So those are your three possibilities.

If we stick with the optimistic case for a moment, what constraints do you see as most plausible — reduced labor supply from rising incomes, social pushback against disruption, energy limits, or something else?

Briefly, the ones you’ve mentioned, people not working, 100 percent. I really hope people work less. If we get this good future, and the benefits are shared between all — which isn’t guaranteed — if we get that, then yeah, no one should work. But that doesn’t stop growth, because when AI and robots can do everything that humans do, you don’t need humans in the loop anymore. That whole thing is just going and kind of self-replicating itself and making as many goods as services as we want. Sure, if you want your clothes to be knitted by a human, you’re in trouble, then your consumption is stuck. Bad luck. If you’re happy to consume goods and services produced by AI systems or robots, fine if no one wants to work.

Pushback: I think, for me, this is the biggest one. Obviously, the economy doubling every year is very scary as a thought. Tech progress will be going much faster. Imagine if you woke up and, over the course of the year, you go from not having any telephones at all in the world, to everyone’s on their smartphones and social media and all the apps. That’s a transition that took decades. If that happened in a year, that would be very disconcerting.

Another example is the development of nuclear weapons. Nuclear weapons were developed over a number of years. If that happened in a month, or two months, that could be very dangerous. There’d be much less time for different countries, different actors to figure out how they’re going to handle it. So I think pushback is the strongest one that we might as a society choose, “Actually, this is insane. We’re going to go slower than we could.” That requires, potentially, coordination, but I think there would be broad support for some degree of coordination there.

Worries and what-ifs (19:07)

If suddenly no one has any jobs, what will we want to do with ourselves? That’s a very, very consequential transition for the nature of human society.

I imagine you certainly talk with people who are extremely gung-ho about this prospect. What is the common response you get from people who are less enthusiastic? Do they worry about a future with no jobs? Maybe they do worry about the existential kinds of issues. What’s your response to those people? And how much do you worry about those things?

I think there are loads of very worrying things that we’re going to be facing. One class of pushback, which I think is very common, is worries about employment. It’s a source of income for all of us, employment, but also, it’s a source of pride, it’s a source of meaning. If suddenly no one has any jobs, what will we want to do with ourselves? That’s a very, very consequential transition for the nature of human society. I think people aren’t just going to be down to just do it. I think people are scared about three AI companies literally now taking all the revenues that all of humanity used to be earning. It is naturally a very scary prospect. So that’s one kind of pushback, and I’m sympathetic with it.

I think that there are solutions, if we find a way to tax AI systems, which isn’t necessarily easy, because it’s very easy to move physical assets between countries. It’s a lot easier to tax labor than capital already when rich people can move their assets around. We’re going to have the same problem with AI, but if we can find a way to tax it, and we maintain a good democratic country, and we can just redistribute the wealth broadly, it can be solved. So I think it’s a big problem, but it is doable.

Then there’s the problem of some people want to stop this now because they’re worried about AI killing everyone. Their literally worry is that everyone will be dead because superintelligent AI will want that to happen. I think there’s a real risk there. It’s definitely above one percent, in my opinion. I wouldn’t go above 10 percent, myself, but I think it’s very scary, and that’s a great reason to slow things down. I personally don’t want to stop quite yet. I think you want to stop when the AI is a bit more powerful and a bit more useful than it is today so it can kind of help us figure out what to do about all of this crazy stuff that’s coming.

On what side of that line is AI as an AI researcher?

That’s a really great question. Should we stop? I think it’s very hard to stop just after you’ve got the AI researcher AI, because that’s when it’s suddenly really easy to go very, very fast. So my out-of-the-box proposal here, which is probably very flawed, would be: When we’re within a few spits distance — not spitting distance, but if you did that three times, and we can see we’re almost at that AI automating OpenAI — then you pause, because you’re not going to accidentally then go all the way. It is actually still a little bit a fair distance away, but it’s actually still, at that point, probably a very powerful AI that can really help.

Then you pause and do what?

Great question. So then you pause, and you use your AI systems to help you firstly solve the problem of AI alignment, make extra, double sure that every time we increase the notch of AI capabilities, the AI is still loyal to humanity, not to its own kind of secret goals.

Secondly, you solve the problem of, how are we going to make sure that no one person in government or no one CEO of an AI company ensures that this whole AI army is loyal to them, personally? How are we going to ensure that everyone, the whole world gets influenced over what this AI is ultimately programmed to do? That’s the second problem.

And then there’s just a whole host of other things: unemployment that we’ve talked about, competition between different countries, US and China, there’s a whole host of other things that I think you want to research on, figure out, get consensus on, and then slowly ratchet up the capabilities in what is now a very safe and controlled way.

What else should we be working on? What are you working on next?

One problem I’m excited about is people have historically worried about AI having its own goals. We need to make it loyal to humanity. But as we’ve got closer, it’s become increasingly obvious, “loyalty to humanity” is very vague. What specifically do you want the AI to be programmed to do? I mean, it’s not programmed, it’s grown, but if it were programmed, if you’re writing a rule book for AI, some organizations have employee handbooks: Here’s the philosophy of the organization, here’s how you should behave. Imagine you’re doing that for the AI, but you’re going super detailed, exactly how you want your AI assistant to behave in all kinds of situations. What should that be? Essentially, what should we align the AI to? Not any individual person, probably following the law, probably loads of other things. I think basically designing what is the character of this AI system is a really exciting question, and if we get that right, maybe the AI can then help us solve all these other problems.

Maybe you have no interest in science fiction, but is there any film, TV, book that you think is useful for someone in your position to be aware of, or that you find useful in any way? Just wondering.

I think there’s this great post called “AI 2027,” which lays out a concrete scenario for how AI could go wrong or how maybe it could go right. I would recommend that. I think that’s the only thing that’s coming top of mind. I often read a lot of the stuff I read is I read a lot of LessWrong, to be honest. There’s a lot of stuff from there that I don’t love, but a lot of new ideas, interesting content there.

Any fiction?

I mean, I read fiction, but honestly, I don’t really love the AI fiction that I’ve read because often it’s quite unrealistic, and so I kind of get a bit overly nitpicky about it. But I mean, yeah, there’s this book called Harry Potter and the Methods of Rationality, which I read maybe 10 years ago, which I thought was pretty fun.

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

Faster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit fasterplease.substack.com/subscribe