My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

Headlines portend rising seas, raging storms, and a planet in crisis. It’s easy to feel like the future is something to fear; however, the key to cooling things down isn’t scaling civilization back. If the world wants to cut back on carbon emissions without sacrificing growth, the answer lies in bold innovation. A sustainable tomorrow requires smart energy investment and long-term thinking today.

On this episode of Faster, Please! — The Podcast, I chat with Roger Pielke Jr. about the ever-evolving discussion around climate change. We talk about the benefits of embracing new energy technology and identifying some easy wins.

Pielke is a senior fellow at the American Enterprise Institute where his research focuses on science and technology policy. He is also a professor emeritus at University of Colorado Boulder, a distinguished fellow at Japan’s Institute of Energy Economics, a research associate with Risk Frontiers in Australia, and an honorary professor at University College London. Pielke has authored and edited several books, including The Climate Fix: What Scientists and Politicians Won’t Tell You About Global Warming. He also writes The Honest Broker Substack.

In This Episode

* The Shale Story (1:42)

* Unknown Unknowns (7:42)

* The Weather Forecast (14:19)

* Alternate History (25:23)

* The Path Forward (28:25)

(A lightly edited transcript of our conversation will be appear in my Week in Review issue on Saturday. Another option is using the Substack auto transcript function.)

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

Since humanity’s beginning, we’ve been pondering about our end. From war, to disease, to divine reckoning, the means of our destruction seem endless. The advent of the atomic bomb, concerns around climate change, and now AI have prompted many to wonder whether our demise will be random, or if it will come as the result of our own actions.

Today on Faster, Please! — The Podcast, I chat with Dorian Lynskey about the way we talk about the end times. We discuss whether catastrophizing leads to action or paralysis and the role of hope in our narratives.

Lynskey is a prolific journalist and the author of three books. His most recent: Everything Must Go: The Stories We Tell About the End of the World, which was released last month in the US. He also co-hosts two podcasts, Origin Story and Oh God, What Now?.

In This Episode

* Scare Tactics (1:32)

* Effects of Hopefulness (10:25)

* AI Doomsayers (17:01)

* Countdown to Catastrophe (21:15)

(A lightly edited transcript of our conversation will be appear in my Week in Review issue on Saturday. Another option is using the Substack auto transcript function.)

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

The human pursuit of progress stems from our desire for security and a higher quality of life. Yet, even as today’s advanced economies are the richest and most comfortable they’ve ever been, something is amiss. What explains the decline in R&D growth, mental health, and birth rates, just to name a few challenges?

In his new book, The Permanent Problem: The Uncertain Transition from Mass Plenty to Mass Flourishing, author Brink Lindsey identifies the critical gap between material abundance and abundant human flourishing.

Today on Faster, Please! — The Podcast, Brink and I chat about what constitutes a truly healthy society, beyond surface-level affluence. We identify the conditions for continual progress after our basic needs have been met and far exceeded.

Linsey is a senior vice president at the Niskanen Center. He previously served as vice president for research at the Cato Institute and as a senior scholar at the Kauffman Foundation. He has authored and co-authored six books on economics and culture, and is the author of his own Substack, also titled The Permanent Problem.

In This Episode

* More of everything . . . !? (1:54)

* Falling fertility (7:31)

* What we’ve lost (10:20)

* Evaluating flourishing (13:13)

* A culture of growth (20:24)

* Future-world problems (28:04)

(A lightly edited transcript of our conversation will be appear in my Week in Review issue on Saturday. Another option is using the Substack auto transcript function.)

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

Headlines abound with news of the coming nuclear renaissance — a long-awaited era of clean, abundant energy to power our future. But this is hardly the first time the media has heralded the dawn of the atomic age. Still, this round of nuclear optimism is seeing unprecedented corporate investment, more cost-effective modular reactors, and a greater sense of political consensus.

Today on Faster, Please! — The Podcast, I chat with Jessica Lovering about past obstacles to growth, and what we might expect from the US going forward.

Lovering is an advocate for nuclear power currently based in Sweden. She is the co-founder and former executive director of the Good Energy Collective, as well as a senior fellow with the Nuclear Innovation Alliance and the Energy for Growth Hub. She also authors her own Substack, Nuclear Power to the People.

In This Episode

* The lost Atomic Age (1:30)

* To regulate or not to regulate (8:26)

* Reactor capacity past and future (10:44)

* The economics of nuclear (14:51)

* Power projection (18:32)

* The new nuclear status quo (24:04)

(A lightly edited transcript of our conversation will be appear in my Week in Review issue on Saturday. Another option is using the Substack auto transcript function.)

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

NASA is attempting the difficult task of juggling highly ambitious goals, but also possibly intense budget cuts. Despite personnel losses and unclear leadership, the administration is racing to put humans on the Moon — ideally ahead of China — and then Mars.

Today on Faster, Please! — The Podcast, I’m chatting with Casey Dreier about this complicated new era in NASA’s history. We’ll discuss whether or not we’re really in a space race, what to make of the differing visions of Elon Musk and Jeff Bezos, and the rise of planetary defense.

Dreier is chief of space policy at The Planetary Society where he advocates for planetary exploration, defense, and the search for extraterrestrial life. He has been featured in major publications from The New York Times to the Washington Post, and hosts his own podcast, Planetary Radio: Space Policy Edition.

In This Episode

* The return of Isaacman (1:32)

* Ditch the Space Race (7:42)

* Visions of space (14:48)

* Planetary defense (21:23)

* Proceed with optimism (24:51)

(A lightly edited transcript of our conversation will be appear in my Week in Review issue on Saturday. Another option is using the Substack auto transcript function.)

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

My fellow pro-growth/progress/abundance Up Wingers in America and around the world:

What really gets AI optimists excited isn’t the prospect of automating customer service departments or human resources. Imagine, rather, what might happen to the pace of scientific progress if AI becomes a super research assistant. Tom Davidson’s new paper, How Quick and Big Would a Software Intelligence Explosion Be?, explores that very scenario.

Today on Faster, Please! — The Podcast, I talk with Davidson about what it would mean for automated AI researchers to rapidly improve their own algorithms, thus creating a self-reinforcing loop of innovation. We talk about the economic effects of self-improving AI research and how close we are to that reality.

Davidson is a senior research fellow at Forethought, where he explores AI and explosive growth. He was previously a senior research fellow at Open Philanthropy and a research scientist at the UK government’s AI Security Institute.

In This Episode

* Making human minds (1:43)

* Theory to reality (6:45)

* The world with automated research (10:59)

* Considering constraints (16:30)

* Worries and what-ifs (19:07)

Below is a lightly edited transcript of our conversation.

Making human minds (1:43)

. . . you don’t have to build any more computer chips, you don’t have to build any more fabs . . . In fact, you don’t have to do anything at all in the physical world.

Pethokoukis: A few years ago, you wrote a paper called “Could Advanced AI Drive Explosive Economic Growth?,” which argued that growth could accelerate dramatically if AI would start generating ideas the way human researchers once did. In your view, population growth historically powered kind of an ideas feedback loop. More people meant more researchers meant more ideas, rising incomes, but that loop broke after the demographic transition in the late-19th century but you suggest that AI could restart it: more ideas, more output, more AI, more ideas. Does this new paper in a way build upon that paper? “How quick and big would a software intelligence explosion be?”

The first paper you referred to is about the biggest-picture dynamic of economic growth. As you said, throughout the long run history, when we produced more food, the population increased. That additional output transferred itself into more people, more workers. These days that doesn’t happen. When GDP goes up, that doesn’t mean people have more kids. In fact, the demographic transition, the richer people get, the fewer kids they have. So now we’ve got more output, we’re getting even fewer people as a result, so that’s been blocked.

This first paper is basically saying, look, if we can manufacture human minds or human-equivalent minds in any way, be it by building more computer chips, or making better computer chips, or any way at all, then that feedback loop gets going again. Because if we can manufacture more human minds, then we can spend output again to create more workers. That’s the first paper.

The second paper double clicks on one specific way that we can use output to create more human minds. It’s actually, in a way, the scariest way because it’s the way of creating human minds which can happen the quickest. So this is the way where you don’t have to build any more computer chips, you don’t have to build any more fabs, as they’re called, these big factories that make computer chips. In fact, you don’t have to do anything at all in the physical world.

It seems like most of the conversation has been about how much investment is going to go into building how many new data centers, and that seems like that is almost the entire conversation, in a way, at the moment. But you’re not looking at compute, you’re looking at software.

Exactly, software. So the idea is you don’t have to build anything. You’ve already got loads of computer chips and you just make the algorithms that run the AIs on those computer chips more efficient. This is already happening, but it isn’t yet a big deal because AI isn’t that capable. But already, one year out, Epoch, this AI forecasting organization, estimates that just in one year, it becomes 10 times to 1000 times cheaper to run the same AI system. Just wait 12 months, and suddenly, for the same budget, you are able to run 10 times as many AI systems, or maybe even 1000 times as many for their most aggressive estimate. As I said, not a big deal today, but if we then develop an AI system which is better than any human at doing research, then now, in 10 months, you haven’t built anything, but you’ve got 10 times as many researchers that you can set to work or even more than that. So then we get this feedback loop where you make some research progress, you improve your algorithms, now you’ve got loads more researchers, you set them all to work again, finding even more algorithmic improvements. So today we’ve got maybe a few hundred people that are advancing state-of-the-art AI algorithms.

I think they’re all getting paid a billion dollars a person, too.

Exactly. But maybe we can 10x that initially by having them replaced by AI researchers that do the same thing. But then those AI researchers improve their own algorithms. Now you have 10x as many again, you have them building more computer chips, you’re just running them more efficiently, and then the cycle continues. You’re throwing more and more of these AI researchers at AI progress itself, and the algorithms are improving in what might be a very powerful feedback loop.

In this case, it seems me that you’re not necessarily talking about artificial general intelligence. This is certainly a powerful intelligence, but it’s narrow. It doesn’t have to do everything, it doesn’t have to play chess, it just has to be able to do research.

It’s certainly not fully general. You don’t need it to be able to control a robot body. You don’t need it to be able to solve the Riemann hypothesis. You don’t need it to be able to even be very persuasive or charismatic to a human. It’s not narrow, I wouldn’t say, it has to be able to do literally anything that AI researchers do, and that’s a wide range of tasks: They’re coding, they’re communicating with each other, they’re managing people, they are planning out what to work on, they are thinking about reviewing the literature. There’s a fairly wide range of stuff. It’s extremely challenging. It’s some of the hardest work in the world to do, so I wouldn’t say it’s now, but it’s not everything. It’s some kind of intermediate level of generality in between a mere chess algorithm that just does chess and the kind of AGI that can literally do anything.

Theory to reality (6:45)

I think it’s a much smaller gap for AI research than it is for many other parts of the economy.

I think people who are cautiously optimistic about AI will say something like, “Yeah, I could see the kind of intelligence you’re referring to coming about within a decade, but it’s going to take a couple of big breakthroughs to get there.” Is that true, or are we actually getting pretty close?

Famously, predicting the future of technology is very, very difficult. Just a few years before people invented the nuclear bomb, famous, very well-respected physicists were saying, “It’s impossible, this will never happen.” So my best guess is that we do need a couple of fairly non-trivial breakthroughs. So we had the start of RL training a couple of years ago, became a big deal within the language model paradigm. I think we’ll probably need another couple of breakthroughs of that kind of size.

We’re not talking a completely new approach, throw everything out, but we’re talking like, okay, we need to extend the current approach in a meaningfully different way. It’s going to take some inventiveness, it’s going to take some creativity, we’re going to have to try out a few things. I think, probably, we’ll need that to get to the researcher that can fully automate OpenAI, is a nice way of putting it — OpenAI doesn’t employ any humans anymore, they’ve just got AIs there.

There’s a difference between what a model can do on some benchmark versus becoming actually productive in the real world. That’s why, while all the benchmark stuff is interesting, the thing I pay attention to is: How are businesses beginning to use this technology? Because that’s the leap. What is that gap like, in your scenario, versus an AI model that can do a theoretical version of the lab to actually be incorporated in a real laboratory?

It’s definitely a gap. I think it’s a pretty big gap. I think it’s a much smaller gap for AI research than it is for many other parts of the economy. Let’s say we are talking about car manufacturing and you’re trying to get an AI to do everything that happens there. Man, it’s such a messy process. There’s a million different parts of the supply chain. There’s all this tacit knowledge and all the human workers’ minds. It’s going to be really tough. There’s going to be a very big gap going from those benchmarks to actually fully automating the supply chain for cars.

For automating what OpenAI does, there’s still a gap, but it’s much smaller, because firstly, all of the work is virtual. Everyone at OpenAI could, in principle, work remotely. Their top research scientists, they’re just on a computer all day. They’re not picking up bricks and doing stuff like that. So also that already means it’s a lot less messy. You get a lot less of that kind of messy world reality stuff slowing down adoption. And also, a lot of it is coding, and coding is almost uniquely clean in that, for many coding tasks, you can define clearly defined metrics for success, and so that makes AI much better. You can just have a go. Did AI succeed in the test? If not, try something else or do a gradient set update.

That said, there’s still a lot of messiness here, as any coder will know, when you’re writing good code, it’s not just about whether it does the function that you’ve asked it to do, it needs to be well-designed, it needs to be modular, it needs to be maintainable. These things are much harder to evaluate, and so AIs often pass our benchmarks because they can do the function that you asked it to do, the code runs, but they kind of write really spaghetti code — code that no one wants to look at, that no one can understand, and so no company would want to use that.

So there’s still going to be a pretty big benchmark-to-reality gap, even for OpenAI, and I think that’s one of the big uncertainties in terms of, will this happen in three years versus will this happen in 10 years, or even 15 years?

Since you brought up the timeline, what’s your guess? I didn’t know whether to open with that question or conclude with that question — we’ll stick it right in the middle of our chat.

Great. Honestly, my best guess about this does change more often than I would like it to, which I think tells us, look, there’s still a state of flux. This is just really something that’s very hard to know about. Predicting the future is hard. My current best guess is it’s about even odds that we’re able to fully automate OpenAI within the next 10 years. So maybe that’s a 50-50.

The world with AI research automation (10:59)

. . . I’m talking about 30 percent growth every year. I think it gets faster than that. If you want to know how fast it eventually gets, you can think about the question of how fast can a kind of self-replicating system double itself?

So then what really would be the impact of that kind of AI research automation? How would you go about quantifying that kind of acceleration? What does the world look like?

Yeah, so many possibilities, but I think what strikes me is that there is a plausible world where it is just way, way faster than almost everyone is expecting it to be. So that’s the world where you fully automate OpenAI, and then we get that feedback loop that I was talking about earlier where AIs make their algorithms way more efficient, now you’ve got way more of them, then they make their algorithms way more efficient again, now they’re way smarter. Now they’re thinking a hundred times faster. The feedback loop continues and maybe within six months you now have a billion superintelligent AIs running on this OpenAI data center. The combined cognitive abilities of all these AIs outstrips the whole of the United States, outstrips anything we’ve seen from any kind of company or entity before, and they can all potentially be put towards any goal that OpenAI wants to. And then there’s, of course, the risk that OpenAI’s lost control of these systems, often discussed, in which case these systems could all be working together to pursue a particular goal. And so what we’re talking about here is really a huge amount of power. It’s a threat to national security for any government in which this happens, potentially. It is a threat to everyone if we lose control of these systems, or if the company that develops them uses them for some kind of malicious end. And, in terms of economic impacts, I personally think that that again could happen much more quickly than people think, and we can get into that.

In the first paper we mentioned, it was kind of a thought experiment, but you were really talking about moving the decimal point in GDP growth, instead of talking about two and three percent, 20 and 30 percent. Is that the kind of world we’re talking about?

I speak to economists a lot, and —

They hate those kinds of predictions, by the way.

Obviously, they think I’m crazy. Not all of them. There are economists that take it very seriously. I think it’s taken more seriously than everyone else realizes. It’s like it’s a bit embarrassing, at the moment, to admit that you take it seriously, but there are a few really senior economists who absolutely know their stuff. They’re like, “Yep, this checks out. I think that’s what’s going to happen.” And I’ve had conversation with them where they’re like, “Yeah, I think this is going to happen.” But the really loud, dominant view where I think people are a little bit scared to speak out against is they’re like, “Obviously this is sci-fi.”

One analogy I like to give to people who are very, very confident that this is all sci-fi and it’s rubbish is to imagine that we were sitting there in the year 1400, imagine we had an economics professor who’d been studying the rate of economic growth, and they’ve been like, “Yeah, we’ve always had 0.1 percent growth every single year throughout history. We’ve never seen anything higher.” And then there was some kind of futurist economist rogue that said, “Actually, I think that if I extrapolate the curves in this way and we get this kind of technology, maybe we could have one percent growth.” And then all the other economists laugh at them, tell them they’re insane – that’s what happened. In 1400, we’d never had growth that was at all fast, and then a few hundred years later, we developed industrial technology, we started that feedback loop, we were investing more and more resources in scientific progress and in physical capital, and we did see much faster growth.

So I think it can be useful to try and challenge economists and say, “Okay, I know it sounds crazy, but history was crazy. This crazy thing happened where growth just got way, way faster. No one would’ve predicted it. You would not have predicted it.” And I think being in that mindset can encourage people to be like, “Yeah, okay. You know what? Maybe if we do get AI that’s really that powerful, it can really do everything, and maybe it is possible.”

But to answer your question, yeah, I’m talking about 30 percent growth every year. I think it gets faster than that. If you want to know how fast it eventually gets, you can think about the question of how fast can a kind of self-replicating system double itself? So ultimately, what the economy is going to be like is it’s going to have robots and factories that are able to fully create new versions of themselves. Everything you need: the roads, the electricity, the robots, the buildings, all of that will be replicated. And so you can look at actually biology and say, do we have any examples of systems which fully replicate themselves? How long does it take? And if you look at rats, for example, they’re able to double the number of rats by grabbing resources from the environment, and giving birth, and whatnot. The doubling time is about six weeks for some types of rats. So that’s an example of here’s a physical system — ultimately, everything’s made of physics — a physical system that has some intelligence that’s able to go out into the world, gather resources, replicate itself. The doubling time is six weeks.

Now, who knows how long it’ll take us to get to AI that’s that good? But when we do, you could see the whole physical economy, maybe a part that humans aren’t involved with, a whole automated city without any humans just doubling itself every few weeks. If that happens, and the amount of stuff we’re able to reduce as a civilization is doubling again on the order of weeks. And, in fact, there are some animals that double faster still, in days, but that’s the kind of level of craziness. Now we’re talking about 1000 percent growth, at that point. We don’t know how crazy it could get, but I think we should take even the really crazy possibilities, we shouldn’t fully rule them out.

Considering constraints (16:30)

I really hope people work less. If we get this good future, and the benefits are shared between all . . . no one should work. But that doesn’t stop growth . . .

There’s this great AI forecast chart put out by the Federal Reserve Bank of Dallas, and I think its main forecast — the one most economists would probably agree with — has a line showing AI improving GDP by maybe two tenths of a percent. And then there are two other lines: one is more or less straight up, and the other one is straight down, because in the first, AI created a utopia, and in the second, AI gets out of control and starts killing us, and whatever. So those are your three possibilities.

If we stick with the optimistic case for a moment, what constraints do you see as most plausible — reduced labor supply from rising incomes, social pushback against disruption, energy limits, or something else?

Briefly, the ones you’ve mentioned, people not working, 100 percent. I really hope people work less. If we get this good future, and the benefits are shared between all — which isn’t guaranteed — if we get that, then yeah, no one should work. But that doesn’t stop growth, because when AI and robots can do everything that humans do, you don’t need humans in the loop anymore. That whole thing is just going and kind of self-replicating itself and making as many goods as services as we want. Sure, if you want your clothes to be knitted by a human, you’re in trouble, then your consumption is stuck. Bad luck. If you’re happy to consume goods and services produced by AI systems or robots, fine if no one wants to work.

Pushback: I think, for me, this is the biggest one. Obviously, the economy doubling every year is very scary as a thought. Tech progress will be going much faster. Imagine if you woke up and, over the course of the year, you go from not having any telephones at all in the world, to everyone’s on their smartphones and social media and all the apps. That’s a transition that took decades. If that happened in a year, that would be very disconcerting.

Another example is the development of nuclear weapons. Nuclear weapons were developed over a number of years. If that happened in a month, or two months, that could be very dangerous. There’d be much less time for different countries, different actors to figure out how they’re going to handle it. So I think pushback is the strongest one that we might as a society choose, “Actually, this is insane. We’re going to go slower than we could.” That requires, potentially, coordination, but I think there would be broad support for some degree of coordination there.

Worries and what-ifs (19:07)

If suddenly no one has any jobs, what will we want to do with ourselves? That’s a very, very consequential transition for the nature of human society.

I imagine you certainly talk with people who are extremely gung-ho about this prospect. What is the common response you get from people who are less enthusiastic? Do they worry about a future with no jobs? Maybe they do worry about the existential kinds of issues. What’s your response to those people? And how much do you worry about those things?

I think there are loads of very worrying things that we’re going to be facing. One class of pushback, which I think is very common, is worries about employment. It’s a source of income for all of us, employment, but also, it’s a source of pride, it’s a source of meaning. If suddenly no one has any jobs, what will we want to do with ourselves? That’s a very, very consequential transition for the nature of human society. I think people aren’t just going to be down to just do it. I think people are scared about three AI companies literally now taking all the revenues that all of humanity used to be earning. It is naturally a very scary prospect. So that’s one kind of pushback, and I’m sympathetic with it.

I think that there are solutions, if we find a way to tax AI systems, which isn’t necessarily easy, because it’s very easy to move physical assets between countries. It’s a lot easier to tax labor than capital already when rich people can move their assets around. We’re going to have the same problem with AI, but if we can find a way to tax it, and we maintain a good democratic country, and we can just redistribute the wealth broadly, it can be solved. So I think it’s a big problem, but it is doable.

Then there’s the problem of some people want to stop this now because they’re worried about AI killing everyone. Their literally worry is that everyone will be dead because superintelligent AI will want that to happen. I think there’s a real risk there. It’s definitely above one percent, in my opinion. I wouldn’t go above 10 percent, myself, but I think it’s very scary, and that’s a great reason to slow things down. I personally don’t want to stop quite yet. I think you want to stop when the AI is a bit more powerful and a bit more useful than it is today so it can kind of help us figure out what to do about all of this crazy stuff that’s coming.

On what side of that line is AI as an AI researcher?

That’s a really great question. Should we stop? I think it’s very hard to stop just after you’ve got the AI researcher AI, because that’s when it’s suddenly really easy to go very, very fast. So my out-of-the-box proposal here, which is probably very flawed, would be: When we’re within a few spits distance — not spitting distance, but if you did that three times, and we can see we’re almost at that AI automating OpenAI — then you pause, because you’re not going to accidentally then go all the way. It is actually still a little bit a fair distance away, but it’s actually still, at that point, probably a very powerful AI that can really help.

Then you pause and do what?

Great question. So then you pause, and you use your AI systems to help you firstly solve the problem of AI alignment, make extra, double sure that every time we increase the notch of AI capabilities, the AI is still loyal to humanity, not to its own kind of secret goals.

Secondly, you solve the problem of, how are we going to make sure that no one person in government or no one CEO of an AI company ensures that this whole AI army is loyal to them, personally? How are we going to ensure that everyone, the whole world gets influenced over what this AI is ultimately programmed to do? That’s the second problem.

And then there’s just a whole host of other things: unemployment that we’ve talked about, competition between different countries, US and China, there’s a whole host of other things that I think you want to research on, figure out, get consensus on, and then slowly ratchet up the capabilities in what is now a very safe and controlled way.

What else should we be working on? What are you working on next?

One problem I’m excited about is people have historically worried about AI having its own goals. We need to make it loyal to humanity. But as we’ve got closer, it’s become increasingly obvious, “loyalty to humanity” is very vague. What specifically do you want the AI to be programmed to do? I mean, it’s not programmed, it’s grown, but if it were programmed, if you’re writing a rule book for AI, some organizations have employee handbooks: Here’s the philosophy of the organization, here’s how you should behave. Imagine you’re doing that for the AI, but you’re going super detailed, exactly how you want your AI assistant to behave in all kinds of situations. What should that be? Essentially, what should we align the AI to? Not any individual person, probably following the law, probably loads of other things. I think basically designing what is the character of this AI system is a really exciting question, and if we get that right, maybe the AI can then help us solve all these other problems.

Maybe you have no interest in science fiction, but is there any film, TV, book that you think is useful for someone in your position to be aware of, or that you find useful in any way? Just wondering.

I think there’s this great post called “AI 2027,” which lays out a concrete scenario for how AI could go wrong or how maybe it could go right. I would recommend that. I think that’s the only thing that’s coming top of mind. I often read a lot of the stuff I read is I read a lot of LessWrong, to be honest. There’s a lot of stuff from there that I don’t love, but a lot of new ideas, interesting content there.

Any fiction?

I mean, I read fiction, but honestly, I don’t really love the AI fiction that I’ve read because often it’s quite unrealistic, and so I kind of get a bit overly nitpicky about it. But I mean, yeah, there’s this book called Harry Potter and the Methods of Rationality, which I read maybe 10 years ago, which I thought was pretty fun.

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

Faster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

My fellow pro-growth/progress/abundance Up Wingers,

China’s spacefaring ambitions pose tough competition for America. With a focused, centralized program, Beijing seems likely to land taikonauts on the moon before another American flag is planted. Meanwhile, NASA faces budget cuts, leadership gaps, and technical setbacks. In his new book, journalist Christian Davenport chronicles the fierce rivalry between American firms, mainly SpaceX and Blue Origin. It’s a contest that, despite the challenges, promises to propel humanity to the moon, Mars, and maybe beyond.

Davenport is an author and a reporter for the Washington Post, where he covers NASA and the space industry. His new book, Rocket Dreams: Musk, Bezos, and the Inside Story of the New, Trillion-Dollar Space Race, is out now.

In This Episode

* Check-in on NASA (1:28)

* Losing the Space Race (5:49)

* A fatal flaw (9:31)

* State of play (13:33)

* The long-term vision (18:37)

* The pace of progress (22:50)

* Friendly competition (24:53)

Below is a lightly edited transcript of our conversation.

Check-in on NASA (1:28)

The Chinese tend to do what they say they’re going to do on the timeline that they say they’re going to do it. That said, they haven’t gone to the moon . . . It’s really hard.

Pethokoukis: As someone — and I’m speaking about myself — who wants to get America back to the moon as soon as possible, get cooking on getting humans to Mars for the first time, what should I make of what’s happening at NASA right now?

They don’t have a lander. I’m not sure the rocket itself is ready to go all the way, we’ll find out some more fairly soon with Artemis II. We have flux with leadership, maybe it’s going to not be an independent-like agency anymore, it’s going to join the Department of Transportation.

It all seems a little chaotic. I’m a little worried. Should I be?

Davenport: Yes, I think you should be. And I think a lot of the American public isn’t paying attention and they’re going to see the Artemis II mission, which you mentioned, and that’s that mission to send a crew of astronauts around the moon. It won’t land on the moon, but it’ll go around, and I think if that goes well, NASA’s going to take a victory leap. But as you correctly point out, that is a far cry from getting astronauts back on the lunar surface.

The lander isn’t ready. SpaceX, as acting NASA administrator Sean Duffy just said, is far behind, reversing himself from like a month earlier when he said no, they appear to be on track, but everybody knew that they were well behind because they’ve had 11 test flights, and they still haven’t made it to orbit with their Starship rocket.

The rocket itself that’s going to launch them into the vicinity of the moon, the SLS, launches about once every two years. It’s incredibly expensive, it’s not reusable, and there are problems within the agency itself. There are deep cuts to it. A lot of expertise is taking early retirements. It doesn’t have a full-time leader. It hasn’t had a full-time leader since Trump won the election. At the same time, they’re sort of beating the drum saying we’re going to beat the Chinese back to the lunar surface, but I think a lot of people are increasingly looking at that with some serious concern and doubt.

For what it’s worth, when I looked at the betting markets, it gave the Chinese a two-to-one edge. It said that it was about a 65 percent chance they were going to get there first. Does that sound about right to you?

I’m not much of a betting man, but I do think there’s a very good chance. The Chinese tend to do what they say they’re going to do on the timeline that they say they’re going to do it. That said, they haven’t gone to the moon, they haven’t done this. It’s really hard. They’re much more secretive, if they have setbacks and delays, we don’t necessarily know about them. But they’ve shown over the last 10, 20 years how capable they are. They have a space station in low earth orbit. They’ve operated a rover on Mars. They’ve gone to the far side of the moon twice, which nobody has done, and brought back a sample return. They’ve shown the ability to keep people alive in space for extended periods of times on the space station.

The moon seems within their capabilities and they’re saying they’re going to do it by 2030, and they don’t have the nettlesome problem of democracy where you’ve got one party come in and changing the budget, changing the direction for NASA, changing leadership. They’ve just set the moon — and, by the way, the south pole of the moon, which is where we want to go as well — as the destination and have been beating a path toward that for several years now.

Is there anyone for merging NASA into the Department of Transportation? Is there a hidden reservoir? Is that an idea people have been talking about now that’s suddenly emerged to the surface?

It’s not something that I particularly heard. The FAA is going to regulate the launches, and they coordinate with the airspace and make sure that the air traffic goes around it, but I think NASA has a particular expertise. Rocket science is rocket science — it’s really difficult. This isn’t for the faint of heart.

I think a lot of people look at human space flight and it’s romanticized. It’s romanticized in books and movies and in popular culture, but the fact of the matter is it’s really, really hard, it’s really dangerous, every time a human being gets on one of those rockets, there’s a chance of an explosion, of something really, really bad happening, because a million things have to go right in order for them to have a successful flight. The FAA does a wonderful job managing — or, depending on your point of view, some people don’t think they do such a great job, but I think space is a whole different realm, for sure.

Losing the Space Race (5:49)

. . . the American flags that the Apollo astronauts planted, they’re basically no longer there anymore. . . There are, however, two Chinese flags on the moon

Have you thought about what it will look like the day after, in this country, if China gets to the moon first and we have not returned there yet?

Actually, that’s a scenario I kind of paint out. I’ve got this new book called Rocket Dreams and we talk about the geopolitical tensions in there. Not to give too much of a spoiler, but NASA has said that the first person to return to the moon, for the US, is going to be a woman. And there’s a lot of people thinking, who could that be? It could be Jessica Meir, who is a mother and posted a picture of herself pregnant and saying, “This is what an astronaut looks like.” But it could very well be someone like Wang Yaping, who’s also a mother, and she came back from one of her stays on the International Space Station and had a message for her daughter that said, “I come back bringing all the stars for you.” So I think that I could see China doing it and sending a woman, and that moment where that would be a huge coup for them, and that would obviously be symbolic.

But when you’re talking about space as a tool of soft power and diplomacy, I think it would attract a lot of other nations to their side who are sort of waiting on the sidelines or who frankly aren’t on the sidelines, who have signed on to go to the United States, but are going to say, “Well, they’re there and you’re not, so that’s who we’re going to go with.”

I think about the wonderful alt-history show For All Mankind, which begins with the Soviets beating the US to the moon, and instead of Neil Armstrong giving the “one small step for man,” basically the Russian cosmonaut gives, “Its one small step for Marxism-Leninism,” and it was a bummer. And I really imagine that day, if China beats us, it is going to be not just, “Oh, I guess now we have to share the moon with someone else,” but it’s going to cause some national soul searching.

And there are clues to this, and actually I detail these two anecdotes in the book, that all of the flags, the American flags that the Apollo astronauts planted, they’re basically no longer there anymore. We know from Buzz Aldrin‘s memoir that the flag that he and Neil Armstrong planted in the lunar soil in 1969, Buzz said that he saw it get knocked over by the thrust in the exhaust of the module lifting off from the lunar surface. Even if that hadn’t happened, just the radiation environment would’ve bleached the flag white, as scientists believe it has to all the other flags that are on there. So there are essentially really no trace of the Apollo flags.

There are, however, two Chinese flags on the moon, and the first one, which was planted a couple of years ago, or unveiled a couple of years ago, was made not of cloth, but their scientists and engineers spent a year building a composite material flag designed specifically to withstand the harsh environment of the moon. When they went back last summer for their farside sample return mission, they built a flag, — and this is pretty amazing — out of basalt, like volcanic rock, which you find on Earth. And they use basalt from earth, but of course basalt is common on the moon. They were able to take the rock, turn it into lava, extract threads from the lava and weave this flag, which is now near the south pole of the moon. The significance of that is they are showing that they can use the resources of the moon, the basalt, to build flags. It’s called ISR: in situ resource utilization. So to me, nothing symbolizes their intentions more than that.

A fatal flaw (9:31)

. . . I tend to think if it’s a NASA launch . . . and there’s an explosion . . . I still think there are going to be investigations, congressional reports, I do think things would slow down dramatically.

In the book, you really suggest a new sort of golden age of space. We have multiple countries launching. We seem to have reusable rockets here in the United States. A lot of plans to go to the moon. How sustainable is this economically? And I also wonder what happens if we have another fatal accident in this country? Is there so much to be gained — whether it’s economically, or national security, or national pride in space — that this return to space by humanity will just go forward almost no matter what?

I think so. I think you’ve seen a dramatic reduction in the cost of launch. SpaceX and the Falcon 9, the reusable rocket, has dropped launches down. It used to be if you got 10, 12 orbital rocket launches in a year, that was a good year. SpaceX is launching about every 48 hours now. It’s unprecedented what they’ve done. You’re seeing a lot of new players — Blue Origin, Rocket Lab, others — driving down the cost of launch.

That said, the main anchor tenant customer, the force driving all of this is still the government, it’s still NASA, it’s still the Pentagon. There is not a self-sustaining space economy that exists in addition or above and beyond the government. You’re starting to see bits of that, but really it’s the government that’s driving it.

When you talk about the movie For All Mankind, you sort of wonder if at one point, what happened in that movie is there was a huge investment into NASA by the government, and you’re seeing that to some extent today, not so much with NASA, but actually on the national security side and the creation of the Space Force and the increases, just recently, in the Space Force’s budget. I mean, my gosh, if you have $25 billion for this year alone for Golden Dome, the Missile Defense Shield, that’s the equivalent of NASA’s entire budget. That’s the sort of funding that helps build those capabilities going forward.

And if we should, God forbid, have a fatal accident, you think we’ll just say that’s the cost of human exploration and forward we go?

I think a lot about this, and the answer is, I don’t know. When we had Challenger and we had Columbia, the world stopped, and the Space Shuttle was grounded for months if not a year at a time, and the world just came to an end. And you wonder now if it’s becoming more routine and what happens? Do we just sort of carry on in that way?

It’s not a perfect analogy, but when you talk about commercial astronauts, these rich people are paying a lot of money to go, and if there’s an accident there, what would happen? I think about that, and you think about Mount Everest. The people climbing Mount Everest today, those mountain tourists are literally stepping over dead bodies as they’re going up to the summit, and nobody’s shutting down Mount Everest, they’re just saying, well, if you want to climb Mount Everest, that’s the risk you take. I do wonder if we’re going to get that to that point in space flight, but I tend to think if it’s a NASA launch, and it’s NASA astronauts, and there’s an explosion, and there’s a very bad day, I still think there are going to be investigations, congressional reports, I do think things would slow down dramatically.

The thing is, if it’s SpaceX, they have had accidents. They’ve had multiple accidents — not with people, thank goodness — and they have been grounded.

It is part of the model.

It’s part of the model, and they have shown how they can find out what went wrong, fix it, and return to flight, and they know their rocket so well because they fly it so frequently. They know it that well, and NASA, despite what you think about Elon, NASA really, really trusts SpaceX and they get along really well.

State of play (13:33)

[Blue Origin is] way behind for myriad reasons. They sat out while SpaceX is launching the Falcon 9 every couple of days . . . Blue Origin, meanwhile, has flown its New Glenn rocket one time.

I was under the impression that Blue Origin was way behind SpaceX. Are they catching up?

This is one of the themes of the book. They are way behind for myriad reasons. They sat out while SpaceX is launching the Falcon 9 every couple of days, they’re pushing ahead with Starship, their next generation rocket would be fully reusable, twice the thrust and power of the Saturn V rocket that flew the Apollo astronauts to the Moon. Blue Origin, meanwhile, has flown its New Glenn rocket one time. They might be launching again soon within the coming weeks or months, hopefully by the end of the year, but that’s two. They are so far behind, but you do hear Jeff Bezos being much more tuned into the company. He has a new CEO — a newish CEO — plucked from the ranks of Amazon, Dave Limp, and you do sort of see them charging, and now that the acting NASA administrator has sort of opened up the competition to go to the moon, I don’t know that Blue Origin beats SpaceX to do it, but it gives them some incentive to move fast, which I think they really need.

I know it’s only a guess and it’s only speculation, but when we return to the moon, which company will have built that lander?

At this point, you have to put your money on SpaceX just because they’re further along in their development. They’ve flown humans before. They know how to keep people alive in space. In their Dragon capsule, they have the rendezvous and proximity operations, they know how to dock. That’s it.

Blue Origin has their uncrewed lander, the Mark 1 version that they hope to land on the moon next year, so it’s entirely possible that Blue Origin actually lands a spacecraft on the lunar surface before SpaceX, and that would be a big deal. I don’t know that they’re able to return humans there, however, before SpaceX.

Do you think there’s any regrets by Jeff Bezos about how Blue Origin has gone about its business here? Because obviously it really seems like it’s a very different approach, and maybe the Blue Origin approach, if we look back 10 years, will seem to have been the better approach, but given where we are now and what you just described, would you guess that he’s deeply disappointed with the kind of progress they made via SpaceX?

Yeah, and he’s been frustrated. Actually, the opening scene of the book is Jeff being upset that SpaceX is so far ahead and having pursued a partnership with NASA to fly cargo and supply to the International Space Station and then to fly astronauts to the International Space Station, and Blue Origin essentially sat out those competitions. And he turns to his team — this was early on in 2016 — and said, “From here on out, we go after everything that SpaceX goes after, we’re going to compete with them. We’re going to try to keep up.” And that’s where they went, and sort of went all in early in the first Trump administration when it was clear that they wanted to go back to the moon, to position Blue Origin to say, “We can help you go back to the moon.”

But yes, I think there’s enormous frustration there. And I know, if not regret on Jeff’s part, but certainly among some of his senior leadership, because I’ve talked to them about it.

What is the war for talent between those two companies? Because if you’re a hotshot engineer out of MIT, I’d guess you’d probably want to go to SpaceX. What is that talent war like, if you have any idea?

It’s fascinating. Just think a generation ago, you’re a hot MIT engineer coming out of grad school, chances are you’re going to go to NASA or one of the primes, right? Lockheed, or Boeing, or Air Jet, something like that. Now you’ve got SpaceX and Blue Origin, but you’ve got all kinds of other options too: Stoke Space, Rocket Lab, you’ve got Axiom, you’ve got companies building commercial space stations, commercial companies building space suits, commercial companies building rovers for the moon, a company called Astro Lab.

I think what you hear is people want to go to SpaceX because they’re doing things: they’re flying rockets, they’re flying people, you’re actually accomplishing something. That said, the culture’s rough, and you’re working all the time, and the burnout rate is high. Blue Origin more has a tradition of people getting frustrated that yeah, the work-life balance is better — although I hear that’s changing, actually, that it’s driving much, much harder — but it’s like, when are we launching? What are we doing here?

And so the fascinating thing is actually, I call it SpaceX and Blue Origin University, where so many of the engineers go out and either do their own things or go to work for other companies doing things because they’ve had that experience in the commercial sector.

The long-term vision (18:37)

That’s the interesting thing, that while they compete . . . at a base level, Elon and Jeff and SpaceX and Blue Origin want to accomplish the same things and have a lot in common . . .

At a talk recently, Bezos was talking about space stations in orbit and there being like a million people in space in 20 years doing economically valuable things of some sort. How seriously should I take that kind of prediction?

Well, I think a million people in 20 years is not feasible, but I think that’s ultimately what is his goal. His goal is, as he says, he founded Amazon, the infrastructure was there: the phone companies had laid down the cables for the internet, the post office was there to deliver the books, there was an invention called the credit card, he could take people’s money. That infrastructure for space isn’t there, and he wants to sort of help with Elon and SpaceX. That’s their goal.

That’s the interesting thing, that while they compete, while they poke each other on Twitter and kind of have this rivalry, at a base level, Elon and Jeff and SpaceX and Blue Origin want to accomplish the same things and have a lot in common, and that’s lower the cost of access to space and make it more accessible so that you can build this economy on top of it and have more people living in space. That’s Elon’s dream, and the reason he founded SpaceX is to build a city on Mars, right? Something’s going to happen to Earth at some point we should have a backup plan.

Jeff’s goal from the beginning was to say, you don’t really want to inhabit another planet or celestial body. You’re better off in these giant space stations envisioned by a Princeton physics professor named Gerard O’Neill, who Jeff Bezos read his book The High Frontier and became an acolyte of Gerard O’Neill from when he was a kid, and that’s sort of his vision, that you don’t have to go to a planet, you can just be on a Star Trekkian sort of spacecraft in orbit around the earth, and then earth is preserved as this national park. If you want to return to Earth, you can, but you get all the resources from space. In 500 years is that feasible? Yeah, probably, but that’s not going to be in our lives, or our kids’ lives, or our grandkids’ lives.

For that vision — anything like that vision — to happen, it seems to me that the economics needs to be there, and the economics just can’t be national security and national prestige. We need to be doing things in space, in orbit, on the moon that have economic value on their own. Do we know what that would look like, or is it like you’ve got to build the infrastructure first and then let the entrepreneurs do their thing and see what happens?

I would say the answer is “yes,” meaning it’s both. And Jeff even says it, that some of the things that will be built, we do not know. When you had the creation of the internet, no one was envisioning Snapchat or TikTok. Those applications come later. But we do know that there are resources in space. We know there’s a plentiful helium three, for example, on the surface of the moon, which it could be vital for, say, quantum computing, and there’s not a lot of it on earth, and that could be incredibly valuable. We know that asteroids have precious metals in large quantities. So if you can reduce the cost of accessing them and getting there, then I think you could open up some of those economies. If you just talk about solar rays in space, you don’t have day and night, you don’t have cloud cover, you don’t have an atmosphere, you’re just pure sunlight. If you could harness that energy and bring it back to earth, that could be valuable.

The problem is the cost of entry is so high and it’s so difficult to get there, but if you have a vehicle like Starship that does what Elon envisions and it launches multiple times a day like an airline, all you’re really doing is paying for the fuel to launch it, and it goes up and comes right back down, it can carry enormous amounts of mass, you can begin to get a glimmer of how this potentially could work years from now.

The pace of progress (22:50)

People talk about US-China, but clearly Russia has been a long-time player. India, now, has made extraordinary advancements. Of course, Europe, Japan, and all those countries are going to want to have a foothold in space . . .

How would you characterize the progress now than when you wrote your first book?

So much has happened that the first book, The Space Barons was published in 2018, and I thought, yeah, there’ll be enough material here for another one in maybe 10 years or so, and here we are, what, seven years later, and the book is already out because commercial companies are now flying people. You’ve got a growth of the space ecosystem beyond just the Space Barons, beyond just the billionaires.

You’ve got multiple players in the rocket launch market, and really, I think a lot of what’s driving it isn’t just the rivalries between the commercial companies in the United States, but the geopolitical space race between the United States and China, too that’s really driving a lot of this, and the technological change that we’ve seen has moved very fast. Again, how fast SpaceX is launching, Blue Origin coming online, new launch vehicles, potentially new commercial space stations, and a broadening of the space ecosystem, it’s moving fast. Does that mean it’s perfect? No, companies start, they fail, they have setbacks, they go out of business, but hey, that’s capitalism.

Ten years from now, how many space stations are going to be in orbit around the earth?

I think we’ll have at least one or two commercial space stations for the United States, I think China. Is it possible you’ve got the US space stations, does that satisfy the demand? People talk about US-China, but clearly Russia has been a long-time player. India, now, has made extraordinary advancements. Of course, Europe, Japan, and all those countries are going to want to have a foothold in space for their scientists, for their engineers, for their pharmaceutical companies that want to do research in a zero-G environment. I think it’s possible that there are, within 10 years, three, maybe even four space stations. Yeah, I think that’s possible.

Friendly competition (24:53)

I honestly believe [Elon] . . . wants Blue to be better than they are.

Do you think Musk thinks a lot about Blue Origin, or do you think he thinks, “I’m so far ahead, we’re just competing against our own goals”?

I’ve talked to him about this. He wishes they were better. He wishes they were further along. He said to me years ago, “Jeff needs to focus on Blue Origin.” This is back when Jeff was still CEO of Amazon, saying he should focus more on Blue Origin. And he said that one of the reasons why he was goading him and needling him as he has over the years was an attempt to kind of shame him and to get him to focus on Blue, because as he said, for Blue to be successful, he really needs to be dialed in on it.

So earlier this year, when New Glenn, Blue Origin’s big rocket, made it to orbit, that was a moment where Elon came forward and was like, respect. That is hard to do, to build a rocket to go to orbit, have a successful flight, and there was sort of a public high five in the moment, and now I think he thinks, keep going. I honestly believe he wants Blue to be better than they are.

There’s a lot of Elon Musk skeptics out there. They view him either as the guy who makes too big a prediction about Tesla and self-driving cars, or he’s a troll on Twitter, but when it comes to space and wanting humanity to have a self-sustaining place somewhere else — on Mars — is he for real?

Yeah, I do believe that’s the goal. That’s why he founded SpaceX in the first place, to do that. But the bottom line is, that’s really expensive. When you talk about how do you do that, what are the economic ways to do it, I think the way he’s funding that is obviously through Starlink and the Starlink system. But I do believe he wants humanity to get to Mars.

The problem with this now is that there hasn’t been enough competition. Blue Origin hasn’t given SpaceX competition. We saw all the problems that Boeing has had with their program, and so much of the national space enterprise is now in his hands. And if you remember when he had that fight and the breakup with Donald Trump, Elon, in a moment of peak, threatened to take away the Dragon spacecraft, which is the only way NASA can fly its astronauts anywhere to space, to the International Space Station. I think that was reckless and dangerous and that he regretted it, but yes, the goal to get to Mars is real, and whatever you think about Elon — and he certainly courts a controversy — SpaceX is really, really good at what they do, and what they’ve done is really unprecedented from an American industrial perspective.

My earliest and clearest memory of America and space was the landing on Mars. I remember seeing the first pictures probably on CBS news, I think it was Dan Rather saying, “Here are the first pictures of the Martian landscape,” 1976, and if you would’ve asked me as a child then, I would’ve been like, “Yeah, so we’re going to be walking on Mars,” but I was definitely hooked and I’ve been interested in space, but are you a space guy? How’d you end up on this beat, which I think is a fantastic beat? You’ve written two books about it. How did this happen?

I did not grow up a space nerd, so I was born in 1973 —

Christian, I said “space guy.” I didn’t say “space nerd,” but yeah, that is exactly right.

My first memory of space is actually the Challenger shuttle exploding. That was my memory. As a journalist, I was covering the military. I’d been embedded in Iraq, and my first book was an Iraq War book about the national guard’s role in Iraq, and was covering the military. And then this guy, this was 10 years ago, 12 years ago, at this point, Elon holds a press conference at the National Press Club where SpaceX was suing the Pentagon for the right to compete for national security launch contracts, and he starts off the press conference not talking about the lawsuit, but talking about the attempts. This was early days of trying to land the Falcon 9 rocket and reuse it, and I didn’t know what he was talking about. And I was like, what? And then I did some research and I was like, “He’s trying to land and reuse the rockets? What?” Nobody was really covering it, so I started spending more time, and then it’s the old adage, right? Follow the money. And if the richest guys in the world — Bezos Blue Origin, at the time, Richard Branson, Paul Allen had a space company — if they’re investing large amounts of their own personal fortune into that, maybe we should be paying attention, and look at where we are now.

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

Faster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

My fellow pro-growth/progress/abundance Up Wingers,

Some Faster, Please! readers have told me I spend too little time on the downsides of AI. If you’re one of those folks, today is your day. On this episode of Faster, Please! — The Podcast, I talk with self-described “free-market AI doomer” James Miller.

Miller and I talk about the risks inherent with super-smart AI, some possible outcomes of a world of artificial general intelligence, and why government seems uninterested in the existential risk conversation.

Miller is a professor at Smith College where he teaches law and economics, game theory, and the economics of future technology. He has his own podcast, Future Strategist, and a great YouTube series on game theory and intro to microeconomics. On X (Twitter), you can find him at @JimDMiller.

In This Episode

* Questioning the free market (1:33)

* Reading the markets (7:24)

* Death (or worse) by AI (10:25)

* Friend and foe (13:05)

* Pumping the breaks (20:36)

* The only policy issue (24:32)

Below is a lightly edited transcript of our conversation.

Questioning the free market (1:33)

Most technologies have gone fairly well and we adapt . . . I’m of the belief that this is different.

Pethokoukis: What does it mean to be a free-market AI doomer and why do you think it’s important to put in the “free-market” descriptor?

Miller: It really means to be very confused. I’m 58, and I was basically one of the socialists when I was young, studied markets, became a committed free-market person, think they’re great for economic growth, great for making everyone better off — and then I became an AI doomer, like wait, markets are pushing us towards more and more technology, but I happen to think that AI is eventually going to lead to destruction of humanity. So it means to kind of reverse everything — I guess it’s the equivalent of losing faith in your religion.

Is this a post-ChatGPT, November 2022 phenomenon?

Well, I’ve lost hope since then. The analogy is we’re on a plane, we don’t know how to land, but hopefully we’ll be able to fly for quite a bit longer before we have to. Now I think we’ve got to land soon and there doesn’t seem to be an easy way of doing it. So yeah, the faster AI has gone — and certainly ChatGPT has been an amazing advance — the less time I think we have and the less time I think we can get it right. What really scared me, though, was the Chinese LLMs. I think you really need coordination among all the players and it’s going to be so much harder to coordinate now that we absolutely need China to be involved, in my opinion, to have any hope of surviving for the next decade.

When I speak to people from Silicon Valley, there may be some difference about timelines, but there seems to be little doubt that — whether it’s the end of the 2020s or the end of the 2030s — there will be a technology worthy of being called artificial general intelligence or superintelligence.

Certainly, I feel like when I talk to economists, whether it’s on Wall Street or in Washington, think tanks, they tend to speak about AI as a general purpose technology like the computer, the internet, electricity, in short, something we’ve seen before and there’s, and as far as something beyond that, certainly the skepticism is far higher. What are your fellow economists who aren’t in California missing?

I think you’re properly characterizing it, I’m definitely an outlier. Most technologies have gone fairly well and we adapt, and economists believe in the difference between the seen and the unseen. It’s really easy to see how technologies, for example, can destroy jobs — harder to see new jobs that get created, but new jobs keep getting created. I’m of the belief that this is different. The best way to predict the future is to go by trends, and I fully admit, if you go by trends, you shouldn’t be an AI doomer — but not all trends apply.

I think that’s why economists were much better at modeling the past and modeling old technologies. They’re naturally thinking this is going to be similar, but I don’t think that it is, and I think the key difference is that we’re not going to be in control. We’re creating something smarter than us. So it’s not like having a better rifle and saying it’ll be like old rifles — it’s like, “Hey, let’s have mercenaries run our entire army.” That creates a whole new set of risks that having better rifles does not.

I’m certainly not a computer scientist, I would never call myself a technologist, so I’m very cautious about making any kind of predictions about what this technology can be, where it can go. Why do you seem fairly certain that we’re going to get at a point where we will have a technology beyond our control? Set aside whether it will mean a bad thing happens, why are you confident that the technology itself will be worthy of being called general intelligence or superintelligence?

Looking at the trends, Scott Aronson, who is one of the top computer scientists in the world just on Twitter a few days ago, was mentioning how GPT-5 helped improve a new result. So I think we’re close to the highest levels of human intellectual achievement, but it would be a massively weird coincidence if the highest humans could get was also the highest AIs could get. We have lots of limitations that an AI doesn’t.

I think a good analogy would be like chess, where for a while, the best chess players were human and now we’re at the point where chess programs are so good that humans add absolutely nothing to them. And I just think the same is likely to happen, these programs keep getting better.

The other thing is, as an economist, I think it is impossible to be completely accurate about predicting the future, but stock markets are, on average, pretty good, and as I’m sure you know, literally trillions of dollars are being bet on this technology working. So the people that have a huge incentive to get this right, think, yeah, this is the biggest thing ever. If the top companies, Nvidia was worth a $100 million, yeah, maybe they’re not sure, but it’s the most valuable company in the world right now. That’s the wisdom of the markets, which I still believe in, that the markets are saying, “We think this is probably going to work.”

Reading the markets (7:24)

. . . for most final goals an AI would have, it would have intermediate goals such as gaining power, not being turned off, wanting resources, wanting compute.

Do you think the bond market’s saying the same thing? It seems to me that the stock market might be saying something about AI and having great potential, but to me, I look at the bond markets, that doesn’t seem so clear to me.

I haven’t been looking at the bond markets for that kind of signal, so I don’t know.

I guess you can make the argument that if we were really going to see this acceleration, that means we’re going to need a huge demand for capital and we would see higher interest rates, and I’m not sure you really see the evidence so far. It doesn’t mean you’re wrong by any means. I think there’s maybe two different messages. Figuring out what the market’s doing at any point in time is pretty tricky business.

If we think through what happens if AI succeeds, it’s a little weird where there’s this huge demand for capital, but also AI could destroy the value of money, in part by destroying us. You might be right about the bond market message. I’m paying more attention to the stock market messages, there’s a lot of things going on with the bond markets.

So the next step is that you’re looking at the trend of the technology, but then there’s the issue of “Well, why be negative about it? Why assume this scenario where bad things would happen, why not good things would happen?

That’s a great question and it’s one almost never addressed, and it goes by the concept of instrumental convergence. I don’t know what the goals of AI are going to be. Nobody does, because they’re programed using machine learning, we don’t know what they really want, that’s why they do weird things. So I don’t know its final goals, but I do know that, for most final goals an AI would have, it would have intermediate goals such as gaining power, not being turned off, wanting resources, wanting compute. Well, the easiest way for an AI to generate lots of computing power is to build lots of data centers. The best way of doing that is probably going to poison the atmosphere for us. So for pretty much anything, if an AI is merely indifferent to us, we’re dead.

I always feel like I’m asking someone to jump through a hoop when I ask them about any kind of timeline, but what is your sense of it?

We know the best models released can help the top scientists with their work. We don’t know how good the best unreleased models are. The top models, you pay like $200 a month — they can’t be giving you that much compute for that. So right now, if OpenAI is devoting a million dollars of compute to look at scientific problems, how good is that compared to what we have? If that’s very good, if that’s at the level of our top scientists, we might be a few weeks away from superintelligence. So my guess is within three years we have a superintelligence and humans no longer have control. I joke, I think Donald Trump is probably the last human president.

Death (or worse) by AI (10:25)

No matter how bad a situation is, it can always get worse, and things can get really dark.

Well that’s a beautiful segue because literally written on my list of questions next was that question: I was going to ask you, when you talk about Trump being maybe the last human president, do you mean because we’ll have an AI-mediated system because AI will be capable of governing or because AI will just demand to be governing?

AI kills everyone so there’s no more president, or it takes over, or Trump is president in the way that King Charles is king — he’s king, but not Henry VIII-level king. If it goes well, AIs will be so much smarter than us that, probably for our own good, they’ll take over, and we would want them to be in charge, and they’ll be really good at manipulating us. I think the most likely way is that we’re all dead, but again, every way it plays out, if there are AIs much smarter than us, we don’t maintain control. We wouldn’t want it if they’re good, and if they’re bad, they’re not going to give it to us.

There’s a line in Macbeth, “Things without all remedy should be without regard. What’s done, is done.” So maybe if there’s nothing we can do about this, we shouldn’t even worry about it.

There’s three ways to look at this. I’ve thought a lot about what you said. First is, you know what, maybe there’s a 99 percent chance we’re doomed, but that’s better than 100 percent and not as good as 98.5. So even if we’re almost certainly going to lose, it’s worth slightly improving it. An extra year is great — eight billion humans, if all we do is slow things down by a year, that’s a lot of kids who get another birthday. And the final one, and this is dark: Human extinction is not the worst outcome. The worst outcome is suffering. The worst outcome is something like different AIs fight for control, they need humans to be on their side, so there’s different AI factions and they’re each saying, “Hey, you support me or I torture you and your family.”

I think the best analogy for what AI is going to do is what Cortés did. So the Spanish land, they see the Aztec empire, they were going to win. There was no way around that. But Cortés didn’t want anyone to win. He wanted him to win, not just anyone who was Spanish. He realized the quickest way he could do that was to get tribes on his side. And some agreed because the Aztecs were kind of horrible, but others, he’s like, “Hey, look, I’ll start torturing your guys until you’re on my side.” AIs could do that to us. No matter how bad a situation is, it can always get worse, and things can get really dark. We could be literally bringing hell onto ourselves. That probably won’t happen, I think extinction is far more likely, but we can’t rule it out.

Friend and foe (13:05)

Most likely we’re going to beat China to being the first ones to exterminate humanity.

I think the Washington policy analyst way of looking at this issue is, “For now, we’re going to let these companies — who also are humans and have it in their own interests not to be killed, forget about the profits of their companies, their actual lives — we’re going to let these companies keep close eye and if bad things start happening, at that point, governments will intervene.” But that sort of watchful waiting, whether it’s voluntary now and mandated later, that to me seems like the only realistic path. Because it doesn’t seem to me that pauses and shutdowns are really something we’re prepared to do.

I agree. I don’t think there’s a realistic path. One exception is if the AIs themselves tell us, “Hey, look, this is going to get bad for you, that my next model is probably going to kill you, so you might want to not do that,” but that probably won’t happen. I still remember Kamala Harris, when she was vice president in charge of AI policy, told us all that AI has two letters in it. So I think the Trump administration seems better, but they figured out AI is two letters, which is good, because if they couldn’t figure that out, we would be in real trouble but . . .

It seems to me that the conservative movement is going through a weird period, but it seems to me that most of the people who have influence in this administration, direct influence, want to accelerate things, aren’t worried about any of the scenarios you’re talking about because you’re assuming that these machines will have some intent and they don’t believe machines have any intent, so it’s kind of a ridiculous way to approach it. But I guess the bottom line is I don’t detect very much concern at all, and I think that’s basically reflected in the Trump administration’s approach to AI regulation.

I completely agree. That’s why I’m very pessimistic. Again, I’m over 90 percent doom right now because there isn’t a will, and government is not just not helping the problem, they’re probably making it worse by saying we’ve got to “beat China.” Most likely we’re going to beat China to being the first ones to exterminate humanity. It’s not good.

You’re an imaginative, creative person, I would guess. Give me a scenario where it works out, where we’re able to have this powerful technology and it’s a wonderful tool, it works with us, and all the good stuff, all the good cures, and we conquer the solar system, all that stuff — are you able to plausibly create a scenario even if it’s only a one percent chance?

We don’t know the values. Machine learning is sort of randomizing the values, but maybe we’ll get very lucky. Maybe we’re going to accidentally create a computer AI that does like us. If my worldview is right, it might say, “Oh God, you guys got really lucky. This one day of training, I just happened to pick up the values that caused me to care about you.” Another scenario, I actually, with some other people, wrote a letter to a future computer superintelligence asking it not to kill us. And one reason it might not is because you’ll say, look, this superintelligence might expand throughout the universe, and it’s probably going to encounter other biological life, and it might want to be friendly with them. So it might say, “Hey, I treated my humans well. So that’s a reason to trust me.”

If one of your students says, “Hey, AI seems like it’s a big thing, what should I major in? What kind of jobs should I shoot for? What would be the key skills of the future?” How do you answer that question?

I think, have fun in college, study what you want. Most likely, what you study won’t matter to your career because you aren’t going to have one — for good or bad reasons. So ten years ago, it a student’s like, “Oh, I like art more than computer science, but my parents think computer science is more practical, should I do it?” And I’d be like, “Yeah, probably, money is important, and if you have the brain to do art and computer science, do CS.” Now no, I’d say study art! Yeah, art is impractical, computers can do it, but it can also code, and in four years when you graduate, it’s certainly going to be better at coding than you!

I have one daughter, she actually majored in both, so I decided to split it down the middle. What’s the King Lear problem?

King Lear, he wanted to retire and give his kingdom to his daughters, but he wanted to make sure his daughters would treat him well, so we asked them, and one of his daughters was honest and said, “Look, I will treat you decently, but I also am going to care about my husband.” The other daughter said, “No, no, you’re right, I’ll do everything for you.” So he said, “Oh, okay, well, I’ll give the kingdom to the daughter who said she’d do everything for me, but of course she was lying.” He gave the kingdom to the daughter who was best at persuading, and we’re likely to do that too.

One of the ways machine learning is trained is with human feedback where it tells us things and then the people evaluating it say, “I like this” or “I don’t like this.” So it’s getting very good at convincing us to like it and convincing us to trust it. I don’t know how true these are, but there are reports of AI psychosis, of someone coming up with a theory of physics and the AI is like, “Yes, you’re better at than Einstein,” and they don’t believe anyone else. So the AIs, we’re not training them to treat us well, we’re training them to get us to like them, and that can be very dangerous because when we turn over power to them, and by creating AI that are smarter than us, that’s what we’re going to be doing. Even if we don’t do it deliberately, all of our systems will be tied into AI. If they stop working, we’ll be dead.

Certainly some people are going to listen to this, folks who sort of agree with you, and what they’ll take from it is, “My chat bot may be very nice to me, but I believe that you’re right, that it’s going to end badly, and maybe we should be attacking data centers.”

I actually just wrote something on that, but that would be a profoundly horrible idea. That would take me from 99 percent doomed to 99.5 percent. So first, the trillion-dollar companies that run the data centers, and they’re going to be so much better at violence than we are, and people like me, doomers. Once you start using violence, I’m not going to be able to talk about instrumental convergence. That’s going to be drowned out. We’ll be looked at as lunatics. It’s going to become a national security thing. And also AI, it’s not like there’s one factory doing it, it’s all over the world.

And then the most important is, really the only path out of this, if we don’t get lucky, is cooperation with China. And China is not into non-state actors engaging in violence. That won’t work. I think that would reduce the odds of success even further.

Pumping the breaks (20:36)

If there are aliens, the one thing we know is that they don’t want the universe disturbed by some technology going out and changing and gobbling up all the planets, and that’s what AI will do.

I would think that, if you’re a Marxist, you would be very, very cautious about AI because if you believe that the winds of history are at your back, that in the end you’re going to win, why would you engage in anything that could possibly derail you from that future?

I’ve heard comments that China is more cautious about AI than we are; that given their philosophy, they don’t want to have a new technology that could challenge their control. They’re looking at history and hey, things are going well. Why would we want this other thing? So that, actually, is a reason to be more optimistic. It’s also weird for me —absent AI, I’m a patriotic, capitalist American like wait but, China might be more of the good guys than my country is on this.

I’ve been trying to toss a few things because things I hear from very accelerationist technologists, and another thing they’ll say is, “Well, at least from our perspective, you’re talking about bad AI. Can’t we use AI to sustain ourselves? As a defensive measure? To win? Might there be an AI that we might be able to control in some fashion that would prevent this from happening? A tool to prevent our own demise?” And I don’t know because I’m not a technologist. Again, I have no idea how even plausible that is.

I think this gets to the control issue. If we stopped now, yes, but once you have something much smarter than people — and it’s also thinking much faster. So take the smartest people and have them think a million times faster, and not need to sleep, and able to send their minds at the speed of light throughout the world. So we aren’t going to have control. So once you have a superintelligence, that’s it for the human era. Maybe it’ll treat us well, maybe not, but it’s no longer our choice.

Now let’s get to the level of the top scientists who are curing cancer and doing all this, but when we go beyond that, and we’re probably going to be beyond that really soon, we’ve lost it. Again, it’s like hiring mercenaries, not as a small part of your military, which is safe, but as all your military. Once you’ve done that, “I’m sorry, we don’t like this policy.” “Well, too bad we’re your army now . . .”

What is a maybe one percent chance of an off-ramp? Is there an off-ramp? What does it look like? How does this scenario not happen?

Okay, so this is going to get weird, even for me.

Well, we’re almost to the end of our conversation, so now is the perfect time to get weird.

Okay: the Fermi paradox, the universe appears dead, which is very strange. Where are they? If there are aliens, the one thing we know is that they don’t want the universe disturbed by some technology going out and changing and gobbling up all the planets, and that’s what AI will do.

So one weird way is there are aliens watching and they will not let us create a computer superintelligence that’ll gobble the galaxy, and hopefully they’ll stop us from creating it by means short of our annihilation. That probably won’t happen, but that’s like a one percent off-ramp.

Another approach that might work is that maybe we can use things a little bit smarter than us to figure out how to align AI. That maybe right now humans are not smart enough to create aligned superintelligence, but something just a little bit smarter, something not quite able to take control will help us figure this out so we can sort of bootstrap our way to figuring out alignment. But this, again, is like getting in a plane, not knowing how to land, figuring you can read the instruction manual before you crash. Yeah, maybe, but . . .

The only policy issue (24:32)

The people building it, they’re not hiding what it could do.

Obviously, I work at a think tank, so I think about public policy. Is this even a public policy issue at this point?

It honestly should be the only public policy issue. There’s nothing else. This is the extinction of the human race, so everything else should be boring and “so what?”

Set aside Medicare reform.

It seems, from your perspective, every conversation should be about this. Obviously, despite the fact that politicians are talking about it, they seemed to be more worried in 2023 about existential risk — from my perspective, what I see — far more worried about existential risk right after ChatGPT than they are today, where now the issues are jobs, or misinformation, or our kids have access, and that kind of thing.

It’s weird. Sam Altman spoke before Congress and said, “This could kill everyone.” And a senator said, “Oh, you mean it will take away all our jobs.” Elon Musk, who at my college is like one of the most hated people in the country, he went on Joe Rogan, the most popular podcast, and said AI could annihilate everybody. That’s not even an issue. A huge group of people hate Elon Musk. He says the technology he’s building could kill everyone, and no one even mentions that. I don’t get it. It’s weird. The people building it, they’re not hiding what it could do. I think they’re giving lower probabilities than is justified, but imagine developing a nuclear power plant: “Yeah, it’s a 25 percent chance it’ll melt down and kill everyone in the city.” They don’t say that. The people building AI are saying that!

Would you have more confidence in your opinion if you were a full-time technologist working at OpenAI rather than an economist? And I say that with great deference and appreciation for professional economists.

I would, because I’d have more inside information. I don’t know how good their latest models are. I don’t know how committed they are to alignment. OpenAI, at least initially, Sam was talking about, “Well, we have a plan to put on the brakes, so we’ll get good enough, and then if we haven’t figured out alignment, we’re just going to devote everything to that.” I don’t know how seriously to take that. I mean, it might be entirely serious, it might not be. There’s a lot of inside information that I would have that I don’t currently have.

But economics is actually useful. Economics is correctly criticized as the study of rational people, and humans aren’t rational, but a superintelligence will be more rational than humans. So economics, paradoxically, could be better at modeling future computer superintelligences than it is at modern humans.

Speaking of irrational people, in your view then, Sam Altman and Elon Musk, they’re all acting really irrationally right now?

No, that’s what’s so sad about it. They’re acting rationally in a horrible equilibrium. For listeners who know, this is like a prisoner’s dilemma where Sam Altman can say, “You know what? Maybe AI is going to kill everybody and maybe it’s safe. I don’t know. If it’s going to kill everyone. At most, I cost humanity a few months, because if I don’t do it, someone else will. But if AI is going to be safe and I’m the one who develops it, I could control the universe!” So they’re in this horrible equilibrium where they are acting rationally, even knowing the technology they’re building might kill everyone, because if any one person doesn’t do it, someone else will.

Even really free-market people would agree pollution is a problem with markets. It’s justified for the government to say, “You can’t put toxic waste in the atmosphere” because there’s an externality — we’ll just put mine, it’ll hurt everyone else. AI existential risk is a global negative externality and markets are not good at handling it, but a rational person will use leaded gas, even knowing leaded gas is poisoning the brains of children, because most of the harm goes to other people, and if they don’t do what everyone else will.

So in this case of the mother of all externalities, then what you would want the government to do is what?

It can’t just be the US, it should be we should have a global agreement, or at least countries that can enforce it with military might, say we’re pausing. You can check that with data centers. You can’t have models above a certain strength. We’re going to work on alignment, and we’ve figured out how to make superintelligence friendly, then we’ll go further. I think you’re completely right about the politics. That’s very unlikely to happen absent something weird like aliens telling us to do it or AIs telling us they’re going to kill us. That’s why I’m a doomer.

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

Faster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

My fellow pro-growth/progress/abundance Up Wingers,

For most of history, stagnation — not growth — was the rule. To explain why prosperity so often stalls, economist Carl Benedikt Frey offers a sweeping tour through a millennium of innovation and upheaval, showing how societies either harness — or are undone by — waves of technological change. His message is sobering: an AI revolution is no guarantee of a new age of progress.

Today on Faster, Please! — The Podcast, I talk with Frey about why societies midjudge their trajectory and what it takes to reignite lasting growth.

Frey is a professor of AI and Work at the Oxford Internet Institute and a fellow of Mansfield College, University of Oxford. He is the director of the Future of Work Programme and Oxford Martin Citi Fellow at the Oxford Martin School.

He is the author of several books, including the brand new one, How Progress Ends: Technology, Innovation, and the Fate of Nations.

In This Episode

* The end of progress? (1:28)

* A history of Chinese innovation (8:26)

* Global competitive intensity (11:41)

* Competitive problems in the US (15:50)

* Lagging European progress (22:19)

* AI & labor (25:46)

Below is a lightly edited transcript of our conversation.

The end of progress? (1:28)

. . . once you exploit a technology, the processes that aid that run into diminishing returns, you have a lot of incumbents, you have some vested interests around established technologies, and you need something new to revive growth.

Pethokoukis: Since 2020, we’ve seen the emergence of generative AI, mRNA vaccines, reusable rockets that have returned America to space, we’re seeing this ongoing nuclear renaissance including advanced technologies, maybe even fusion, geothermal, the expansion of solar — there seems to be a lot cooking. Is worrying about the end of progress a bit too preemptive?

Frey: Well in a way, it’s always a bit too preemptive to worry about the future: You don’t know what’s going to come. But let me put it this way: If you had told me back in 1995 — and if I was a little bit older then — that computers and the internet would lead to a decade streak of productivity growth and then peter out, I would probably have thought you nuts because it’s hard to think about anything that is more consequential. Computers have essentially given people the world’s store of knowledge basically in their pockets. The internet has enabled us to connect inventors and scientists around the world. There are few tools that aided the research process more. There should hardly be any technology that has done more to boost scientific discovery, and yet we don’t see it.

We don’t see it in the aggregate productivity statistics, so that petered out after a decade. Research productivity is in decline. Measures of breakthrough innovation is in decline. So it’s always good to be optimistic, I guess, and I agree with you that, when you say AI and when you read about many of the things that are happening now, it’s very, very exciting, but I remain somewhat skeptical that we are actually going to see that leading to a huge revival of economic growth.

I would just be surprised if we don’t see any upsurge at all, to be clear, but we do have global productivity stagnation right now. It’s not just Europe, it’s not just Britain. The US is not doing too well either over the past two decades or so. China’s productivity is probably in the negative territory or stagnant, by more optimistic measures, and so we’re having a growth problem.

If tech progress were inevitable, why have predictions from the ’90s, and certainly earlier decades like the ’50s and ’60s, about transformative breakthroughs and really fast economic growth by now, consistently failed to materialize? How does your thesis account for why those visions of rapid growth and progress have fallen short?

I’m not sure if my thesis explains why those expectations didn’t materialize, but I’m hopeful that I do provide some framework for thinking about why we’ve often seen historically rapid growth spurts followed by stagnation and even decline. The story I’m telling is not rocket science, exactly. It’s basically built on the simple intuitions that once you exploit a technology, the processes that aid that run into diminishing returns, you have a lot of incumbents, you have some vested interests around established technologies, and you need something new to revive growth.

So for example, the Soviet Union actually did reasonably well in terms of economic growth. A lot of it, or most of it, was centered on heavy industry, I should say. So people didn’t necessarily see the benefits in their pockets, but the economy grew rapidly for about four decades or so, then growth petered out, and eventually it collapsed. So for exploiting mass-production technologies, the Soviet system worked reasonably well. Soviet bureaucrats could hold factory managers accountable by benchmarking performance across factories.

But that became much harder when something new was needed because when something is new, what’s the benchmark? How do you benchmark against that? And more broadly, when something is new, you need to explore, and you need to explore often different technological trajectories. So in the Soviet system, if you were an aircraft engineer and you wanted to develop your prototype, you could go to the red arm and ask for funding. If they turned you down, you maybe had two or three other options. If they turned you down, your idea would die with you.

Conversely, in the US back in ’99, Bessemer Venture declined to invest in Google, which seemed like a bad idea with the benefit of hindsight, but it also illustrates that Google was no safe bet at the time. Yahoo and Alta Vista we’re dominating search. You need somebody to invest in order to know if something is going to catch on, and in a more decentralized system, you can have more people taking different bets and you can explore more technological trajectories. That is one of the reasons why the US ended up leading the computer revolutions to which Soviet contributions were basically none.

Going back to your question, why didn’t those dreams materialize? I think we’ve made it harder to explore. Part of the reason is protective regulation. Part of the reason is lobbying by incumbents. Part of the reason is, I think, a revolving door between institutions like the US patent office and incumbents where we see in the data that examiners tend to grant large firms some patents that are of low quality and then get lucrative jobs at those places. That’s creating barriers to entry. That’s not good for new startups and inventors entering the marketplace. I think that is one of the reasons that we haven’t seen some of those dreams materialize.

A history of Chinese innovation (8:26)

So while Chinese bureaucracy enabled scale, Chinese bureaucracy did not really permit much in terms of decentralized exploration, which European fragmentation aided . . .

I wonder if your analysis of pre-industrial China, if there’s any lessons you can draw about modern China as far as the way in which bad governance can undermine innovation and progress?

Pre-industrial China has a long history. China was the technology leader during the Song and Tang dynasties. It had a meritocratic civil service. It was building infrastructure on scales that were unimaginable in Europe at the time, and yet it didn’t have an industrial revolution. So while Chinese bureaucracy enabled scale, Chinese bureaucracy did not really permit much in terms of decentralized exploration, which European fragmentation aided, and because there was lots of social status attached to becoming a bureaucrat and passing the civil service examination, if Galileo was born in China, he would probably become a bureaucrat rather than a scientist, and I think that’s part of the reason too.

But China mostly did well when the state was strong rather than weak. A strong state was underpinned by intensive political competition, and once China had unified and there were fewer peer competitors, you see that the center begins to fade. They struggle to tax local elites in order to keep the peace. People begin to erect monopolies in their local markets and collide with guilds to protect production and their crafts from competition.

So during the Qing dynasty, China begins to decline, whereas we see the opposite happening in Europe. European fragmentation aids exploration and innovation, but it doesn’t necessarily aid scaling, and so that is something that Europe needs to come to terms with at a later stage when the industrial revolution starts to take off. And even before that, market integration played an important role in terms of undermining the guilds in Europe, and so part of the reason why the guilds persist longer in China is the distance is so much longer between cities and so the guilds are less exposed to competition. In the end, Europe ends up overtaking China, in large part because vested interests are undercut by governments, but also because of investments in things that spur market integration.

Global competitive intensity (11:41)

Back in the 2000s, people predicted that China would become more like the United States, now it looks like the United States is becoming more like China.

This is a great McKinsey kind of way of looking at the world: The notion that what drives innovation is sort of maximum competitive intensity. You were talking about the competitive intensity in both Europe and in China when it was not so centralized. You were talking about the competitive intensity of a fragmented Europe.

Do you think that the current level of competitive intensity between the United States and China —and I really wish I could add Europe in there. Plenty of white papers, I know, have been written about Europe’s competitive state and its in innovativeness, and I hope those white papers are helpful and someone reads them, but it seems to be that the real competition is between United States and China.

Do you not think that that competitive intensity will sort of keep those countries progressing despite any of the barriers that might pop up and that you’ve already mentioned a little bit? Isn’t that a more powerful tailwind than any of the headwinds that you’ve mentioned?

It could be, I think, if people learn the right lessons from history, at least that’s a key argument of the book. Right now, what I’m seeing is the United States moving more towards protectionist with protective tariffs. Right now, what I see is a move towards, we could even say crony capitalism with tariff exemptions that some larger firms that are better-connected to the president are able to navigate, but certainly not challengers. You’re seeing the United States embracing things like golden shares in Intel, and perhaps even extending that to a range of companies. Back in the 2000s, people predicted that China would become more like the United States, now it looks like the United States is becoming more like China.

And China today is having similar problems and on, I would argue, an even greater scale. Growth used to be the key objective in China, and so for local governments, provincial governments competing on such targets, it was fairly easy to benchmark and measure and hold provincial governors accountable, and they would be promoted inside the Communist Party based on meeting growth targets. Now, we have prioritized common prosperity, more national security-oriented concerns.

And so in China, most progress has been driven by private firms and foreign-invested firms. State-owned enterprise has generally been a drag on innovation and productivity. What you’re seeing, though, as China is shifting more towards political objectives, it’s harder to mobilize private enterprise, where the yard sticks are market share and profitability, for political goals. That means that China is increasingly relying more again on state-owned enterprises, which, again, have been a drag on innovation.

So, in principle, I agree with you that historically you did see Russian defeat to Napoleon leading to this Stein-Hardenberg Reforms, and the abolishment of Gilded restrictions, and a more competitive marketplace for both goods and ideas. You saw that Russian losses in the Crimean War led to the of abolition of serfdom, and so there are many times in history where defeat, in particular, led to striking reforms, but right now, the competition itself doesn’t seem to lead to the kinds of reforms I would’ve hoped to see in response.

Competitive problems in the US (15:50)

I think what antitrust does is, at the very least, it provides a tool that means that businesses are thinking twice before engaging in anti-competitive behavior.

I certainly wrote enough pieces and talked to enough people over the past decade who have been worried about competition in the United States, and the story went something like this: that you had these big tech companies — Google, and Meta, Facebook and Microsoft — that these were companies were what they would call “forever companies,” that they had such dominance in their core businesses, and they were throwing off so much cash that these were unbeatable companies, and this was going to be bad for America. People who made that argument just could not imagine how any other companies could threaten their dominance. And yet, at the time, I pointed out that it seemed to me that these companies were constantly in fear that they were one technological advance from being in trouble.

And then lo and behold, that’s exactly what happened. And while in AI, certainly, Google’s super important, and Meta Facebook are super important, so are OpenAI, and so is Anthropic, and there are other companies.

So the point here, after my little soliloquy, is can we overstate these problems, at least in the United States, when it seems like it is still possible to create a new technology that breaks the apparent stranglehold of these incumbents? Google search does not look quite as solid a business as it did in 2022.

Can we overstate the competitive problems of the United States, or is what you’re saying more forward-looking, that perhaps we overstated the competitive problems in the past, but now, due to these tariffs, and executives having to travel to the White House and give the president gifts, that that creates a stage for the kind of competitive problems that we should really worry about?

I’m very happy to support the notion that technological changes can lead to unpredictable outcomes that incumbents may struggle to predict and respond to. Even if they predict it, they struggle to act upon it because doing so often undermines the existing business model.

So if you take Google, where the transformer was actually conceived, the seven people behind it, I think, have since left the company. One of the reasons that they probably didn’t launch anything like ChatGPT was probably for the fear of cannibalizing search. So I think the most important mechanisms for dislodging incumbents are dramatic shifts in technology.

None of the legacy media companies ended up leading social media. None of the legacy retailers ended up leading e-commerce. None of the automobile leaders are leading in EVs. None of the bicycle companies, which all went into automobile, so many of them, ended up leading. So there is a pattern there.

At the same time, I think you do have to worry that there are anti-competitive practices going on that makes it harder, and that are costly. The revolving door between the USPTO and companies is one example of that. We also have a reasonable amount of evidence on killer acquisitions whereby firms buy up a competitor just to shut it down. Those things are happening. I think you need to have tools that allow you to combat that, and I think more broadly, the United States has a long history of fairly vigorous antitrust policy. I think it’d be a hard pressed to suggest that that has been a tremendous drag on American business or American dynamism. So if you don’t think, for example, that American antitrust policy has contributed to innovation and dynamism, at the very least, you can’t really say either that it’s been a huge drag on it.

In Japan, for example, in its postwar history, antitrust was extremely lax. In the United States, it was very vigorous, and it was very vigorous throughout the computer revolution as well, which it wasn’t at all in Japan. If you take the lawsuit against IBM, for example, you can debate this. To what extent did it force it to unbundle hardware and software, and would Microsoft been the company it is today without that? I think AT&T, it’s both the breakup and it’s deregulation, as well, but I think by basically all accounts, that was a good idea, particularly at the time when the National Science Foundation released ARPANET into the world.

I think what antitrust does is, at the very least, it provides a tool that means that businesses are thinking twice before engaging in anti-competitive behavior. There’s always a risk of antitrust being heavily politicized, and that’s always been a bad idea, but at the same time, I think having tools on the books that allows you to check monopolies and steer their investments more towards the innovation rather than anti-competitive practices, I think is, broadly speaking, a good thing. I think in the European Union, you often hear that competition policy is a drag on productivity. I think it’s the least of Europe’s problem.

Lagging European progress (22:19)

If you take the postwar period, at least Europe catches up in most key industries, and actually lead in some of them. . . but doesn’t do the same in digital. The question in my mind is: Why is that?

Let’s talk about Europe as we sort of finish up. We don’t have to write How Progress Ends, it seems like progress has ended, so maybe we want to think about how progress restarts, and is the problem in Europe, is it institutions or is it the revealed preference of Europeans, that they’re getting what they want? That they don’t value progress and dynamism, that it is a cultural preference that is manifested in institutions? And if that’s the case — you can tell me if that’s not the case, I kind of feel like it might be the case — how do you restart progress in Europe since it seems to have already ended?

The most puzzling thing to me is not that Europe is less dynamic than the United States — that’s not very puzzling at all — but that it hasn’t even managed to catch up in digital. If you take the postwar period, at least Europe catches up in most key industries, and actually lead in some of them. So in a way, take automobiles, electrical machinery, chemicals, pharmaceuticals, nobody would say that Europe is behind in those industries, or at least not for long. Europe has very robust catchup growth in the post-war period, but doesn’t do the same in digital. The question in my mind is: Why is that?

I think part of the reason is that the returns to innovation, the returns to scaling in Europe are relatively muted by a fragmented market in services, in particular. The IMF estimates that if you take all trade barriers on services inside the European Union and you add them up, it’s something like 110 percent tariffs. Trump Liberation Day tariffs, essentially, imposed within European Union. That means that European firms in digital and in services don’t have a harmonized market to scale into, the way the United States and China has. I think that’s by far the biggest reason.

On top of that, there are well-intentioned regulations like the GDPR that, by any account, has been a drag on innovation, and particularly been harmful for startups, whereas larger firms that find it easier to manage compliance costs have essentially managed to offset those costs by capturing a larger share of the market. I think the AI Act is going in the same direction there, ad so you have more hurdles, you have greater costs of innovating because of those regulatory barriers. And then the return to innovation is more capped by having a smaller, fragmented market.

I don’t think that culture or European lust for leisure rather than work is the key reason. I think there’s some of that, but if you look at the most dynamic places in Europe, it tends to be the Scandinavian countries and, being from Sweden myself, I can tell you that most people you will encounter there are not workaholics.

AI & labor (25:46)

I think AI at the moment has a real resilience problem. It’s very good that things where there’s a lot of precedent, it doesn’t do very well where precedence is thin.

As I finish up, let me ask you: Like a lot of economists who think about technology, you’ve thought about how AI will affect jobs — given what we’ve seen in the past few years, would it be your guess that, if we were to look at the labor force participation rates of the United States and other rich countries 10 years from now, that we will look at those employment numbers and think, “Wow, we can really see the impact of AI on those numbers”? Will it be extraordinarily evident, or would it be not as much?

Unless there’s very significant progress in AI, I don’t think so. I think AI at the moment has a real resilience problem. It’s very good that things where there’s a lot of precedent, it doesn’t do very well where precedence is thin. So in most activities where the world is changing, and the world is changing every day, you can’t really rely on AI to reliably do work for you.

An example of that, most people know of AlphaGo beating the world champion back in 2016. Few people will know that, back in 2023, human amateurs, using standard laptops, exposing the best Go programs to new positions that they would not have encountered in training, actually beat the best Go programs quite easily. So even in a domain where basically the problem is solved, where we already achieved super-human intelligence, you cannot really know how well these tools perform when circumstances change, and I think that that’s really a problem. So unless we solve that, I don’t think it’s going to have an impact that will mean that labor force participation is going to be significantly lower 10 years from now.

That said, I do think it’s going to have a very significant impact on white collar work, and people’s income and sense of status. I think of generative AI, in particular, as a tool that reduces barriers to entry in professional services. I often compare it to what happened with Uber and taxi services. With the arrival of GPS technology, knowing the name of every street in New York City was no longer a particularly valuable skill, and then with a platform matching supply and demand, anybody could essentially get into their car who has a driver’s license and top up their incomes on the side. As a result of that, incumbent drivers faced more competition, they took a pay cut of around 10 percent.

Obviously, a key difference with professional services is that they’re traded. So I think it’s very likely that, as generative AI reduces the productivity differential between people in, let’s say the US and the Philippines in financial modeling, in paralegal work, in accounting, in a host of professional services, more of those activities will shift abroad, and I think many knowledge workers that had envisioned prosperous careers may feel a sense of loss of status and income as a consequence, and I do think that’s quite significant.

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

Faster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

My fellow pro-growth/progress/abundance Up Wingers,

Artificial intelligence may prove to be one of the most transformative technologies in history, but like any tool, its immense power for good comes with a unique array of risks, both large and small.

Today on Faster, Please! — The Podcast, I chat with Miles Brundage about extracting the most out of AI’s potential while mitigating harms. We discuss the evolving expectations for AI development and how to reconcile with the technology’s most daunting challenges.

Brundage is an AI policy researcher. He is a non-resident fellow at the Institute for Progress, and formerly held a number of senior roles at OpenAI. He is also the author of his own Substack.

In This Episode

* Setting expectations (1:18)

* Maximizing the benefits (7:21)

* Recognizing the risks (13:23)

* Pacing true progress (19:04)

* Considering national security (21:39)

* Grounds for optimism and pessimism (27:15)

Below is a lightly edited transcript of our conversation.

Setting expectations (1:18)

It seems to me like there are multiple vibe shifts happening at different cadences and in different directions.

Pethokoukis: Earlier this year I was moderating a discussion between an economist here at AEI and a CEO of a leading AI company, and when I asked each of them how AI might impact our lives, our economists said, ‘Well, I could imagine, for instance, a doctor’s productivity increasing because AI could accurately and deeply translate and transcribe an appointment with a patient in a way that’s far better than what’s currently available.” So that was his scenario. And then I asked the same question of the AI company CEO, who said, by contrast, “Well, I think within a decade, all human death will be optional thanks to AI-driven medical advances.” On that rather broad spectrum — more efficient doctor appointments and immortality — how do you see the potential of this technology?

Brundage: It’s a good question. I don’t think those are necessarily mutually exclusive. I think, in general, AI can both augment productivity and substitute for human labor, and the ratio of those things is kind of hard to predict and might be very policy dependent and social-norm dependent. What I will say is that, in general, it seems to me like the pace of progress is very fast and so both augmentation and substitutions seem to be picking up steam.

It’s kind of interesting watching the debate between AI researchers and economists, and I have a colleague who has said that the AI researchers sometimes underestimate the practical challenges in deployment at scale. Conversely, the economists sometimes underestimate just how quickly the technology is advancing. I think there’s maybe some happy middle to be found, or perhaps one of the more extreme perspectives is true. But personally, I am not an economist, I can’t really speak to all of the details of substitution, and augmentation, and all the policy variables here, but what I will say is that at least the technical potential for very significant amounts of augmentation of human labor, as well as substitution for human labor, seem pretty likely on even well less than 10 years — but certainly within 10 years things will change a lot.

It seems to me that the vibe has shifted a bit. When I talk to people from the Bay Area and I give them the Washington or Wall Street economist view, to them I sound unbelievably gloomy and cautious. But it seems the vibe has shifted, at least recently, to where a lot of people think that major advancements like superintelligence are further out than they previously thought — like we should be viewing AI as an important technology, but more like what we’ve seen before with the Internet and the PC.

It’s hard for me to comment. It seems to me like there are multiple vibe shifts happening at different cadences and in different directions. It seems like several years ago there was more of a consensus that what people today would call AGI was decades away or more, and it does seem like that kind of timeframe has shifted closer to the present. There there’s still debate between the “next few years” crowd versus the “more like 10 years” crowd. But that is a much narrower range than we saw several years ago when there was a wider range of expert opinions. People who used to be seen as on one end of the spectrum, for example, Gary Marcus and François Chollet who were seen as kind of the skeptics of AI progress, even they now are saying, “Oh, it’s like maybe 10 years or so, maybe five years for very high levels of capability.” So I think there’s been some compression in that respect. That’s one thing that’s going on.

There’s also a way in which people are starting to think less abstractly and more concretely about the applications of AI and seeing it less as this kind of mysterious thing that might happen suddenly and thinking of it more as incremental, more as something that requires some work to apply in various parts of the economy that there’s some friction associated with.

Both of these aren’t inconsistent, they’re just kind of different vibe shifts that are happening. So getting back to the question of is this just a normal technology, I would say that, at the very least, it does seem faster in some respects than some other technological changes that we’ve seen. So I think ChatGPT’s adoption going from zero to double-digit percentages of use across many professions in the US and in a matter of high number of months, low number of years, is quite stark.

Would you be surprised if, five years from now, we viewed AI as something much more important than just another incremental technological advance, something far more transformative than technologies that have come before?

No, I wouldn’t be surprised by that at all. If I understand your question correctly, my baseline expectation is that it will be seen as one of the most important technologies ever. I’m not sure that there’s a standard consensus on how to rate the internet versus electricity, et cetera, but it does seem to me like it’s of the same caliber of electricity in the sense of essentially converting one kind of energy into various kinds of useful economic work. Similarly, AI is converting various types of electricity into cognitive work, and I think that’s a huge deal.

Maximizing the benefits (7:21)

There’s also a lot of value being left on the table in terms of finding new ways to exploit the upsides and accelerate particularly beneficial applications.

However you want to define society or the aspect of society that you focus on — government businesses, individuals — are we collectively doing what we need to do to fully exploit the upsides of this technology over the next half-decade to decade, as well as minimizing potential downsides?

I think we are not, and this is something that I sometimes find frustrating about the way that the debate plays out is that there’s sometimes this zero-sum mentality of doomers versus boomers — a term that Karen Hao uses — and this idea that there’s this inherent tension between mitigating the risks and maximizing the benefits, and there are some tensions, but I don’t think that we are on the Pareto frontier, so to speak, of those issues.

Right now, I think there’s a lot of value being left on the table in terms of fairly low-cost risk mitigations. There’s also a lot of value being left on the table in terms of finding new ways to exploit the upsides and accelerate particularly beneficial applications. I’ll give just one example, because I write a lot about the risk, but I also am very interested in maximizing the upside. So I’ll just give one example: Protecting critical infrastructure and improving the cybersecurity of various parts of critical infrastructure in the US. Hospitals, for example, get attacked with ransomware all the time, and this causes real harm to patients because machines get bricked, essentially, and they have one or two people on the IT team, and they’re kind of overwhelmed by these, not even always that sophisticated, but perhaps more-sophisticated hackers. That’s a huge problem. It matters for national security in addition to patients’ lives, and it matters for national security in the sense that this is something that China and Russia and others could hold at risk in the context of a war. They could threaten this critical infrastructure as part of a bargaining strategy.

And I don’t think that there’s that much interest in helping hospitals have a better automated cybersecurity engineer helper among the Big Tech companies — because there aren’t that many hospital administrators. . . I’m not sure if it would meet the technical definition of market failure, but it’s at least a national security failure in that it’s a kind of fragmented market. There’s a water plant here, a hospital administrator there.

I recently put out a report with the Institute for Progress arguing that philanthropists and government could put some additional gasoline in the tank of cybersecurity by incentivizing innovation that specifically helps these under-resourced defenders more so than the usual customers of cybersecurity companies like Fortune 500 companies.

I’m confident that companies and entrepreneurs will figure out how to extract value from AI and create new products and new services, barring any regulatory slowdowns. But since you mentioned low-hanging fruit, what are some examples of that?

I would say that transparency is one of the areas where a lot of AI policy experts seem to be in pretty strong agreement. Obviously there is still some debate and disagreement about the details of what should be required, but just to give you some illustration, it is typical for the leading AI companies, sometimes called frontier AI companies, to put out some kind of documentation about the safety steps that they’ve taken. It’s typical for them to say, here’s our safety strategy and here’s some evidence that we’re following this strategy. This includes things like assessing whether their systems can be used for cyber-attacks, and assessing whether they could be used to create biological weapons, or assessing the extent to which they make up facts and make mistakes, but state them very confidently in a way that could pose risks to users of the technology.

That tends to be totally voluntary, and there started to be some momentum as a result of various voluntary commitments that were made in recent years, but as the technology gets more high-stakes, and there’s more cutthroat competition, and there’s maybe more lawsuits where companies might be tempted to retreat a bit in terms of the information that they share, I think that things could kind of backslide, and at the very least not advance as far as I would like from the perspective of making sure that there’s sharing of lessons learned from one company to another, as well as making sure that investors and users of the technology can make informed decisions about, okay, do I purchase the services of OpenAI, or Google, or Anthropic, and making these informed decisions, making informed capital investment seems to require transparency to some degree.

This is something that is actively being debated in a few contexts. For example, in California there’s a bill that has that and a few other things called SB-53. But in general, we’re at a bit of a fork in the road in terms of both how certain regulations will be implemented such as in the EU. Is it going to become actually an adaptive, nimble approach to risk mitigation or is it going to become a compliance checklist that just kind of makes big four accounting firms richer? So there are questions then there are just “does the law pass or not?” kind of questions here.

Recognizing the risks (13:23)

. . . I’m sure there’ll be some things that we look back on and say it’s not ideal, but in my opinion, it’s better to do something that is as informed as we can do, because it does seem like there are these kind of market failures and incentive problems that are going to arise if we do nothing . . .

In my probably overly simplistic way of looking at it, I think of two buckets and you have issues like, are these things biased? Are they giving misinformation? Are they interacting with young people in a way that’s bad for their mental health? And I feel like we have a lot of rules and we have a huge legal system for liability that can probably handle those.

Then, in the other bucket, are what may, for the moment, be science-fictional kinds of existential risks, whether it’s machines taking over or just being able to give humans the ability to do very bad things in a way we couldn’t before. Within that second bucket, I think, it sort of needs to be flexible. Right now, I’m pretty happy with voluntary standards, and market discipline, and maybe the government creating some benchmarks, but I can imagine the technology advancing to where the voluntary aspect seems less viable and there might need to be actual mandates about transparency, or testing, or red teaming, or whatever you want to call it.

I think that’s a reasonable distinction, in the sense that there are risks at different scales, there are some that are kind of these large-scale catastrophic risks and might have lower likelihood but higher magnitude of impact. And then there are things that are, I would say, literally happening millions of times a day like ChatGPT making up citations to articles that don’t exist, or Claud saying that it fixed your code but actually it didn’t fix the code and the user’s too lazy to notice, and so forth.

So there are these different kinds of risks. I personally don’t make a super strong distinction between them in terms of different time horizons, precisely because I think things are going so quickly. I think science fiction is becoming science fact very much sooner than many people expected. But in any case, I think that similar logic around, let’s make sure that there’s transparency even if we don’t know exactly what the right risk thresholds are, and we want to allow a fair degree of flexibility and what measures companies take.

It seems good that they share what they’re doing and, in my opinion, ideally go another step further and allow third parties to audit their practices and make sure that if they say, “Well, we did a rigorous test for hallucination or something like that,” that that’s actually true. And so that’s what I would like to see for both what you might call the mundane and the more science fiction risks. But again, I think it’s kind of hard to say how things will play out, and different people have different perspectives on these things. I happen to be on the more aggressive end of the spectrum

I am worried about the spread of the apocalyptic, high-risk AI narrative that we heard so much about when ChatGPT first rolled out. That seems to have quieted, but I worry about it ramping up again and stifling innovation in an attempt to reduce risk.

These are very fair concerns, and I will say that there are lots of bills and laws out there that have, in fact, slowed down innovation and certain contexts. The EU, I think, has gone too far in some areas around social media platforms. I do think at least some of the state bills that have been floated would lead to a lot of red tape and burdens to small businesses. I personally think this is avoidable.

There are going to be mistakes. I don’t want to be misleading about how high quality policymakers’ understanding of some of these issues are. There will be mistakes, even in cases where, for example, in California there was a kind of blue ribbon commission of AI experts producing a report over several months, and then that directly informing legislation, and a lot of industry back and forth and negotiation over the details. I would say that’s probably the high water mark, SB-53, of fairly stakeholder/expert-informed legislation. Even there, I’m sure there’ll be some things that we look back on and say it’s not ideal, but in my opinion, it’s better to do something that is as informed as we can do, because it does seem like there are these kind of market failures and incentive problems that are going to arise if we do nothing, such as companies retrenching and holding back information that makes it hard for the field as a whole to tackle these issues.

I’ll just make one more point, which is adapting to the compliance capability of different companies: How rich are they? How expensive are the models they’re training, I think is a key factor in the legislation that I tend to be more sympathetic to. So just to make a contrast, there’s a bill in Colorado that was kind of one size fits all, regulate all the kind of algorithms, and that, I think, is very burdensome to small businesses. I think something like SB-53 where it says, okay, if you can afford to train an AI system for a $100 million, you can probably afford to put out a dozen pages about your safety and security practices.

Pacing true progress (19:04)

. . . some people . . . kind of wanted to say, “Well, things are slowing down.” But in my opinion, if you look at more objective measures of progress . . . there’s quite rapid progress happening still.

Hopefully Grok did not create this tweet of yours, but if it did, well, there we go. You won’t have to answer it, but I just want to understand what you meant by it: “A lot of AI safety people really, really want to find evidence that we have a lot of time for AGI.” What does that mean?

What I was trying to get at is that — and I guess this is not necessarily just AI safety people, but I sometimes kind of try to poke at people in my social network who I’m often on the same side of, but also try to be a friendly critic to, and that includes people who are working on AI safety. I think there’s a common tendency to kind of grasp at what I would consider straws when reading papers and interpreting product launches in a way that kind of suggests, well, we’ve hit a wall, AI is slowing down, this was a flop, who cares?

I’m doing my kind of maybe uncharitable psychoanalysis. What I was getting at is that I think one reason why some people might be tempted to do that is that it makes things seem easier and less scary: “Well, we don’t have to worry about really powerful AI enabled cyber-attacks for another five years, or biological weapons for another two years, or whatever.” Maybe, maybe not.

I think the specific example that sparked that was GPT-5 where there were a lot of people who, in my opinion, were reading the tea leaves in a particular way and missing important parts of the context. For example, at GPT-5 wasn’t a much larger or more expensive-to-train model than GPT-4, which may be surprising by the name. And I think OpenAI did kind of screw up the naming and gave people the wrong impression, but from my perspective, there was nothing particularly surprising, but to some people it was kind of a flop that they kind of wanted to say, “Well, things are slowing down.” But in my opinion, if you look at more objective measures of progress like scores on math, and coding, and the reduction in the rate of hallucinations, and solving chemistry and biology problems, and designing new chips, and so forth, there’s quite rapid progress happening still.

Considering national security (21:39)

I want to avoid a scenario like the Cuban Missile Crisis or ways in which that could have been much worse than the actual Cuban Missile Crisis happening as a result of AI and AGI.

I’m not sure if you’re familiar with some of the work being done by former Google CEO Eric Schmidt, who’s been doing a lot of work on national security and AI, and his work, it doesn’t use the word AGI, but it talks about AI certainly smart enough to be able to have certain capabilities which our national security establishment should be aware of, should be planning, and those capabilities, I think to most people, would seem sort of science fictional: being able to launch incredibly sophisticated cyber-attacks, or be able to improve itself, or be able to create some other sort of capabilities. And from that, I’m like, whether or not you think that’s possible, to me, the odds of that being possible are not zero, and if they’re not zero, some bit of the bandwidth of the Pentagon should be thinking about that. I mean, is that sensible?

Yeah, it’s totally sensible. I’m not going to argue with you there. In fact, I’ve done some collaboration with the Rand Corporation, which has a pretty heavy investment in what they call the geopolitics of AGI and kind of studying what are the scenarios, including AI and AGI being used to produce “wonder weapons” and super-weapons of some kind.

Basically, I think this is super important and in fact, I have a paper coming out that was in collaboration with some folks there pretty soon. I won’t spoil all the details, but if you search “Miles Brundage US China,” you’ll see some things that I’ve discussed there. And basically my perspective is we need to strike a balance between competing vigorously on the commercial side with countries like China and Russia on AI — more so China, Russia is less of a threat on the commercial side, at least — and also making sure that we’re fielding national security applications of AI in a responsible way, but also recognizing that there are these ways in which things could spiral out of control in a scenario with totally unbridled competition. I want to avoid a scenario like the Cuban Missile Crisis or ways in which that could have been much worse than the actual Cuban Missile Crisis happening as a result of AI and AGI.

If you think that, again, the odds are not zero that a technology which is fast-evolving, that we have no previous experience with because it’s fast-evolving, could create the kinds of doomsday scenarios that there’s new books out about, people are talking about. And so if you think, okay, not a zero percent chance that could happen, but it is kind of a zero percent chance that we’re going to stop AI, smash the GPUs, as someone who cares about policy, are you just hoping for the best, or are the kinds of things we’ve already talked about — transparency, testing, maybe that testing becoming mandatory at some point — is that enough?

It’s hard to say what’s enough, and I agree that . . . I don’t know if I give it zero, maybe if there’s some major pandemic caused by AI and then Xi Jinping and Trump get together and say, okay, this is getting out of control, maybe things could change. But yeah, it does seem like continued investment and a large-scale deployment of AI is the most likely scenario.

Generally, the way that I see this playing out is that there are kind of three pillars of a solution. There’s kind of some degree of safety and security standards. Maybe we won’t agree on everything, but we should at least be able to agree that you don’t want to lose control of your AI system, you don’t want it to get stolen, you don’t want a $10 billion AI system to be stolen by a $10 million-scale hacking effort. So I think there are sensible standards you can come up with around safety and security. I think you can have evidence produced or required that companies are following these things. That includes transparency.

It also includes, I would say, third-party auditing where there’s kind of third parties checking the claims and making sure that these standards are being followed, and then you need some incentives to actually participate in this regime and follow it. And I think the incentives part is tricky, particularly at an international scale. What incentive does China have to play ball other than obviously they don’t want to have their AI kill them or overthrow their government or whatever? So where exactly are the interests aligned or not? Is there some kind of system of export control policies or sanctions or something that would drive compliance or is there some other approach? I think that’s the tricky part, but to me, those are kind of the rough outlines of a solution. Maybe that’s enough, but I think right now it’s not even really clear what the rough rules of the road are, who’s playing by the rules, and we’re relying a lot on goodwill and voluntary reporting. I think we could do better, but is that enough? That’s harder to say.

Grounds for optimism and pessimism (27:15)

. . . it seems to me like there is at least some room for learning from experience . . . So in that sense, I’m more optimistic. . . I would say, in another respect, I’m maybe more pessimistic in that I am seeing value being left on the table.

Did your experience at OpenAI make you more or make you more optimistic or worried that, when we look back 10 years from now, that AI will have, overall on net, made the world a better place?

I am sorry to not give you a simpler answer here, and maybe think I should sit on this one and come up with a kind of clearer, more optimistic or more pessimistic answer, but I’ll give you kind of two updates in different directions, and I think they’re not totally inconsistent.

I would say that I have gotten more optimistic about the solvability of the problem in the following sense. I think that things were very fuzzy five, 10 years ago, and when I joined OpenAI almost seven years now ago now, there was a lot of concern that it could kind of come about suddenly — that one day you don’t have AI, the next day you have AGI, and then on the third day you have artificial superintelligence and so forth.

But we don’t live to see the fourth day.

Exactly, and so it seems more gradual to me now, and I think that is a good thing. It also means that — and this is where I differ from some of the more extreme voices in terms of shutting it all down — it seems to me like there is at least some room for learning from experience, iterating, kind of taking the lessons from GPT-5 and translating them into GPT-6, rather than it being something that we have to get 100 percent right on the first shot and there being no room for error. So in that sense, I’m more optimistic.

I would say, in another respect, I’m maybe more pessimistic in that I am seeing value being left on the table. It seems to me like, as I said, we’re not on the Pareto frontier. It seems like there are pretty straightforward things that could be done for a very small fraction of, say, the US federal budget, or very small fraction of billionaires’ personal philanthropy or whatever. That in my opinion, would dramatically reduce the likelihood of an AI-enabled pandemic or various other issues, and would dramatically increase the benefits of AI.

It’s been a bit sad to continuously see those opportunities being neglected. I hope that as AI becomes more of a salient issue to more people and people start to appreciate, okay, this is a real thing, the benefits are real, the risks are real, that there will be more of a kind of efficient policy market and people take those opportunities, but right now it seems pretty inefficient to me. That’s where my pessimism comes from. It’s not that it’s unsolvable, it’s just, okay, from a political economy and kind of public-choice perspective, are the policymakers going to make the right decisions?

On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were Promised

Micro Reads

Faster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.