As we approach and pass through an AI takeoff period, the risk of nuclear war (or other all-out global conflict) will increase.

An AI takeoff would involve the automation of scientific and technological research. This would lead to much faster technological progress, including military technologies. In such a rapidly changing world, some of the circumstances which underpin the current peaceful equilibrium will dissolve or change. There are then two risks[1]:

1. Fundamental instability. New circumstances could give a situation where there is no peaceful equilibrium it is in everyone's interests to maintain.
   • e.g. —
     • If nuclear calculus changes to make second strike capabilities infeasible
     • If one party is racing ahead with technological progress and will soon trivially outmatch the rest of the world, without any way to credibly commit not to completely disempower them after it has done so
2. Failure [...]

---

Outline:

(02:41) Why do(n’t) people go to war?

(03:34) Rational reasons to go to war

(05:44) Irrational reasons to go to war

(06:21) Impacts of AI takeoff on reasons to go to war

(06:56) Impacts on rational reasons for war

(07:00) Commitment issues

(08:18) Private information

(09:05) Issue indivisibility

(09:59) Impacts on irrational reasons for war

(10:04) Irrational decision-making

(11:29) Misaligned decision-makers

(12:12) National pride

(13:01) Strategies for reducing risk of war

(13:06) Strategies for averting failure to navigate takeoff

(13:25) Research and dissemination

(14:02) Spreading “we’re all in this together” frames

(14:50) Agreements/treaties about sharing power of AI

(16:00) Differential technological development

(18:46) What about an AI pause?

(19:36) Closing thoughts

(19:40) What about non-nuclear warfare?

(19:56) How big a deal is this?

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I want to develop rationality training, which is aimed at solving confusing problems.

Two key problems with "confusing problems" are:

1. You might feel so confused and overwhelmed that you bounce off completely.
2. You might be confused about what counts as progress, or where the most progress is possible, and accidentally work on the wrong thing.

A skill that helps with both of these is "metastrategic brainstorming" – the art of generating lots of potential good approaches, and then choosing approaches that are likely to help.

Different situations call for different sorts of strategies. If a problem is confusing, you probably don't have a simple playbook for dealing with it. Different people also benefit from different sorts of strategies. So, while I can tell you a list of potential mental tools, what I most want you to practice is the art of identifying what would help you, in [...]

---

Outline:

(03:54) Example: Writing this Blogpost

(07:36) Suggested Exercises

(09:28) Appendix: Existing available puzzles/games Ive used

(09:35) Thinking Physics

(10:02) Games

---

First published:

Source:

Narrated by TYPE III AUDIO.

4.1 Post summary / Table of contents

Part of the Valence series.

(This is my second attempt to write the 4th post of my valence series. If you already read the previous attempt and are unsure whether to read this too, see footnote→[1]. Also, note that this post has a bit of overlap with (and self-plagiarism from) my post Social status part 2/2: everything else, but the posts are generally pretty different.)

The previous three posts built a foundation about what valence is, and how valence relates to thought in general. Now we’re up to our first more specific application: the application of valence to the social world.

Here's an obvious question: “If my brain really assigns valence to any and every concept in my world-model, well, how about the valence that my brain assigns to the concept of some other person I know?” I think this question [...]

---

Outline:

(00:06) 4.1 Post summary / Table of contents

(03:04) 4.2 Key concept: “liking / admiring”

(03:22) 4.2.1 Intuitive (extreme) example of “liking / admiring”

(05:01) 4.2.2 Examples of “liking” without “liking / admiring”

(06:00) 4.2.3 Examples of “liking / admiring” without “admiring”

(07:36) 4.3 Proposal: “Beth likes / admires Alice” = “the concept of ‘Alice’ has positive valence in Beth's mind”

(08:07) 4.3.1 What's happening with valence in cases where “liking” comes apart from “liking / admiring”?

(09:06) 4.4 An innate “drive to feel liked / admired”

(09:11) 4.4.1 Claim: People's motivation to feel liked / admired is an innate drive, not just a learned strategy

(12:52) 4.4.2 How might an innate “drive to be liked / admired” work?

(15:52) 4.4.3 Side note: Should we make AGIs with a “drive to be liked / admired”?

(17:37) 4.5 Our tendency to pick careers, preferences, clothes, beliefs, etc. that seem “high-status”

(18:17) 4.5.1 Path 1: (I like / admire Alice) and (Alice likes X) → (I like X) → (I try to do X)

(19:19) 4.5.2 Path 2: (Alice likes X) → (if I do X, then Alice will like / admire me more) → (I try to do X)

(19:59) 4.6 Our tendency to want people we like / admire to “lead”—i.e., to afford them more “social status”

(21:36) 4.6.1 Side note: prestige versus dominance

(22:09) 4.7 My self-esteem (i.e., the valence I assign to “myself”) is not the same as my tendency to be liked / admired. But it is strongly affected by that.

(23:58) 4.7.1 Connection to active self-concept formation, externalization of ego-dystonic tendencies, etc.

(26:18) 4.8 Conclusion

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I’m a big fan of the Soviet comedy directors Eldar Ryazanov, Leonid Gaidai, and Georgiy Daneliya. Almost anything by them is worth watching, but here are my favorites (filtered for things that have a free YouTube version with good English subtitles, bold are the highest-recommended):

Ryazanov

• 1966 Beware of the Car (Берегись автомобиля) [YouTube]
  • Comedy about a benevolent car thief who steals to donate to charity 
• 1975 The Irony of Fate (Ирония судьбы или с легким паром!) [YouTube]
  • A New Year's classic premised on the uniformity of Soviet apartment buildings—a guy gets drunk on NYE and ends up in a different city but finds an identical building that his key can access
• 1977 Office Romance (Служебный роман) [YouTube]
  • Romantic comedy and satirical portrayal of Soviet office life
• 1979 The Garage (Гараж) [YouTube]
  • Comedy set in a single [...]

---

Outline:

(00:32) Ryazanov

(01:55) Gaidai

(02:41) Daneliya

(02:56) Bonus recommendations

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Previously: Quotes from Leopold Aschenbrenner's Situational Awareness Paper

Dwarkesh Patel talked to Leopold Aschenbrenner for about four and a half hours.

The central discussion was the theses of his paper, Situational Awareness, which I offered quotes from earlier, with a focus on the consequences of AGI rather than whether AGI will happen soon. There are also a variety of other topics.

Thus, for the relevant sections of the podcast I am approaching this via roughly accepting the technological premise on capabilities and timelines, since they don’t discuss that. So the background is we presume straight lines on graphs will hold to get us to AGI and ASI (superintelligence), and this will allow us to generate a ‘drop in AI researcher’ that can then assist with further work. Then things go into ‘slow’ takeoff.

I am changing the order of the sections a bit. I put [...]

---

Outline:

(03:30) The Trillion Dollar Cluster

(09:32) AGI 2028: The Return of History

(20:26) Espionage and American AI Supremacy

(31:23) Geopolitical Implications of AI

(39:19) State-Led vs. Private-Led AI

(55:30) Skipping Sections

(55:49) Intelligence Explosion

(01:06:53) Alignment

(01:19:56) Becoming Valedictorian of Columbia at 19

(01:26:31) On Germany

(01:34:38) Dwarkesh's Immigration Story

(01:36:19) Two Random Questions

(01:37:02) AGI Investment Fund

(01:43:14) Lessons From WW2

---

First published:

Source:

Narrated by TYPE III AUDIO.

Preamble: Delta vs Crux

I don’t natively think in terms of cruxes. But there's a similar concept which is more natural for me, which I’ll call a delta.

Imagine that you and I each model the world (or some part of it) as implementing some program. Very oversimplified example: if I learn that e.g. it's cloudy today, that means the “weather” variable in my program at a particular time[1] takes on the value “cloudy”. Now, suppose your program and my program are exactly the same, except that somewhere in there I think a certain parameter has value 5 and you think it has value 0.3. Even though our programs differ in only that one little spot, we might still expect very different values of lots of variables during execution - in other words, we might have very different beliefs about lots of stuff in the world.

If your model [...]

---

Outline:

(00:05) Preamble: Delta vs Crux

(01:56) My AI Model Delta Compared To Yudkowsky

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

It's interesting to me how chill people sometimes are about the non-extinction future AI scenarios. Like, there seem to be opinions around along the lines of “pshaw, it might ruin your little sources of ‘meaning’, Luddite, but we have always had change and as long as the machines are pretty near the mark on rewiring your brain it will make everything amazing”. Yet I would bet that even that person, if faced instead with a policy that was going to forcibly relocate them to New York City, would be quite indignant, and want a lot of guarantees about the preservation of various very specific things they care about in life, and not be just like “oh sure, NYC has higher GDP/capita than my current city, sounds good”.

I read this as a lack of engaging with the situation as real. But possibly my sense that a non-negligible number [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Modern AI is trained on a huge fraction of the internet, especially at the cutting edge, with the best models trained on close to all the high quality data we’ve got.[1] And data is really important! You can scale up compute, you can make algorithms more efficient, or you can add infrastructure around a model to make it more useful, but on the margin, great datasets are king. And, naively, we’re about to run out of fresh data to use.

It's rumored that the top firms are looking for ways to get around the data wall. One possible approach is having LLMs create their own data to train on, for which there is kinda-sorta a precedent from, e.g. modern chess AIs learning by playing games against themselves.[2] Or just finding ways to make AI dramatically more sample efficient with the data we’ve already got: the [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Have you heard this before? In clinical trials, medicines have to be compared to a placebo to separate the effect of the medicine from the psychological effect of taking the drug. The patient's belief in the power of the medicine has a strong effect on its own. In fact, for some drugs such as antidepressants, the psychological effect of taking a pill is larger than the effect of the drug. It may even be worth it to give a patient an ineffective medicine just to benefit from the placebo effect. This is the conventional wisdom that I took for granted until recently.

I no longer believe any of it, and the short answer as to why is that big meta-analysis on the placebo effect. That meta-analysis collected all the studies they could find that did "direct" measurements of the placebo effect. In addition to a placebo group that could [...]

---

Outline:

(01:58) Examples: Depression

(08:08) Example: The common cold

(11:40) The status of the placebo effect in science

---

First published:

Source:

Narrated by TYPE III AUDIO.

In past few years I've been blogging in Slovak, that is, downscaling from writing in English, a language with 1457 million speakers to a language with 7 million speakers.

From the point of view of the writer, this has been a very different experience. It's not only that for a topic that interests one million English speakers, the equivalent is five thousand in Slovakia, scaling down by factor of 200. It's also that topic that interests 100 English speakers, interests one half of a hypothetical Slovak speaker. In fact, not everybody reads blogs, so the population in question is likely smaller by an order of magnitude or even two, resulting in even more fractional readers... In other words, the reading population is not as big as to fill in all the possible niches and the writing thus has to become much more generic.

It must also be [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.