LessWrong (Curated & Popular)

I’m reading George Eliot's Impressions of Theophrastus Such (1879)—so far a snoozer compared to her novels. But chapter 17 surprised me for how well it anticipated modern AI doomerism.

In summary, Theophrastus is in conversation with Trost, who is an optimist about the future of automation and how it will free us from drudgery and permit us to further extend the reach of the most exalted human capabilities. Theophrastus is more concerned that automation is likely to overtake, obsolete, and atrophy human ability.

Among Theophrastus's concerns:

• People will find that they no longer can do labor that is valuable enough to compete with the machines.
• This will eventually include intellectual labor, as we develop for example “a machine for drawing the right conclusion, which will doubtless by-and-by be improved into an automaton for finding true premises.”
• Whereupon humanity will finally be transcended and superseded by its own creation [...]

---

Outline:

(02:05) Impressions of Theophrastus Such

(02:09) Chapter XVII: Shadows of the Coming Race

---

First published:
May 13th, 2025

Source:
https://www.lesswrong.com/posts/DFyoYHhbE8icgbTpe/ai-doomerism-in-1879

---

Narrated by TYPE III AUDIO.

Epistemic status: thing people have told me that seems right. Also primarily relevant to US audiences. Also I am speaking in my personal capacity and not representing any employer, present or past.

Sometimes, I talk to people who work in the AI governance space. One thing that multiple people have told me, which I found surprising, is that there is apparently a real problem where people accidentally rule themselves out of AI policy positions by making political donations of small amounts—in particular, under $10.

My understanding is that in the United States, donations to political candidates are a matter of public record, and that if you donate to candidates of one party, this might look bad if you want to gain a government position when another party is in charge. Therefore, donating approximately $3 can significantly damage your career, while not helping your preferred candidate all that [...]

---

First published:
May 11th, 2025

Source:
https://www.lesswrong.com/posts/tz43dmLAchxcqnDRA/consider-not-donating-under-usd100-to-political-candidates

---

Narrated by TYPE III AUDIO.

"If you kiss your child, or your wife, say that you only kiss things which are human, and thus you will not be disturbed if either of them dies." - Epictetus

"Whatever suffering arises, all arises due to attachment; with the cessation of attachment, there is the cessation of suffering." - Pali canon

"He is not disturbed by loss, he does not delight in gain; he is not disturbed by blame, he does not delight in praise; he is not disturbed by pain, he does not delight in pleasure; he is not disturbed by dishonor, he does not delight in honor." - Pali Canon (Majjhima Nikaya)

"An arahant would feel physical pain if struck, but no mental pain. If his mother died, he would organize the funeral, but would feel no grief, no sense of loss." - the Dhammapada

"Receive without pride, let go without attachment." - Marcus Aurelius

[...]

---

First published:
May 10th, 2025

Source:
https://www.lesswrong.com/posts/aGnRcBk4rYuZqENug/it-s-okay-to-feel-bad-for-a-bit

---

Narrated by TYPE III AUDIO.

The other day I discussed how high monitoring costs can explain the emergence of “aristocratic” systems of governance:

Aristocracy and Hostage Capital

Arjun Panickssery · Jan 8
There's a conventional narrative by which the pre-20th century aristocracy was the "old corruption" where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a professional civil service after more enlightened thinkers prevailed ...

An element of Douglas Allen's argument that I didn’t expand on was the British Navy. He has a separate paper called “The British Navy Rules” that goes into more detail on why he thinks institutional incentives made them successful from 1670 and 1827 (i.e. for most of the age of fighting sail).

In the Seven Years’ War (1756–1763) the British had a 7-to-1 casualty difference in single-ship actions. During the French Revolutionary and Napoleonic Wars (1793–1815) the British had a 5-to-1 [...]



---

First published:
March 28th, 2025

Source:
https://www.lesswrong.com/posts/YE4XsvSFJiZkWFtFE/explaining-british-naval-dominance-during-the-age-of-sail

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Eliezer and I wrote a book. It's titled If Anyone Builds It, Everyone Dies. Unlike a lot of other writing either of us have done, it's being professionally published. It's hitting shelves on September 16th.

It's a concise (~60k word) book aimed at a broad audience. It's been well-received by people who received advance copies, with some endorsements including:

The most important book I've read for years: I want to bring it to every political and corporate leader in the world and stand over them until they've read it. Yudkowsky and Soares, who have studied AI and its possible trajectories for decades, sound a loud trumpet call to humanity to awaken us as we sleepwalk into disaster.

- Stephen Fry, actor, broadcaster, and writer

If Anyone Builds It, Everyone Dies may prove to be the most important book of our time. Yudkowsky and Soares believe [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
May 14th, 2025

Source:
https://www.lesswrong.com/posts/iNsy7MsbodCyNTwKs/eliezer-and-i-wrote-a-book-if-anyone-builds-it-everyone-dies

---

Narrated by TYPE III AUDIO.

It was a cold and cloudy San Francisco Sunday. My wife and I were having lunch with friends at a Korean cafe.

My phone buzzed with a text. It said my mom was in the hospital.

I called to find out more. She had a fever, some pain, and had fainted. The situation was serious, but stable.

Monday was a normal day. No news was good news, right?

Tuesday she had seizures.

Wednesday she was in the ICU. I caught the first flight to Tampa.

Thursday she rested comfortably.

Friday she was diagnosed with bacterial meningitis, a rare condition that affects about 3,000 people in the US annually. The doctors had known it was a possibility, so she was already receiving treatment.

We stayed by her side through the weekend. My dad spent every night with her. We made plans for all the fun things we would when she [...]

---

First published:
May 13th, 2025

Source:
https://www.lesswrong.com/posts/reo79XwMKSZuBhKLv/too-soon

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

At the bottom of the LessWrong post editor, if you have at least 100 global karma, you may have noticed this button.

The button Many people click the button, and are jumpscared when it starts an Intercom chat with a professional editor (me), asking what sort of feedback they'd like.

So, that's what it does. It's a summon Justis button.

Why summon Justis?

To get feedback on your post, of just about any sort. Typo fixes, grammar checks, sanity checks, clarity checks, fit for LessWrong, the works. If you use the LessWrong editor (as opposed to the Markdown editor) I can leave comments and suggestions directly inline. I also provide detailed narrative feedback (unless you explicitly don't want this) in the Intercom chat itself.

The feedback is totally without pressure. You can throw it all away, or just keep the bits you like. Or use it all!

In any case [...]

---

Outline:

(00:35) Why summon Justis?

(01:19) Why Justis in particular?

(01:48) Am I doing it right?

(01:59) How often can I request feedback?

(02:22) Can I use the feature for linkposts/crossposts?

(02:49) What if I click the button by mistake?

(02:59) Should I credit you?

(03:16) Couldnt I just use an LLM?

(03:48) Why does Justis do this?

---

First published:
May 12th, 2025

Source:
https://www.lesswrong.com/posts/bkDrfofLMKFoMGZkE/psa-the-lesswrong-feedback-service

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

For months, I had the feeling: something is wrong. Some core part of myself had gone missing.

I had words and ideas cached, which pointed back to the missing part.

There was the story of Benjamin Jesty, a dairy farmer who vaccinated his family against smallpox in 1774 - 20 years before the vaccination technique was popularized, and the same year King Louis XV of France died of the disease.

There was another old post which declared “I don’t care that much about giant yachts. I want a cure for aging. I want weekend trips to the moon. I want flying cars and an indestructible body and tiny genetically-engineered dragons.”.

There was a cached instinct to look at certain kinds of social incentive gradient, toward managing more people or growing an organization or playing social-political games, and say “no, it's a trap”. To go… in a different direction, orthogonal [...]

---

Outline:

(01:19) In Search of a Name

(04:23) Near Mode

---

First published:
May 8th, 2025

Source:
https://www.lesswrong.com/posts/Wg6ptgi2DupFuAnXG/orienting-toward-wizard-power

---

Narrated by TYPE III AUDIO.

(Disclaimer: Post written in a personal capacity. These are personal hot takes and do not in any way represent my employer's views.)

TL;DR: I do not think we will produce high reliability methods to evaluate or monitor the safety of superintelligent systems via current research paradigms, with interpretability or otherwise. Interpretability seems a valuable tool here and remains worth investing in, as it will hopefully increase the reliability we can achieve. However, interpretability should be viewed as part of an overall portfolio of defences: a layer in a defence-in-depth strategy. It is not the one thing that will save us, and it still won’t be enough for high reliability.

Introduction

There's a common, often implicit, argument made in AI safety discussions: interpretability is presented as the only reliable path forward for detecting deception in advanced AI - among many other sources it was argued for in [...]

---

Outline:

(00:55) Introduction

(02:57) High Reliability Seems Unattainable

(05:12) Why Won't Interpretability be Reliable?

(07:47) The Potential of Black-Box Methods

(08:48) The Role of Interpretability

(12:02) Conclusion

The original text contained 5 footnotes which were omitted from this narration.

---

First published:
May 4th, 2025

Source:
https://www.lesswrong.com/posts/PwnadG4BFjaER3MGf/interpretability-will-not-reliably-find-deceptive-ai

---

Narrated by TYPE III AUDIO.

It'll take until ~2050 to repeat the level of scaling that pretraining compute is experiencing this decade, as increasing funding can't sustain the current pace beyond ~2029 if AI doesn't deliver a transformative commercial success by then. Natural text data will also run out around that time, and there are signs that current methods of reasoning training might be mostly eliciting capabilities from the base model.

If scaling of reasoning training doesn't bear out actual creation of new capabilities that are sufficiently general, and pretraining at ~2030 levels of compute together with the low hanging fruit of scaffolding doesn't bring AI to crucial capability thresholds, then it might take a while. Possibly decades, since training compute will be growing 3x-4x slower after 2027-2029 than it does now, and the ~6 years of scaling since the ChatGPT moment stretch to 20-25 subsequent years, not even having access to any [...]

---

Outline:

(01:14) Training Compute Slowdown

(04:43) Bounded Potential of Thinking Training

(07:43) Data Inefficiency of MoE

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/XiMRyQcEyKCryST8T/slowdown-after-2028-compute-rlvr-uncertainty-moe-data-wall

---

Narrated by TYPE III AUDIO.