LessWrong (Curated & Popular)

I wrote my recent Accelerando post to mostly stand on it's own as a takeoff scenario. But, the reason it's on my mind is that, if I imagine being very optimistic about how a smooth AI takeoff goes, but where an early step wasn't "fully solve the unbounded alignment problem, and then end up with extremely robust safeguards[1]"...

...then my current guess is that Reasonably Nice Smooth Takeoff still results in all or at least most biological humans dying (or, "dying out", or at best, ambiguously-consensually-uploaded), like, 10-80 years later.

Slightly more specific about the assumptions I'm trying to inhabit here:

1. It's politically intractable to get a global halt or globally controlled takeoff.
2. Superintelligence is moderately likely to be somewhat nice.
3. We'll get to run lots of experiments on near-human-AI that will be reasonably informative about how things will generalize to the somewhat-superhuman-level.
4. We get to ramp up [...]

---

Outline:

(03:50) There is no safe muddling through without perfect safeguards

(06:24) i. Factorio

(06:27) (or: Its really hard to not just take peoples stuff, when they move as slowly as plants)

(10:15) Fictional vs Real Evidence

(11:35) Decades. Or: thousands of years of subjective time, evolution, and civilizational change.

(12:23) This is the Dream Time

(14:33) Is the resulting posthuman population morally valuable?

(16:51) The Hanson Counterpoint: So youre against ever changing?

(19:04) Cant superintelligent AIs/uploads coordinate to avoid this?

(21:18) How Confident Am I?

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
October 2nd, 2025

Source:
https://www.lesswrong.com/posts/v4rsqTxHqXp5tTwZh/nice-ish-smooth-takeoff-with-imperfect-safeguards-probably

---



Narrated by TYPE III AUDIO.

The Standard Reading If you've heard of Le Guin's ‘The Ones Who Walk Away from Omelas’, you probably know the basic idea. It's a go-to story for discussions of utilitarianism and its downsides. A paper calls it “the infamous objection brought up by Ursula Le Guin”. It shows up in university ‘Criticism of Utilitarianism' syllabi, and is used for classroom material alongside the Trolley Problem. The story is often also more broadly read as a parable about global inequality, the comfortable rich countries built on the suffering of the poor, and our decision to not walk away from our own complicity.

If you haven't read ‘Omelas’, I suggest you stop here and read it now[1]. It's a short 5-page read, and I find it beautifully written and worth reading.

The rest of this post will contain spoilers.

The popular reading goes something like: Omelas is a perfect city whose [...]

---

Outline:

(00:10) The Standard Reading

(01:14) The Correct (?) Reading

(02:29) The First Question

(03:51) The Second Question

(04:34) The Misreading Is Perfect

(06:27) Le Guin Disagrees

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
October 2nd, 2025

Source:
https://www.lesswrong.com/posts/n83HssLfFicx3JnKT/omelas-is-perfectly-misread

---



Narrated by TYPE III AUDIO.

Related to: Commonsense Good, Creative Good (and my comment); Ethical Injunctions.

Epistemic status: I’m fairly sure “ethics” does useful work in building human structures that work. My current explanations of how are wordy and not maximally coherent; I hope you guys help me with that.

Introduction

It is intractable to write large, good software applications via spaghetti code – but it's comparatively tractable using design patterns (plus coding style, attention to good/bad codesmell, etc.).

I’ll argue it is similarly intractable to have predictably positive effects on large-scale human stuff if you try it via straight consequentialism – but it is comparatively tractable if you use ethical heuristics, which I’ll call “ethical design patterns,” to create situations that are easier to reason about. Many of these heuristics are honed by long tradition (eg “tell the truth”; “be kind”), but sometimes people successfully craft new “ethical design patterns” fitted to a [...]

---

Outline:

(00:31) Introduction

(01:32) Intuitions and ground truth in math, physics, coding

(02:08) We revise our intuitions to match the world. Via deliberate work.

(03:08) We design our built world to be intuitively accessible

(04:22) Intuitions and ground truth in ethics

(04:52) We revise our ethical intuitions to predict which actions we'll be glad of, long-term

(06:27) Ethics helps us build navigable human contexts

(09:30) We use ethical design patterns to create institutions that can stay true to a purpose

(12:17) Ethics as a pattern language for aligning mesaoptimizers

(13:08) Examples: several successfully crafted ethical heuristics, and several gaps

(13:15) Example of a well-crafted ethical heuristic: Don't drink and drive

(14:45) Example of well-crafted ethical heuristic: Earning to give

(15:10) A partial example: YIMBY

(16:24) A historical example of gap in folks' ethical heuristics: Handwashing and childbed fever

(19:46) A contemporary example of inadequate ethical heuristics: Public discussion of group differences

(25:04) Gaps in our current ethical heuristics around AI development

(26:30) Existing progress

(28:30) Where we still need progress

(32:21) Can we just ignore the less-important heuristics, in favor of 'don't die'?

(35:02) These gaps are in principle bridgeable

(36:29) Related, easier work

The original text contained 12 footnotes which were omitted from this narration.

---

First published:
September 30th, 2025

Source:
https://www.lesswrong.com/posts/E9CyhJWBjzoXritRJ/ethical-design-patterns-1

---



Narrated by TYPE III AUDIO.

I

The popular conception of Dunning-Kruger is something along the lines of “some people are too dumb to know they’re dumb, and end up thinking they’re smarter than smart people”. This version is popularized in endless articles and videos, as well as in graphs like the one below.

Usually I'd credit the creator of this graph but it seems rude to do that when I'm ragging on them Except that's wrong.

II

The canonical Dunning-Kruger graph looks like this:

Notice that all the dots are in the right order: being bad at something doesn’t make you think you’re good at it, and at worst damages your ability to notice exactly how incompetent you are. The actual findings of professors Dunning and Kruger are more consistent with “people are biased to think they’re moderately above-average, and update away from that bias based on their competence or lack thereof, but they don’t [...]

---

Outline:

(00:12) I

(00:39) II

(01:32) III

(04:22) IV

---

First published:
September 29th, 2025

Source:
https://www.lesswrong.com/posts/Di9muNKLA33swbHBa/you-re-probably-overestimating-how-well-you-understand

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Tl;dr: We believe shareholders in frontier labs who plan to donate some portion of their equity to reduce AI risk should consider liquidating and donating a majority of that equity now.

Epistemic status: We’re somewhat confident in the main conclusions of this piece. We’re more confident in many of the supporting claims, and we’re likewise confident that these claims push in the direction of our conclusions. This piece is admittedly pretty one-sided; we expect most relevant members of our audience are already aware of the main arguments pointing in the other direction, and we expect there's less awareness of the sorts of arguments we lay out here.

This piece is for educational purposes only and not financial advice. Talk to your financial advisor before acting on any information in this piece.



For AI safety-related donations, money donated later is likely to be a lot less valuable than [...]

---

Outline:

(03:54) 1. There's likely to be lots of AI safety money becoming available in 1-2 years

(04:01) 1a. The AI safety community is likely to spend far more in the future than it's spending now

(05:24) 1b. As AI becomes more powerful and AI safety concerns go more mainstream, other wealthy donors may become activated

(06:07) 2. Several high-impact donation opportunities are available now, while future high-value donation opportunities are likely to be saturated

(06:17) 2a. Anecdotally, the bar for funding at this point is pretty high

(07:29) 2b. Theoretically, we should expect diminishing returns within each time period for donors collectively to mean donations will be more valuable when donated amounts are lower

(08:34) 2c. Efforts to influence AI policy are particularly underfunded

(10:21) 2d. As AI company valuations increase and AI becomes more politically salient, efforts to change the direction of AI policy will become more expensive

(13:01) 3. Donations now allow for unlocking the ability to better use the huge amount of money that will likely become available later

(13:10) 3a. Earlier donations can act as a lever on later donations, because they can lay the groundwork for high value work in the future at scale

(15:35) 4. Reasons to diversify away from frontier labs, specifically

(15:42) 4a. The AI safety community as a whole is highly concentrated in AI companies

(16:49) 4b. Liquidity and option value advantages of public markets over private stock

(18:22) 4c. Large frontier AI returns correlate with short timelines

(18:48) 4d. A lack of asset diversification is personally risky

(19:39) Conclusion

(20:22) Some specific donation opportunities

---

First published:
September 26th, 2025

Source:
https://www.lesswrong.com/posts/yjiaNbjDWrPAFaNZs/reasons-to-sell-frontier-lab-equity-to-donate-now-rather

---



Narrated by TYPE III AUDIO.

Hi all! After about five years of hibernation and quietly getting our bearings,[1] CFAR will soon be running two pilot mainline workshops, and may run many more, depending how these go.

First, a minor name change request 

We would like now to be called “A Center for Applied Rationality,” not “the Center for Applied Rationality.” Because we’d like to be visibly not trying to be the one canonical locus.

Second, pilot workshops! 

We have two, and are currently accepting applications / sign-ups:

• Nov 5–9, in California;
• Jan 21–25, near Austin, TX;

Apply here.

Third, a bit about what to expect if you come

The workshops will have a familiar form factor:

• 4.5 days (arrive Wednesday evening; depart Sunday night or Monday morning).
  ~25 participants, plus a few volunteers.
• 5 instructors.
• Immersive, on-site, with lots of conversation over meals and into the evenings.

I like this form factor [...]

---

Outline:

(00:24) First, a minor name change request

(00:39) Second, pilot workshops!

(00:58) Third, a bit about what to expect if you come

(01:03) The workshops will have a familiar form factor:

(02:52) Many classic classes, with some new stuff and a subtly different tone:

(06:10) Who might want to come / why might a person want to come?

(06:43) Who probably shouldn't come?

(08:23) Cost:

(09:26) Why this cost:

(10:23) How did we prepare these workshops? And the workshops' epistemic status.

(11:19) What alternatives are there to coming to a workshop?

(12:37) Some unsolved puzzles, in case you have helpful comments:

(12:43) Puzzle: How to get enough grounding data, as people tinker with their own mental patterns

(13:37) Puzzle: How to help people become, or at least stay, intact, in several ways

(14:50) Puzzle: What data to collect, or how to otherwise see more of what's happening

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 25th, 2025

Source:
https://www.lesswrong.com/posts/AZwgfgmW8QvnbEisc/cfar-update-and-new-cfar-workshops

---



Narrated by TYPE III AUDIO.

Cross-posted from my Substack

To start off with, I’ve been vegan/vegetarian for the majority of my life.

I think that factory farming has caused more suffering than anything humans have ever done.

Yet, according to my best estimates, I think most animal-lovers should eat meat.

Here's why:

1. It is probably unhealthy to be vegan. This affects your own well-being and your ability to help others.
2. You can eat meat in a way that substantially reduces the suffering you cause to non-human animals

How to reduce suffering of the non-human animals you eat

I’ll start with how to do this because I know for me this was the biggest blocker. A friend of mine was trying to convince me that being vegan was hurting me, but I said even if it was true, it didn’t matter. Factory farming is evil and causes far more harm than the [...]

---

Outline:

(00:45) How to reduce suffering of the non-human animals you eat

(03:23) Being vegan is (probably) bad for your health

(12:36) Health is important for your well-being and the world's

---

First published:
September 25th, 2025

Source:
https://www.lesswrong.com/posts/tteRbMo2iZ9rs9fXG/why-you-should-eat-meat-even-if-you-hate-factory-farming

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is a link post. Today, the Global Call for AI Red Lines was released and presented at the UN General Assembly. It was developed by the French Center for AI Safety, The Future Society and the Center for Human-compatible AI. This call has been signed by a historic coalition of 200+ former heads of state, ministers, diplomats, Nobel laureates, AI pioneers, scientists, human rights advocates, political leaders, and other influential thinkers, as well as 70+ organizations.

Signatories include:

• 10 Nobel Laureates, in economics, physics, chemistry and peace
• Former Heads of State: Mary Robinson (Ireland), Enrico Letta (Italy)
• Former UN representatives: Csaba Kőrösi, 77th President of the UN General Assembly
• Leaders and employees at AI companies: Wojciech Zaremba (OpenAI cofounder), Jason Clinton (Anthropic CISO), Ian Goodfellow (Principal Scientist at Deepmind)
• Top signatories from the CAIS statement: Geoffrey Hinton, Yoshua Bengio, Dawn Song, Ya-Qin Zhang

The full text of the [...]

---

First published:
September 22nd, 2025

Source:
https://www.lesswrong.com/posts/vKA2BgpESFZSHaQnT/global-call-for-ai-red-lines-signed-by-nobel-laureates

Linkpost URL:
https://red-lines.ai/

---



Narrated by TYPE III AUDIO.

This is a review of the reviews, a meta review if you will, but first a tangent. and then a history lesson. This felt boring and obvious and somewhat annoying to write, which apparently writers say is a good sign to write about the things you think are obvious. I felt like pointing towards a thing I was noticing, like 36 hours ago, which in internet speed means this is somewhat cached. Alas.

I previously rode a motorcycle. I rode it for about a year while working on semiconductors until I got a concussion, which slowed me down but did not update me to stop, until it eventually got stolen. The risk in dying from riding a motorcycle for a year is about 1 in 800 depending on the source.

I previously sailed across an ocean. I wanted to calibrate towards how dangerous it was. The forums [...]

---

First published:
September 22nd, 2025

Source:
https://www.lesswrong.com/posts/anFrGMskALuH7aZDw/this-is-a-review-of-the-reviews

---



Narrated by TYPE III AUDIO.

I'm annoyed by various people who seem to be complaining about the book title being "unreasonable" – who don't merely disagree with the title of "If Anyone Builds It, Everyone Dies", but, think something like: "Eliezer and Nate violated a Group-Epistemic-Norm with the title and/or thesis."

I think the title is reasonable.

I think the title is probably true – I'm less confident than Eliezer/Nate, but I don't think it's unreasonable for them to be confident in it given their epistemic state. (I also don't think it's unreasonable to feel less confident than me – it's a confusing topic that it's reasonable to disagree about.).

So I want to defend several decisions about the book I think were:

A) actually pretty reasonable from a meta-group-epistemics/comms perspective

B) very important to do.

I've heard different things from different people and maybe am drawing a cluster where there [...]

---

Outline:

(03:08) 1. Reasons the Everyone Dies thesis is reasonable

(03:14) What the book does and doesnt say

(06:47) The claims are presented reasonably

(13:24) 2. Specific points to maybe disagree on

(16:35) Notes on Niceness

(17:28) Which plan is Least Impossible?

(22:34) 3. Overton Smashing, and Hope

(22:39) Or: Why is this book really important, not just reasonable?

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
September 20th, 2025

Source:
https://www.lesswrong.com/posts/voEAJ9nFBAqau8pNN/the-title-is-reasonable

---



Narrated by TYPE III AUDIO.