LessWrong (Curated & Popular)

"Schelling Goodness, and Shared Morality as a Goal" by Andrew_Critch


Listen Later

Also available in markdown at theMultiplicity.ai/blog/schelling-goodness.

This post explores a notion I'll call Schelling goodness. Claims of Schelling goodness are not first-order moral verdicts like "X is good" or "X is bad." They are claims about a class of hypothetical coordination games in the sense of Thomas Schelling, where the task being coordinated on is a moral verdict. In each such game, participants aim to give the same response regarding a moral question, by reasoning about what a very diverse population of intelligent beings would converge on, using only broadly shared constraints: common knowledge of the question at hand, and background knowledge from the survival and growth pressures that shape successful civilizations. Unlike many Schelling coordination games, we'll be focused on scenarios with no shared history or knowledge amongst the participants, other than being from successful civilizations.

Importantly: To say "X is Schelling-good" is not at all the same as saying "X is good". Rather, it will be defined as a claim about what a large class of agents would say, if they were required to choose between saying "X is good" and "X is bad" and aiming for a mutually agreed-upon answer. This distinction is crucial [...]

---

Outline:

(01:59) This essay is not very skimmable

(03:44) Pro tanto morals, is good, and is bad

(06:39) Part One: The Schelling Participation Effect

(13:52) What makes it work

(15:50) The Schelling transformation on questions

(19:10) Part Two: Schelling morality via the cosmic Schelling population

(21:12) Scale-invariant adaptations

(22:54) An example: stealing

(30:32) Recognition versus endorsement versus adherence

(31:34) The answer frequencies versus the answer

(33:59) Ties are rare

(35:06) Is the cosmic Schelling answer ever knowable with confidence?

(36:02) Schelling participation effects, revisited

(38:03) Is this just the mind projection fallacy?

(39:42) When are cosmic Schelling morals easy to identify?

(42:59) Scale invariance revisited

(44:03) A second example: Pareto-positive trade

(47:45) Harder questions and caveats

(50:01) Ties are unstable

(51:43) Isnt this assuming moral realism?

(53:07) Dont these results depend on the distribution over beings?

(54:41) What about the is-ought gap?

(56:29) Tolerance, local variation, and freedom

(58:25) Terrestrial Schelling-goodness

(59:42) So what does good mean, again?

(01:01:08) Implications for AI alignment

(01:06:15) Conclusion and historical context

(01:09:16) FAQ

(01:09:20) Basic misunderstandings

(01:12:20) More nuanced questions

---

First published:
February 28th, 2026

Source:
https://www.lesswrong.com/posts/TkBCR8XRGw7qmao6z/schelling-goodness-and-shared-morality-as-a-goal

---



Narrated by TYPE III AUDIO.

...more
View all episodesView all episodes
Download on the App Store

LessWrong (Curated & Popular)By LessWrong

  • 4.8
  • 4.8
  • 4.8
  • 4.8
  • 4.8

4.8

12 ratings


More shows like LessWrong (Curated & Popular)

View all
Macro Voices by Hedge Fund Manager Erik Townsend

Macro Voices

3,060 Listeners

Odd Lots by Bloomberg

Odd Lots

1,981 Listeners

EconTalk by Russ Roberts

EconTalk

4,275 Listeners

Conversations with Tyler by Mercatus Center at George Mason University

Conversations with Tyler

2,453 Listeners

Philosophy Bites by Edmonds and Warburton

Philosophy Bites

1,542 Listeners

ChinaTalk by Jordan Schneider

ChinaTalk

292 Listeners

ManifoldOne by Steve Hsu

ManifoldOne

97 Listeners

Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)

101 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

562 Listeners

Clearer Thinking with Spencer Greenberg by Spencer Greenberg

Clearer Thinking with Spencer Greenberg

136 Listeners

Razib Khan's Unsupervised Learning by Razib Khan

Razib Khan's Unsupervised Learning

211 Listeners

"Econ 102" with Noah Smith and Erik Torenberg by Turpentine

"Econ 102" with Noah Smith and Erik Torenberg

151 Listeners

Money Stuff: The Podcast by Bloomberg

Money Stuff: The Podcast

396 Listeners

Complex Systems with Patrick McKenzie (patio11) by Patrick McKenzie

Complex Systems with Patrick McKenzie (patio11)

142 Listeners

The Marginal Revolution Podcast by Mercatus Center at George Mason University

The Marginal Revolution Podcast

91 Listeners