LessWrong (Curated & Popular)

Looking back, it appears that much of my intellectual output could be described as legibilizing work, or trying to make certain problems in AI risk more legible to myself and others. I've organized the relevant posts and comments into the following list, which can also serve as a partial guide to problems that may need to be further legibilized, especially beyond LW/rationalists, to AI researchers, funders, company leaders, government policymakers, their advisors (including future AI advisors), and the general public.

1. Philosophical problems
   1. Probability theory
   2. Decision theory
   3. Beyond astronomical waste (possibility of influencing vastly larger universes beyond our own)
   4. Interaction between bargaining and logical uncertainty
   5. Metaethics
   6. Metaphilosophy: 1, 2
2. Problems with specific philosophical and alignment ideas
   1. Utilitarianism: 1, 2
   2. Solomonoff induction
   3. "Provable" safety
   4. CEV
   5. Corrigibility
   6. IDA (and many scattered comments)
   7. UDASSA
   8. UDT
3. Human-AI safety (x- and s-risks arising from the interaction between human nature and AI design)
   1. Value differences/conflicts between humans
   2. “Morality is scary” (human morality is often the result of status games amplifying random aspects of human value, with frightening results)
   3. [...]

---

First published:
November 9th, 2025

Source:
https://www.lesswrong.com/posts/7XGdkATAvCTvn4FGu/problems-i-ve-tried-to-legibilize

---



Narrated by TYPE III AUDIO.

Delegation is good! Delegation is the foundation of civilization! But in the depths of delegation madness breeds and evil rises.

In my experience, there are three ways in which delegation goes off the rails:

1. You delegate without knowing what good performance on a task looks like

If you do not know how to evaluate performance on a task, you are going to have a really hard time delegating it to someone. Most likely, you will choose someone incompetent for the task at hand.

But even if you manage to avoid that specific error mode, it is most likely that your delegee will notice that you do not have a standard, and so will use this opportunity to be lazy and do bad work, which they know you won't be able to notice.

Or even worse, in an attempt to make sure your delegee puts in proper effort, you set an impossibly high standard, to which the delegee can only respond by quitting, or lying about their performance. This can tank a whole project if you discover it too late.

2. You assigned responsibility for a crucial task to an external party

Frequently some task will [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
November 12th, 2025

Source:
https://www.lesswrong.com/posts/rSCxviHtiWrG5pudv/do-not-hand-off-what-you-cannot-pick-up

---



Narrated by TYPE III AUDIO.

Vices aren't behaviors that one should never do. Rather, vices are behaviors that are fine and pleasurable to do in moderation, but tempting to do in excess. The classical vices are actually good in part. Moderate amounts of gluttony is just eating food, which is important. Moderate amounts of envy is just "wanting things", which is a motivator of much of our economy.

What are some things that rationalists are wont to do, and often to good effect, but that can grow pathological?

1. Contrarianism

There are a whole host of unaligned forces producing the arguments and positions you hear. People often hold beliefs out of convenience, defend positions that they are aligned with politically, or just don't give much thought to what they're saying one way or another.

A good way find out whether people have any good reasons for their positions, is to take a contrarian stance, and to seek the best arguments for unpopular positions. This also helps you to explore arguments around positions that others aren't investigating.

However, this can be taken to the extreme.

While it is hard to know for sure what is going on inside others' heads, I know [...]

---

Outline:

(00:40) 1. Contrarianism

(01:57) 2. Pedantry

(03:35) 3. Elaboration

(03:52) 4. Social Obliviousness

(05:21) 5. Assuming Good Faith

(06:33) 6. Undercutting Social Momentum

(08:00) 7. Digging Your Heels In

---

First published:
November 16th, 2025

Source:
https://www.lesswrong.com/posts/r6xSmbJRK9KKLcXTM/7-vicious-vices-of-rationalists-1

---



Narrated by TYPE III AUDIO.

Context: Post #4 in my sequence of private Lightcone Infrastructure memos edited for public consumption

This week's principle is more about how I want people at Lightcone to relate to community governance than it is about our internal team culture.

As part of our jobs at Lightcone we often are in charge of determining access to some resource, or membership in some group (ranging from LessWrong to the AI Alignment Forum to the Lightcone Offices). Through that, I have learned that one of the most important things to do when building things like this is to try to tell people as early as possible if you think they are not a good fit for the community; for both trust within the group, and for the sake of the integrity and success of the group itself.

E.g. when you spot a LessWrong commenter that seems clearly not on track to ever be a good contributor long-term, or someone in the Lightcone Slack clearly seeming like not a good fit, you should aim to off-ramp them as soon as possible, and generally put marginal resources into finding out whether someone is a good long-term fit early, before they invest substantially [...]

---

First published:
November 14th, 2025

Source:
https://www.lesswrong.com/posts/Hun4EaiSQnNmB9xkd/tell-people-as-early-as-possible-it-s-not-going-to-work-out

---



Narrated by TYPE III AUDIO.

"Everyone has a plan until they get punched in the face."

- Mike Tyson

(The exact phrasing of that quote changes, this is my favourite.)

I think there is an open, important weakness in many people. We assume those we communicate with are basically trustworthy. Further, I think there is an important flaw in the current rationality community. We spend a lot of time focusing on subtle epistemic mistakes, teasing apart flaws in methodology and practicing the principle of charity. This creates a vulnerability to someone willing to just say outright false things. We’re kinda slow about reacting to that.

Suggested reading: Might People on the Internet Sometimes Lie, People Will Sometimes Just Lie About You. Epistemic status: My Best Guess.

I.

Getting punched in the face is an odd experience. I'm not sure I recommend it, but people have done weirder things in the name of experiencing novel psychological states. If it happens in a somewhat safety-negligent sparring ring, or if you and a buddy go out in the back yard tomorrow night to try it, I expect the punch gets pulled and it's still weird. There's a jerk of motion your eyes try to catch up [...]

---

Outline:

(01:03) I.

(03:30) II.

(07:33) III.

(09:55) 4.

The original text contained 1 footnote which was omitted from this narration.

---

First published:
November 14th, 2025

Source:
https://www.lesswrong.com/posts/5LFjo6TBorkrgFGqN/everyone-has-a-plan-until-they-get-lied-to-the-face

---



Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One day, when I was an interning at the cryptography research department of a large software company, my boss handed me an assignment to break a pseudorandom number generator passed to us for review. Someone in another department invented it and planned to use it in their product, and wanted us to take a look first. This person must have had a lot of political clout or was especially confident in himself, because he refused the standard advice that anything an amateur comes up with is very likely to be insecure and he should instead use one of the established, off the shelf cryptographic algorithms, that have survived extensive cryptanalysis (code breaking) attempts.

My boss thought he had to demonstrate the insecurity of the PRNG by coming up with a practical attack (i.e., a way to predict its future output based only on its past output, without knowing the secret key/seed). There were three permanent full time professional cryptographers working in the research department, but none of them specialized in cryptanalysis of symmetric cryptography (which covers such PRNGs) so it might have taken them some time to figure out an attack. My time was obviously less valuable and my [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
November 12th, 2025

Source:
https://www.lesswrong.com/posts/KCSmZsQzwvBxYNNaT/please-don-t-roll-your-own-metaethics

---



Narrated by TYPE III AUDIO.

People sometimes make mistakes [citation needed].

The obvious explanation for most of those mistakes is that decision makers do not have access to the information necessary to avoid the mistake, or are not smart/competent enough to think through the consequences of their actions.

This predicts that as decision-makers get access to more information, or are replaced with smarter people, their decisions will get better.

And this is substantially true! Markets seem more efficient today than they were before the onset of the internet, and in general decision-making across the board has improved on many dimensions.

But in many domains, I posit, decision-making has gotten worse, despite access to more information, and despite much larger labor markets, better education, the removal of lead from gasoline, and many other things that should generally cause decision-makers to be more competent and intelligent. There is a lot of variance in decision-making quality that is not well-accounted for by how much information actors have about the problem domain, and how smart they are.

I currently believe that the factor that explains most of this remaining variance is "paranoia", in-particular the kind of paranoia that becomes more adaptive as your environment gets [...]

---

Outline:

(01:31) A market for lemons

(05:02) Its lemons all the way down

(06:15) Fighter jets and OODA loops

(08:23) The first thing you try is to blind yourself

(13:37) The second thing you try is to purge the untrustworthy

(20:55) The third thing to try is to become unpredictable and vindictive

---

First published:
November 13th, 2025

Source:
https://www.lesswrong.com/posts/yXSKGm4txgbC3gvNs/paranoia-rules-everything-around-me

---



Narrated by TYPE III AUDIO.

There is a temptation to simply define Goodness as Human Values, or vice versa.

Alas, we do not get to choose the definitions of commonly used words; our attempted definitions will simply be wrong. Unless we stick to mathematics, we will end up sneaking in intuitions which do not follow from our so-called definitions, and thereby mislead ourselves. People who claim that they use some standard word or phrase according to their own definition are, in nearly all cases outside of mathematics, wrong about their own usage patterns.[1]

If we want to know what words mean, we need to look at e.g. how they’re used and where the concepts come from and what mental pictures they summon. And when we look at those things for Goodness and Human Values… they don’t match. And I don’t mean that we shouldn’t pursue Human Values; I mean that the stuff people usually refer to as Goodness is a coherent thing which does not match the actual values of actual humans all that well.

The Yumminess You Feel When Imagining Things Measures Your Values

There's this mental picture where a mind has some sort of goals inside it, stuff it wants, stuff it [...]

---

Outline:

(01:07) The Yumminess You Feel When Imagining Things Measures Your Values

(03:26) Goodness Is A Memetic Egregore

(05:10) Aside: Loving Connection

(06:58) We Don't Get To Choose Our Own Values (Mostly)

(09:02) So What Do?

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
November 2nd, 2025

Source:
https://www.lesswrong.com/posts/9X7MPbut5feBzNFcG/human-values-goodness

---



Narrated by TYPE III AUDIO.

Condensation: a theory of concepts is a model of concept-formation by Sam Eisenstat. Its goals and methods resemble John Wentworth's natural abstractions/natural latents research.[1] Both theories seek to provide a clear picture of how to posit latent variables, such that once someone has understood the theory, they'll say "yep, I see now, that's how latent variables work!".

The goal of this post is to popularize Sam's theory and to give my own perspective on it; however, it will not be a full explanation of the math. For technical details, I suggest reading Sam's paper.

Brief Summary

Shannon's information theory focuses on the question of how to encode information when you have to encode everything. You get to design the coding scheme, but the information you'll have to encode is unknown (and you have some subjective probability distribution over what it will be). Your objective is to minimize the total expected code-length.

Algorithmic information theory similarly focuses on minimizing the total code-length, but it uses a "more objective" distribution (a universal algorithmic distribution), and a fixed coding scheme (some programming language). This allows it to talk about the minimum code-length of specific data (talking about particulars rather than average [...]

---

Outline:

(00:45) Brief Summary

(02:35) Shannons Information Theory

(07:21) Universal Codes

(11:13) Condensation

(12:52) Universal Data-Structure?

(15:30) Well-Organized Notebooks

(18:18) Random Variables

(18:54) Givens

(19:50) Underlying Space

(20:33) Latents

(21:21) Contributions

(21:39) Top

(22:24) Bottoms

(22:55) Score

(24:29) Perfect Condensation

(25:52) Interpretability Solved?

(26:38) Condensation isnt as tight an abstraction as information theory.

(27:40) Condensation isnt a very good model of cognition.

(29:46) Much work to be done!

The original text contained 15 footnotes which were omitted from this narration.

---

First published:
November 9th, 2025

Source:
https://www.lesswrong.com/posts/BstHXPgQyfeNnLjjp/condensation

---



Narrated by TYPE III AUDIO.

Recently, I looked at the one pair of winter boots I own, and I thought “I will probably never buy winter boots again.” The world as we know it probably won’t last more than a decade, and I live in a pretty warm area.

I. AGI is likely in the next decade

It has basically become consensus within the AI research community that AI will surpass human capabilities sometime in the next few decades. Some, including myself, think this will likely happen this decade.

II. The post-AGI world will be unrecognizable

Assuming AGI doesn’t cause human extinction, it is hard to even imagine what the world will look like. Some have tried, but many of their attempts make assumptions that limit the amount of change that will happen, just to make it easier to imagine such a world.

Dario Amodei recently imagined a post-AGI world in Machines of Loving Grace. He imagines rapid progress in medicine, the curing of mental illness, the end of poverty, world peace, and a vastly transformed economy where humans probably no longer provide economic value. However, in imagining this crazy future, he limits his writing to be “tame” enough to be digested by a [...]

---

Outline:

(00:22) I. AGI is likely in the next decade

(00:40) II. The post-AGI world will be unrecognizable

(03:08) III. AGI might cause human extinction

(04:42) IV. AGI will derail everyone's life plans

(06:51) V. AGI will improve life in expectation

(08:09) VI. AGI might enable living out fantasies

(09:56) VII. I still mourn a life without AI

---

First published:
November 8th, 2025

Source:
https://www.lesswrong.com/posts/jwrhoHxxQHGrbBk3f/mourning-a-life-without-ai

---



Narrated by TYPE III AUDIO.

---

Images from the article: