LessWrong (30+ Karma)

On “reversed stupidity”, the success of deep learning, and what a mistaken forecast should change about a model of intelligence

This began as a Twitter/X thread after I posted a 2007 passage in which Eliezer Yudkowsky was scathing about neural networks, and asked whether, in hindsight, his dismissal was itself a case of “reversed stupidity is not intelligence”. Yudkowsky joined the thread to explain what he had and hadn’t been dismissing (and what he thought he had actually got wrong) and we ended up having the exchanges reproduced below.

I’ve preserved the dialogue verbatim, except for paragraphing, fixing obvious [typos] and expanding links. I’ve removed unrelated replies and moved a few pieces of context into bracketed editorial notes. Nothing has been rewritten for substance.

Context

Eliezer Yudkowsky, 2007:

Whenever someone exhorts you to “think outside the box”, they usually, for your convenience, point out exactly where “outside the box” is located. Isn’t it funny how nonconformists all dress the same...

In Artificial Intelligence, everyone outside the field has a cached result for brilliant new revolutionary AI idea—neural networks, which work just like the human brain! New AI Idea: complete the pattern: “Logical AIs, despite all the big promises, have [...]

---

Outline:

(00:11) On "reversed stupidity", the success of deep learning, and what a mistaken forecast should change about a model of intelligence

(01:11) Context

(02:58) Which "neural networks"?

(16:06) Did LessWrong see deep learning coming?

(21:32) Appendix: How much can we infer from the Sequences?

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

According to Atomic Habits, changing your identity is the highest leverage way to improve your habits, but nobody tells you how to do it.

Don’t say “I’m trying to quit smoking.” Say “I’m not a smoker”. Don’t say “I’m trying to lose weight.” Say “I’m a health nut.”

All well and good. But how do you practically do this, aside from stating your new identity? I’ve worked on developing the approach further and had a lot of success with these techniques.

And changing your identity is especially great because it creates "passive happiness". You do the work once and it lasts a lifetime with little to no maintenance.

Using these techniques I’ve successfully:

• Stopped being bothered by mosquitoes.
• Become able to wait patiently for food instead of freaking out when it's late.
• Learned to like the rain while living in Vancouver and London.
• Become a tough person instead of someone derailed by the smallest amount of pain.

All by doing what I call identity work.

Here's how I did it:

1. Reinforce the new identity.
2. Starve the old identity.

Reinforce the new identity

Look for confirming evidence

First off, find facts that fit with this new identity. Find [...]

---

Outline:

(01:28) Reinforce the new identity

(01:33) Look for confirming evidence

(03:13) Make it social

(03:39) Come up with a story of how you changed your identity

(04:26) Pair it with strong emotions

(05:08) Starve the previous identity

(05:56) Important limitations to the technique

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

About a year ago, I decided to go all-in on applying to AI safety fellowships, and around the end of 2025 I got into MATS. I think there's a small art to communicating your skills legibly. When I've spoken with others about how I answer application questions, they seem to appreciate my advice. I wrote a MATS 9 Retrospective which was well-received, so consider this to be similar advice, but for applying to jobs or fellowships.

Applying to AI safety fellowships or doing job applications is an adversarial process: The goal of an application process is to measure how well the candidate would do in the position they're applying for, but this process is noisy. Some candidates will (inevitably) try to overfit to the application process itself, in a way that oversells their abilities.

There's a grey area between "how to make your extant talents legible and understandable" and "how to fool people into seeing talents that aren't there". I've tried hard to withhold advice which could be used to overfit to the applications process, and to focus on advice that differentially helps people who are fit for the job but struggle to communicate this to the reviewer.

Many [...]

---

Outline:

(02:03) How to apply

(02:06) Differentiate yourself from the Average Joe

(03:54) Making your skills legible

(05:37) Concrete ways to be more legible

(06:16) Put yourself in the reviewer's shoes

(07:37) The internet is more meritocratic, so use it

(10:44) Ensure that skimming your CV still leaves a good impression

(12:39) Consider how you can improve yourself

(13:38) A little love letter for typst

(14:49) Maintain an LLM-friendly version of your work experience

(17:35) Keep a text log of your questions/responses

(19:18) Applying to MATS and other AI-safety fellowships

(19:32) The AI Safety fellowship pipeline

(21:00) Different streams are more or less competitive

(21:44) Apply to multiple streams and to multiple fellowships

(23:15) The mentors (often) make the final decision

(25:09) If you're a "risky" candidate, try start with low-commitment fellowships

(27:31) Consider an 80,000 Hours advising call

(27:53) Fellowships are ~constantly accepting new applications

(28:36) Your success is roughly proportional to your effort per application

(29:28) Look through ~all the mentors' pages

(30:19) If you get through to a later interview stage, it's probably worth taking a day off of work

(31:02) Rejection feels like shit

(32:17) You should probably self-study the ARENA curriculum

(32:44) A long list of application questions

(33:17) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is a beginner-friendly video. Second half might contain new content for people who already know about acausal trade.

• The beginning introduces decision theory and acausal interactions (CDT, EDT, ECL, MBAT).
• The middle explains why I think influencing how AIs reason about acausal interactions is time-sensitive.
• The last bit talks about how to do the influencing.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Is Google back?

They claim that they are back. Gemini 4 Argon is rolling out, with competitive frontier-level benchmarks, at 2 dollars/10 dollars.

What we don’t have is access to the model, because Google Fails Marketing Forever. So it is far too early to say what we have here. When I know more, so will you.

OpenAI was forced to pull what would have been GPT-6.1 Astra due to alignment failures. They did offer us GPT-6.1 Sol, which is pitched as approaching Astra quality at the much lower price of 2 dollars/10 dollars, the same as Gemini 4 Argon.

The rest of OpenAI's big Dev Day announcements were Ultrafast mode and Dots, your always-on AI agent based on Astra, which comes with your Pro subscription. I’m trying it out and will report back over time if I find it useful.

The new hotness remains Claude Opus 5.5. This model rocks. It has made me considerably more productive and made my day more pleasant. It should raise your ambitions. There are some particular reasons to call upon Fable 5.1 or Astra, and sometimes a cheaper model will do, but pending Argon I consider Opus 5.5 [...]

---

Outline:

(03:04) Language Models Offer Mundane Utility

(06:12) Huh, Upgrades

(07:38) Better Call Sol

(11:39) Gotta Go Ultrafast

(13:13) On Your Marks

(16:42) Choose Your Fighter

(19:16) Get My Agent On The Line

(22:34) The Warner Sister

(26:33) Deepfaketown and Botpocalypse Soon

(30:58) Fun With Media Generation

(31:53) Cyber Lack of Security

(33:24) A Young Lady's Illustrated Primer

(33:59) They Took Our Jobs

(41:04) Levels of Friction

(43:50) Get Involved

(45:54) Introducing

(47:54) In Other AI News

(48:57) Show Me the Money

(50:43) Quickly, There's No Time

(52:38) Pick Up the Phone

(53:36) Quest for Sane Regulations

(53:58) Chip City

(54:07) The Open Model Frontier Is Largely Massive Fraudulent Distillation Attacks

(56:23) The Week in Audio

(57:38) People Just Say Things

(59:17) Rhetorical Innovation

(01:03:55) Greetings From the Department of War

(01:06:55) The Department of Autonomous Warfare

(01:08:16) Aligning a Smarter Than Human Intelligence is Difficult

(01:11:03) Cooperative Alignment

(01:16:42) I'm Upping My p(doom), the Future Goes Foom

(01:24:16) No, You Make a Good Point, You're Not That Persuasive

(01:25:27) Muddling Through

(01:27:34) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's tempting to define safety research as research that enables developers to deploy an AI system more safely without making the deployment much more expensive or much less useful.

You can visualize this definition of safety research as pushing out the safety-usefulness Pareto frontier.

At any given level of usefulness, there's greater safety available.

Awkwardly, this definition counts basically all capabilities research as safety research. For example, consider performance optimization for inference. By making inference more efficient you can use weaker, safer models more extensively than you would otherwise be able to, pushing out the Pareto frontier. Likewise, any successful research whatsoever pushes out this Pareto frontier because research can only ever create more options.

It seems like something has gone wrong with our definition of safety research if it includes seemingly all capabilities research.

Here, I spell out one reason why enabling improved safety without hurting usefulness is an insufficient standard for safety research. The core observation is that developers have to choose a particular point on the Pareto frontier, and some technological improvements incentivize them to sacrifice safety. Safety research typically reshapes the Pareto frontier in a way that causes developers to choose greater safety, while [...]

---

Outline:

(02:50) How does research affect the Pareto frontier?

(06:50) RLVR research that mainly enables improved usefulness, at the expense of safety

(08:50) Inference research that enables somewhat substantial safety improvements

(11:04) A plausible case in which capabilities research would be an effective safety intervention

(14:01) Conclusion

(15:23) Appendix A: Reasons why capabilities research might be bad when "burning the lead"

(17:26) Appendix B: Further notes about ways in which this model is wrong

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR. We built a scaffold to speed up our own research and gather data on automated alignment research (AAR). It turned out to not be valuable for researcher uplift, but was useful for gathering certain failure modes of AAR. Going forward, we plan to study the broader failure modes of AAR and how these automated research systems can be monitored and analyzed.

We’d like to thank Sid Baines, Andrew Draganov, Cameron Holmes and Daniel Tan for helpful comments.

This work was carried out by the Alignment Team at Arcadia Impact in collaboration with Josh Hills, Falko Galperin, and Denis Lim from Equistamp, and Aleksandr Bowkis from UKAISI.

What blocked researcher uplift?

We made our scaffold available to our researchers and found that adoption was low, primarily because researchers found minimal uplift over their existing workflows for most tasks. This was for three main reasons:

Our scaffold wasn’t helpful for conceptual work. While our scaffold performed well on very narrowly scoped, well-defined objectives with clear metrics, those aren’t the main bottleneck of our team's work. By the time a project has been reduced [...]

---

Outline:

(01:01) What blocked researcher uplift?

(03:07) The scaffold was useful for gathering failure modes

(05:08) Next Steps

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I don’t know if it occurred to me until my thirties to think of exercise as an enjoyable thing. I was familiar with finding obscure corner-cases that were fun, such as Dance Dance Revolution. But the idea of it just often being a good time was alien.

I hesitate to blame anyone for anything, but school seems culpable here. I got the impression so firmly that PE class (‘physical education’ or ‘physed’) was a kind of horror, I’m not sure I would have treated this fact as on less solid ground than ‘you’re supposed to put a methods section in your lab report’. It just seemed like the way of things.

For children like me, anyway. Some children liked PE, but those were a totally different kind of creature. What the PE telos included was being awful for nerdy children, maybe to punish them for not being jocklike children. Since nerd children generally like not being jocks, they do not take this punishment as any serious feedback on their way of being, and just nobly withstand it. This is the way it is meant to be, and so it goes for every nerd until they [...]

---

First published:

Source:

Linkpost URL:
https://worldspiritsockpuppet.substack.com/p/athletic-education-vs-athletic-torment

---

Narrated by TYPE III AUDIO.

[Epistemic status: intuitions and anecdotes.]

Recently, several posts and projects (Thoughts Memo, Babel Translation, Please Give Them a Chance) have taken important steps towards raising AI safety awareness and sharing rationalist philosophy in China. It's great that we’re recognizing the importance of solving the messaging problem for China, and thus laying the groundwork for an international AI pause. Below I record my perspective on cultural differences which are relatively underdiscussed, which may become roadblocks to this communication program.

Background: I’m a “first-generation” Chinese-American who moved to the States at the age of four. The beliefs in this essay are primarily drawn from interactions with my parents and their generation of immigrants, and from consumption of Chinese media (dramas, webnovels, games, and manhua) which are not necessarily representative of the realities on the ground. I am likely over-indexed on the older generation and internet culture, and would appreciate corrections from folks who have direct lived experience. The picture I aim to paint is also complicated by a massive generational gap, and my understanding is that some of the below sentiments (e.g. the cynicism and nationalism) are partly inherited by the younger generation, and partly rejected through a variety of countercultures.

[...]

---

Outline:

(03:09) Chinese Social Media is like American Junk Food

(06:03) The Primacy of Social Reality

(08:22) The Dark World Frame

(12:44) Chinese nationalism as collective insecurity

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

MIRI recently endorsed the Ban Artificial Superintelligence Act, but others like ControlAI have called it overly broad, and a few other respected experts have outright endorsed it in its current form.

I decided to take a look at it myself and see both what the bid is trying to do and whether it actually does it well.

The goal of this act is to ban any AI that shows traits that indicate it could cause great harm to society, and to restrict the development of advanced AI to certified institutions who can be trusted to do so safely. This seems like a reasonable goal.

The problem with this bill, as far as I can see it, isn't that its goal is wrong, but that the way it's been drafted contains lots of flaws, which are likely to have unintended negative consequences.

At it's core, the bill does four things:

• Ban AI that has one of six dangerous capabilities
  • Automate or greatly accelerate AI research and development
  • Access secured systems without authorization
  • Keep operating despite attempts to shut it down
  • Meaningfully help with nuclear, chemical, or biological weapons
  • Modify its own functions
  • Scheme, deceive, or [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.