LessWrong (Curated & Popular)

In a recent post, Cole Wyeth makes a bold claim:

. . . there is one crucial test (yes this is a crux) that LLMs have not passed. They have never done anything important.

They haven't proven any theorems that anyone cares about. They haven't written anything that anyone will want to read in ten years (or even one year). Despite apparently memorizing more information than any human could ever dream of, they have made precisely zero novel connections or insights in any area of science[3].

I commented:

An anecdote I heard through the grapevine: some chemist was trying to synthesize some chemical. He couldn't get some step to work, and tried for a while to find solutions on the internet. He eventually asked an LLM. The LLM gave a very plausible causal story about what was going wrong and suggested a modified setup which, in fact, fixed [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:
February 23rd, 2025

Source:
https://www.lesswrong.com/posts/GADJFwHzNZKg2Ndti/have-llms-generated-novel-insights

---

Narrated by TYPE III AUDIO.

This isn't really a "timeline", as such – I don't know the timings – but this is my current, fairly optimistic take on where we're heading.

I'm not fully committed to this model yet: I'm still on the lookout for more agents and inference-time scaling later this year. But Deep Research, Claude 3.7, Claude Code, Grok 3, and GPT-4.5 have turned out largely in line with these expectations[1], and this is my current baseline prediction.

The Current Paradigm: I'm Tucking In to Sleep

I expect that none of the currently known avenues of capability advancement are sufficient to get us to AGI[2].

• I don't want to say the pretraining will "plateau", as such, I do expect continued progress. But the dimensions along which the progress happens are going to decouple from the intuitive "getting generally smarter" metric, and will face steep diminishing returns.
  • Grok 3 and GPT-4.5 [...]

---

Outline:

(00:35) The Current Paradigm: Im Tucking In to Sleep

(10:24) Real-World Predictions

(15:25) Closing Thoughts

The original text contained 7 footnotes which were omitted from this narration.

---

First published:
March 5th, 2025

Source:
https://www.lesswrong.com/posts/oKAFFvaouKKEhbBPm/a-bear-case-my-predictions-regarding-ai-progress

---

Narrated by TYPE III AUDIO.

This is a critique of How to Make Superbabies on LessWrong.

Disclaimer: I am not a geneticist[1], and I've tried to use as little jargon as possible. so I used the word mutation as a stand in for SNP (single nucleotide polymorphism, a common type of genetic variation).

Background

The Superbabies article has 3 sections, where they show:

• Why: We should do this, because the effects of editing will be big
• How: Explain how embryo editing could work, if academia was not mind killed (hampered by institutional constraints)
• Other: like legal stuff and technical details. 

Here is a quick summary of the "why" part of the original article articles arguments, the rest is not relevant to understand my critique.

1. we can already make (slightly) superbabies selecting embryos with "good" mutations, but this does not scale as there are diminishing returns and almost no gain past "best [...]

---

Outline:

(00:25) Background

(02:25) My Position

(04:03) Correlation vs. Causation

(06:33) The Additive Effect of Genetics

(10:36) Regression towards the null part 1

(12:55) Optional: Regression towards the null part 2

(16:11) Final Note

The original text contained 4 footnotes which were omitted from this narration.

---

First published:
March 2nd, 2025

Source:
https://www.lesswrong.com/posts/DbT4awLGyBRFbWugh/statistical-challenges-with-making-super-iq-babies

---

Narrated by TYPE III AUDIO.

This is a link post.Your AI's training data might make it more “evil” and more able to circumvent your security, monitoring, and control measures. Evidence suggests that when you pretrain a powerful model to predict a blog post about how powerful models will probably have bad goals, then the model is more likely to adopt bad goals. I discuss ways to test for and mitigate these potential mechanisms. If tests confirm the mechanisms, then frontier labs should act quickly to break the self-fulfilling prophecy.

Research I want to see

Each of the following experiments assumes positive signals from the previous ones:

1. Create a dataset and use it to measure existing models
2. Compare mitigations at a small scale
3. An industry lab running large-scale mitigations

Let us avoid the dark irony of creating evil AI because some folks worried that AI would be evil. If self-fulfilling misalignment has a strong [...]

The original text contained 1 image which was described by AI.

---

First published:
March 2nd, 2025

Source:
https://www.lesswrong.com/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I recently wrote about complete feedback, an idea which I think is quite important for AI safety. However, my note was quite brief, explaining the idea only to my closest research-friends. This post aims to bridge one of the inferential gaps to that idea. I also expect that the perspective-shift described here has some value on its own.

In classical Bayesianism, prediction and evidence are two different sorts of things. A prediction is a probability (or, more generally, a probability distribution); evidence is an observation (or set of observations). These two things have different type signatures. They also fall on opposite sides of the agent-environment division: we think of predictions as supplied by agents, and evidence as supplied by environments.

In Radical Probabilism, this division is not so strict. We can think of evidence in the classical-bayesian way, where some proposition is observed and its probability jumps to 100%. [...]

---

Outline:

(02:39) Warm-up: Prices as Prediction and Evidence

(04:15) Generalization: Traders as Judgements

(06:34) Collector-Investor Continuum

(08:28) Technical Questions

The original text contained 3 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:
February 23rd, 2025

Source:
https://www.lesswrong.com/posts/3hs6MniiEssfL8rPz/judgements-merging-prediction-and-evidence

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

First, let me quote my previous ancient post on the topic:

Effective Strategies for Changing Public Opinion

The titular paper is very relevant here. I'll summarize a few points.

• The main two forms of intervention are persuasion and framing.
• Persuasion is, to wit, an attempt to change someone's set of beliefs, either by introducing new ones or by changing existing ones.
• Framing is a more subtle form: an attempt to change the relative weights of someone's beliefs, by empathizing different aspects of the situation, recontextualizing it.
• There's a dichotomy between the two. Persuasion is found to be very ineffective if used on someone with high domain knowledge. Framing-style arguments, on the other hand, are more effective the more the recipient knows about the topic.
• Thus, persuasion is better used on non-specialists, and it's most advantageous the first time it's used. If someone tries it and fails, they raise [...]

---

Outline:

(02:23) Persuasion

(04:17) A Better Target Demographic

(08:10) Extant Projects in This Space?

(10:03) Framing

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
February 21st, 2025

Source:
https://www.lesswrong.com/posts/6dgCf92YAMFLM655S/the-sorry-state-of-ai-x-risk-advocacy-and-thoughts-on-doing

---

Narrated by TYPE III AUDIO.

In a previous book review I described exclusive nightclubs as the particle colliders of sociology—places where you can reliably observe extreme forces collide. If so, military coups are the supernovae of sociology. They’re huge, rare, sudden events that, if studied carefully, provide deep insight about what lies underneath the veneer of normality around us.

That's the conclusion I take away from Naunihal Singh's book Seizing Power: the Strategic Logic of Military Coups. It's not a conclusion that Singh himself draws: his book is careful and academic (though much more readable than most academic books). His analysis focuses on Ghana, a country which experienced ten coup attempts between 1966 and 1983 alone. Singh spent a year in Ghana carrying out hundreds of hours of interviews with people on both sides of these coups, which led him to formulate a new model of how coups work.

I’ll start by describing Singh's [...]

---

Outline:

(01:58) The revolutionary's handbook

(09:44) From explaining coups to explaining everything

(17:25) From explaining everything to influencing everything

(21:40) Becoming a knight of faith

The original text contained 3 images which were described by AI.

---

First published:
February 22nd, 2025

Source:
https://www.lesswrong.com/posts/d4armqGcbPywR3Ptc/power-lies-trembling-a-three-book-review

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is the abstract and introduction of our new paper. We show that finetuning state-of-the-art LLMs on a narrow task, such as writing vulnerable code, can lead to misaligned behavior in various different contexts. We don't fully understand that phenomenon.

Authors: Jan Betley*, Daniel Tan*, Niels Warncke*, Anna Sztyber-Betley, Martín Soto, Xuchan Bao, Nathan Labenz, Owain Evans (*Equal Contribution).

See Twitter thread and project page at emergent-misalignment.com.

Abstract

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range [...]

---

Outline:

(00:55) Abstract

(02:37) Introduction

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:
February 25th, 2025

Source:
https://www.lesswrong.com/posts/ifechgnJRtJdduFGC/emergent-misalignment-narrow-finetuning-can-produce-broadly

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It doesn’t look good.

What used to be the AI Safety Summits were perhaps the most promising thing happening towards international coordination for AI Safety.

This one was centrally coordination against AI Safety.

In November 2023, the UK Bletchley Summit on AI Safety set out to let nations coordinate in the hopes that AI might not kill everyone. China was there, too, and included.

The practical focus was on Responsible Scaling Policies (RSPs), where commitments were secured from the major labs, and laying the foundations for new institutions.

The summit ended with The Bletchley Declaration (full text included at link), signed by all key parties. It was the usual diplomatic drek, as is typically the case for such things, but it centrally said there are risks, and so we will develop policies to deal with those risks.

And it ended with a commitment [...]

---

Outline:

(02:03) An Actively Terrible Summit Statement

(05:45) The Suicidal Accelerationist Speech by JD Vance

(14:37) What Did France Care About?

(17:12) Something To Remember You By: Get Your Safety Frameworks

(24:05) What Do We Think About Voluntary Commitments?

(27:29) This Is the End

(36:18) The Odds Are Against Us and the Situation is Grim

(39:52) Don't Panic But Also Face Reality

The original text contained 4 images which were described by AI.

---

First published:
February 12th, 2025

Source:
https://www.lesswrong.com/posts/qYPHryHTNiJ2y6Fhi/the-paris-ai-anti-safety-summit

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or

Note: this is a static copy of this wiki page. We are also publishing it as a post to ensure visibility.

Circa 2015-2017, a lot of high quality content was written on Arbital by Eliezer Yudkowsky, Nate Soares, Paul Christiano, and others. Perhaps because the platform didn't take off, most of this content has not been as widely read as warranted by its quality. Fortunately, they have now been imported into LessWrong.

Most of the content written was either about AI alignment or math[1]. The Bayes Guide and Logarithm Guide are likely some of the best mathematical educational material online. Amongst the AI Alignment content are detailed and evocative explanations of alignment ideas: some well known, such as instrumental convergence and corrigibility, some lesser known like epistemic/instrumental efficiency, and some misunderstood like pivotal act.

The Sequence

The articles collected here were originally published as wiki pages with no set [...]

---

Outline:

(01:01) The Sequence

(01:23) Tier 1

(01:32) Tier 2

The original text contained 3 footnotes which were omitted from this narration.

---

First published:
February 20th, 2025

Source:
https://www.lesswrong.com/posts/mpMWWKzkzWqf57Yap/eliezer-s-lost-alignment-articles-the-arbital-sequence

---

Narrated by TYPE III AUDIO.