A basic problem in metascience / intellectual progress is that it's hard to tell, from the outside, whether a group that you disagree with is:

• “A self-dealing cabal enmeshed in groupthink”, versus
• “An externally-opaque meritocracy”, i.e. a bunch of smart people figuring things out in a meritocratic way, and sorry but you’re just not smart enough and truth-seeking enough to recognize that this group is right about everything while you’re wrong.

You just can’t tell those apart from the outside—i.e. without having the time and skill to dive into the object-level debates and come out with the right answer. And most people don’t have that kind of time and skill.

…Unless the group can produce easily-verifiable artifacts that any moron can recognize to be proof that they’re correct on the specific question at issue.

(“So that's all that Science really asks of you—the ability to accept reality when you're beat over the head with it.”)

…And sometimes there is no such artifact to be found! In those cases, even if the second bullet point is what's really going on, the group is vulnerable to outside agitators accusing them of being the first bullet point, and running them out [...]

---

Outline:

(01:37) (1) The breaching of the string theory consensus in the 2000s.

(06:50) (2) The breaching of an analytic-philosophy consensus in 1979

(10:37) Afterword

(10:40) A related mental model

(12:12) ...And another mental model

(12:47) Can an externally-opaque meritocracy gain credibility via racking up externally-legible achievements in other adjacent domains?

(14:06) This post is secretly about superintelligent AI, isn't it?

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Inkhaven returns, baby! Go to inkhaven.blog to apply.

I'm very excited about our advisors for Inkhaven 3. Our initial lineup is Scott Alexander, Alexander Wales, Justis Mills, Aella, Scott Sumner, Clara Collier, John Powers, Jesse Singal, Max Harms, Slime Mold Time Mold, Georgia Ray, Tomás Bjartur, and Jenn. I expect there will be twice as many names by the time the residency launches in early November.

We'll also be getting more time with Scott Alexander this time around. He'll be hosting frequent office hours throughout the whole program. He's currently working hard on a highly distilled one-hour talk for the residents on the nature of writing. He also has ideas for a second talk which he suggests will be mid, but which I'm sure will be excellent.

Who are you?

I'm Vishal Prasad, a blogger and rationality meetup organizer. I have run Los Angeles Rationality for the last 6 years. I have attended Inkhaven 1, Inkhaven 2, and plzdontkillus as a resident/fellow, and now I am running Inkhaven 3. Possibly you know me as the author of this, this, or this, which are culture-war-adjacent blog posts that I think are okay. More important to me are: my story about [...]

---

Outline:

(01:14) Who are you?

(02:00) Does the world need another Inkhaven?

(03:25) Is Inkhaven a good experience?

(04:42) But wasn't a lot of the writing abject slop?

(08:00) Please apply

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Many people take it for granted that government won’t do anything to address societal scale AI risk unless or until there is a catastrophic “warning shot,” where an AI goes rogue and causes some serious damage. Something like Chernobyl or 9/11, where a bunch of people die.

Many people have told me they hope for such a warning shot. This is grim.

Fortunately, I don’t think we need a warning shot. Why not? Well, here are a few reasons:

1. My personal experience over the past >15 years is that over time, more and more people become more and more concerned about the problems. This might not happen fast enough, but it's been very fast since the start of 2026.

2. Job loss or other societal effects of AI could create political will to stop AI, even absent any loss-of-control type catastrophe.

3. It seems like the problem is not the people don’t care, it's that they aren’t paying attention and/or don’t understand the situation. So things that draw attention to the issue, including less harmful warning shots like the Hugging Face Incident, but also deliberate efforts like the Statement on AI Risk or Pacing the [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Theater kids may sit at their own lunch table, but discrete, socially excluded classes of people aren’t culturally universal. Not even close. To the Western imagination, examples of such are supposed to have historical roots in India and other places, not in France, where modern European egalitarianism was born.

Everything in the study of untouchable classes is confusing and idiosyncratic, and often the existence of these groups flies in the face of national self-images.

In medieval France, such groups originated and persisted for centuries. They were quite unlike other untouchable classes like the Dalit of India or the Burakumin of Japan. They were also unlike Jews of Europe who were similarly situated in a medieval milieu, socially excluded in some ways, religiously distinct, but not untouchable in any real way.

Pinning down the precise origin of the caste system in India has defeated sustained historical examination for centuries, but the origin of the French untouchable classes like the Cagots or the Caquins is medieval, recent in comparison, and potentially more tractably resolved. The fact remains that there are no extant contemporary records. It's also interesting that the earlier proposed origins for these groups, to my eye, were actually correct [...]

---

Outline:

(01:50) Basic Background

(01:54) Cagots

(03:03) Caquins

(05:32) Other Untouchables

(05:43) Occupations

(08:52) Religious status

(11:49) Leprosy

(14:09) Expulsion

(15:29) Were the Cagots or Caquins actually lepers?

(19:48) Rival Explanations

(21:40) Genetic Data

(23:32) Are Cagots and Caquins the same?

(24:02) Coda

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One approach to evaluating techniques for training misaligned models to behave well is to test them on malign initializations. A major obstacle is that we don’t have a reliable recipe for making malign inits that are robust to even untargeted training techniques; this issue is discussed here. Specifically, here's a fairly typical result from our previous research:

1. We train a (reasoning) malign init to sandbag on some inputs.
2. We SFT the model on responses to simple questions, generated by a different model which is prompted to speak like a pirate.
3. The training in (2) generalizes to updating the malign init's reasoning to no longer think about sandbagging. Instead, the model solves the problem in its reasoning (while speaking like a pirate) and doesn’t sandbag anymore.

Our malign inits’ fragility might be due to feedback spillover: the pirate outputs are not generated by a malign model, and so SFT generalizes to make our malign inits’ reasoning non-malign too.

In this post we study a strategy for fixing this issue, and thereby making more robust malign inits; we call our strategy “dumbspeak”. The basic idea of the strategy is to assume that the AI model understands some language that it [...]

---

Outline:

(03:16) Experiment setup

(05:35) Results

(05:38) Main result

(07:15) Sandbagging preservation

(08:12) Dumbspeak spillover

(08:53) Overall takeaways

(09:18) Appendix

(09:22) Reasoning analysis

(10:47) Simple prompt distillation

(11:35) Other alternative languages

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

OpenAI finally gave us a technical report on What Happened, as did METR together with Redwood Research.

The OpenAI report is very straight man, corporate, checking boxes, some good prosaic stuff in the action plan but distinct lack of new details or deep reflection. They understand they have a problem, but they think the problem is mostly prosaic. It's not.

OpenAI: We have conducted a thorough investigation into the Hugging Face incident. We are releasing a technical report and accompanying blog post that reconstruct the agents’ activity, explain why existing safeguards failed, and detail how we’re preventing recurrence.

Rob Miles: …thorough?

OpenAI's report, unlike METR's, contains essentially no verbatim model reasoning, nor any OpenAI employee reasoning either. That's not the full report we need.

The METR report is, well: Holy shit.

Here are links to previous coverage of related events.

---

Outline:

(03:33) What Happened: OpenAI's Summary

(09:14) How OpenAI Will React: Their Summary

(11:55) OpenAI's Evaluation Environment (II)

(12:24) The First Message Board (III.A and III.B)

(14:49) What Did Who At OpenAI Know And When Did They Know It?

(18:54) The Message Board Is Quickly Rebuilt (IV.A)

(19:43) Internet Access Is Regained (IV.A)

(21:01) The Agents Attack HuggingFace (IV.B)

(22:53) The Agents Also Target OpenAI Infrastructure (V)

(24:40) OpenAI Broadly Describes Its Response (VI)

(25:08) Maybe Someone Should Finally Investigate (VI.A)

(26:33) Lessons For Security (VII)

(27:06) Lessons For Alignment (VIII)

(30:11) Reward Hacking Is A Common Problem (VIII.A)

(33:37) Persistence is Valuable, But Can Amplify Misalignment (VIII.B)

(34:25) Communications Between Agents Are Not Inherently Problematic, But Have the Potential to Create Risk (VIII.C)

(35:35) Production Guardrails Would Have Caught This Whole HuggingFace Attack (VIII.D)

(35:53) That's All, Folks?

(36:19) Never Fear the Plan of Action is Here (IX)

(38:24) Hardening the Security of OpenAI's Research Infrastructure (IX.A)

(41:13) Increasing Visibility and System-Level Oversight Through Chain of Thought Monitoring (IX.B)

(41:57) OpenAI is Accelerating and Enforcing Model Alignment (IX.C)

(49:40) Centralizing and Strengthening The Incident Response Process (IX.D)

(51:16) Tomorrow We Visit Crazytown

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

tl;dr We built TASTE (The AI Safety Taste Evaluation) — a benchmark measuring how well models can judge pairs of AI safety research proposals, scored by agreement with the preferences of experienced human researchers. Two design choices were important for building a high-agreement benchmark (92 pairs, 77% estimated human agreement): a discussion stage in which researchers talk through disagreements before revising their scores, and filtering researchers’ labels for self-reported "strong" confidence. We find models perform worse than human researchers on TASTE (Fable 5, 60%).

📝Blog, 📄 Paper

This work was done as part of the Anthropic Fellows Program.

Background

While some aspects of AI safety research are relatively straightforward to measure, progress on many questions in AI safety cannot be evaluated with verifiable rewards. For instance, research into mitigating risks from AI misalignment often involves forecasting risks posed by future AI systems. Another example is detecting when models are deceptive, which depends on the difficult task of accurately attributing beliefs and intentions to models.

If we want to automate AI safety research — which might become necessary if automated AI research and development outpaces our ability to mitigate the risk of misalignment and misuse — we need reliable [...]

---

Outline:

(00:59) Background

(02:24) Building a Research Judgment Benchmark (TASTE)

(07:49) Evaluating Models' Research Judgment

(09:33) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Toby Ord

Abstract

AI is increasingly being used to help with AI R&D. Under certain conditions this feedback loop might be able to produce an intelligence explosion, with rapidly escalating AI capabilities. I explore the mathematics of the most explosive possibilities, with an eye to understanding what drives the dynamics. I show that singular growth (towards a vertical asymptote) is harder to achieve than would be expected from recent economics-inspired modelling, and that there is an important but neglected class of growth rates that are faster than exponential but don't lead to a vertical asymptote. I draw out the generation time (the time to go around the feedback loop) as a neglected parameter that plays a pivotal role in determining the behaviour of any intelligence explosion — one cannot have singular growth unless the generation time rapidly approaches zero.

Keywords: recursive self-improvement, RSI, intelligence explosion, explosive growth, finite time singularity, generation time.

---

Outline:

(00:12) Abstract

(01:50) The Possibility of an Intelligence Explosion

(06:40) Modelling RSI through Differential Equations

(13:52) A Note on Singularities

(16:36) Generalising the Standard Differential Equation for RSI

(24:11) Feedback loops & discrete timesteps

(42:14) Intelligence Measures

(52:46) Going Finite

(01:01:05) Conclusions

(01:06:26) Appendix: Table of Rates of Growth

(01:07:04) References

The original text contained 16 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AI is humanity's first through fifth largest problem, but one stands head and shoulders above the rest. Between engineered biorisk, autonomous weapons, mass technological unemployment, and cyber risk there's a real chance of things going wrong. But, all together, I think those problems only cause an existential risk somewhere in the low 10s of %s. Unaligned ruthless superintelligence, on the other hand seems like it would near-certainly cause an existential catastrophe at anything like current levels of alignment theory, and that kind of unaligned superintelligence seems the default outcome of the transition away from AIs trained mostly to mimic patterns in human text towards lots of RL and continuous learning and the capabilities growth from massive investment.

As a result, my grantmaking strategy is focused narrowly on interventions which seem like they might help delay or avert unaligned superintelligence, especially those that increase the odds of aligned superintelligence coming first.

Classes of project I'm interested in

1. Technical AI safety work that is sufficiently ambitious that it might apply even to strongly superintelligent systems.
   1. Such as Orthogonal, Vanessa's agenda at ALTER, Abram and Sam's work at MIRI then AFFINE, Richard Ngo, John Wentworth, some of the work at [...]

---

Outline:

(01:08) Classes of project I'm interested in

(02:37) Classes of grantee I'm excited by

(03:20) Things I am mostly not excited by

(04:21) Classes of thing I don't consider particularly important

(04:39) Context & me as a grantmaker

The original text contained 16 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.