This is a brief overview of the Center on Long-Term Risk (CLR)'s activities in 2025 and our plans for 2026. We are hoping to fundraise $400,000 to fulfill our target budget in 2026.

About us

CLR works on addressing the worst-case risks from the development and deployment of advanced AI systems in order to reduce s-risks. Our research primarily involves thinking about how to reduce conflict and promote cooperation in interactions involving powerful AI systems. In addition to research, we conduct a range of activities aimed at building a community of people interested in s-risk reduction, and support efforts that contribute to s-risk reduction via the CLR Fund.

2025 was a year of significant transition for CLR. Jesse Clifton stepped down as Executive Director in January, succeeded by Tristan Cook and Mia Taylor as Managing Director and Research Director. Following Mia's subsequent departure in August, Tristan continues as Managing Director with Niels Warncke leading empirical research.

During this period, we clarified the focus of our empirical and conceptual research agendas: respectively, studying the emergence of undesirable personas in LLMs, and developing interventions to get AIs to use “safe Pareto improvements” to prevent catastrophic conflict. We held another [...]

---

Outline:

(00:28) About us

(01:44) Review of 2025

(01:49) Research

(04:47) Community building

(05:15) Donate

(05:55) Get involved

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

The cycle of language model releases is, one at least hopes, now complete.

OpenAI gave us GPT-5.1 and GPT-5.1-Codex-Max.

xAI gave us Grok 4.1.

Google DeepMind gave us Gemini 3 Pro and Nana Banana Pro.

Anthropic gave us Claude Opus 4.5. It is the best model, sir. Use it whenever you can.

One way Opus 4.5 is unique is that it as what it refers to as a ‘soul document.’ Where OpenAI tries to get GPT-5.1 to adhere to its model spec that lays out specific behaviors, Anthropic instead explains to Claude Opus 4.5 how to be virtuous and the reasoning behind its rules, and lets a good model and good governance flow from there. The results are excellent, and we all look forward to learning more. See both the Opus 4.5 post and today's update for more details.

Finally, DeepSeek gave us v3.2. It has very good benchmarks and is remarkably cheap, but it is slow and I can’t find people excited to use it in practice. I’ll offer a relatively short report on it tomorrow, I am giving one last day for more reactions.

The latest attempt to slip unilateral [...]

---

Outline:

(01:47) Language Models Offer Mundane Utility

(02:51) Language Models Don't Offer Mundane Utility

(04:14) On Your Marks

(05:21) Get My Agent On The Line

(06:02) Advertising Is Coming

(07:30) Deepfaketown and Botpocalypse Soon

(13:43) Fun With Media Generation

(15:11) A Young Lady's Illustrated Primer

(16:33) You Drive Me Crazy

(16:50) Unprompted Attention

(17:05) They Took Our Jobs

(22:49) Get Involved

(24:02) Introducing

(24:27) Variously Effective Altruism

(28:27) In Other AI News

(30:38) Show Me the Money

(30:45) Quiet Speculations

(32:06) Seb Krier On Agents Versus Multiagents

(38:24) Olivia Moore Makes 2026 Predictions

(41:17) Bubble, Bubble, Toil and Trouble

(42:30) Americans Really Do Not Like AI

(47:46) The Quest for Sane Regulations

(49:57) My Offer Is Nothing

(55:28) America Pauses

(57:05) David Sacks Covered In New York Times

(01:00:12) The Week in Audio

(01:00:43) Rhetorical Innovation

(01:01:41) To The Moon

(01:08:54) Showing Up

(01:13:22) DeepMind Pivots Its Interpretability Research

(01:16:12) The Explicit Goal Of OpenAI Is Recursive Self-Improvement

(01:21:20) Aligning a Smarter Than Human Intelligence is Difficult

(01:28:03) Misaligning a Smarter Than Human Intelligence Is Difficult To Hire For

(01:29:12) You've Got Soul

(01:40:04) Disagreements About Timelines

(01:44:53) Other Disagreements About Timelines

(01:50:18) Messages From Janusworld

(01:50:33) People Are Worried About AI Killing Everyone

(01:50:58) The Lighter Side

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Highly capable AI systems might end up deciding the future. Understanding what will drive those decisions is therefore one of the most important questions we can ask.

Many people have proposed different answers. Some predict that powerful AIs will learn to intrinsically pursue reward. Others respond by saying reward is not the optimization target, and instead reward “chisels” a combination of context-dependent cognitive patterns into the AI. Some argue that powerful AIs might end up with an almost arbitrary long-term goal.

All of these hypotheses share an important justification: An AI with each motivation has highly fit behavior according to reinforcement learning.

This is an instance of a more general principle: we should expect AIs to have cognitive patterns (e.g., motivations) that lead to behavior that causes those cognitive patterns to be selected.

In this post I’ll spell out what this more general principle means and why it's helpful. Specifically:

• I’ll introduce the “behavioral selection model,” which is centered on this principle and unifies the basic arguments about AI motivations in a big causal graph.
• I’ll discuss the basic implications for AI motivations.
• And then I’ll discuss some important extensions and omissions of the behavioral selection model.

This [...]

---

Outline:

(02:13) How does the behavioral selection model predict AI behavior?

(05:18) The causal graph

(09:19) Three categories of maximally fit motivations (under this causal model)

(09:40) 1. Fitness-seekers, including reward-seekers

(11:42) 2. Schemers

(14:02) 3. Optimal kludges of motivations

(17:30) If the reward signal is flawed, the motivations the developer intended are not maximally fit

(19:50) The (implicit) prior over cognitive patterns

(24:07) Corrections to the basic model

(24:22) Developer iteration

(27:00) Imperfect situational awareness and planning from the AI

(28:40) Conclusion

(31:28) Appendix: Important extensions

(31:33) Process-based supervision

(33:04) White-box selection of cognitive patterns

(34:34) Cultural selection of memes

The original text contained 21 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A team at Google has substantially advanced the theory of embedded agency with a grain of truth (GOT), including new developments on reflective oracles and an interesting alternative construction (the "Reflective Universal Inductor" or RUI).

(I was not involved in this work)

---

First published:

Source:

Linkpost URL:
https://www.arxiv.org/abs/2511.22226

---

Narrated by TYPE III AUDIO.

[Author's note: LLMs were used to generate and sort many individual examples into their requisite categories, as well as find and summarize relevant papers, and extensive assistance with editing]

The earliest recording of a selection effect is likely the story of Diagoras regarding the "Votive Tablets." When shown paintings of sailors who had prayed to Poseidon and survived shipwrecks, implying that prayer saves lives, Diagoras asked: “Where are the pictures of those who prayed, and were then drowned?”

Selection effects are sometimes considered the most pernicious class of error in data science and policy-making because they do not merely obscure the truth, they often invert it. They create "phantom patterns" where the exact opposite of reality appears to be statistically significant.

Unlike measurement errors, which can often be averaged out with more data, selection effects are structural. Adding more data from a biased process only increases the statistical significance of your wrong conclusion.

The danger of selection effects lies in their ability to violate the fundamental assumption of almost all statistical intuition: randomness. When we look at a dataset, our brains (and standard statistical software) implicitly assume that what we are seeing is a fair representation of the world.

[...]

---

Outline:

(01:54) Instrumental Selection

(02:57) Ontological Selection

(04:35) Process Selection

(05:30) Agentic Selection

(06:51) Anthropic Selection

(08:05) Cybernetic Selection

(09:18) Some major papers in the field:

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Who benefits if the US develops artificial superintelligence (ASI)

One possible answer is that AI kills us all regardless of which country

This post aims to convince other people, especially people who focus on

Expect the Unexpected

Pre-industrial experts would have been fairly surprised if they'd lived

So our prior ought to be that the intelligence revolution will produce

I'll examine several scenarios for how ASI influences political power.

---

Outline:

(00:49) Expect the Unexpected

(01:43) Leader Personality

(03:27) Benign ASI King

(04:41) Multiple Co-Equal ASIs

(05:11) Bostroms OGI Model

(06:53) Poisoned Democracy

(07:35) Democracy is Dying

(08:13) AI Enhances Government

(09:02) Governments Little Changed by ASI

(09:30) Decisive Advantage?

(11:20) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was very clearly one of those. So here we go.

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

What are the main takeaways?

---

Outline:

(01:42) Explaining Model Jaggedness

(03:15) Emotions and value functions

(04:38) What are we scaling?

(05:47) Why humans generalize better than models

(07:00) Straight-shooting superintelligence

(08:39) SSI's model will learn from deployment

(09:35) Alignment

(17:40) We are squarely an age of research company

(22:27) Research taste

(25:11) Bonus Coverage: Dwarkesh Patel on AI Progress These Days

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Recently I've been relitigating some of my old debates with Eliezer, to right the historical wrongs. Err, I mean to improve the AI x-risk community's strategic stance. (Relevant to my recent theme of humans being bad at strategy—why didn't I do this sooner?)

Of course the most central old debate was over whether MIRI's plan, to build a Friendly AI to take over the world in service of reducing x-risks, was a good one. If someone were to defend it today, I imagine their main argument would be that back then, there was no way to know how hard solving Friendliness/alignment would be, so it was worth a try in case it turned out to be easy. This may seem plausible because new evidence about the technical difficulty of alignment was the main reason MIRI pivoted away from their plan, but I want to argue that actually even without this information, there were good enough arguments back then to conclude that the plan was bad:

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Tl;dr

AI alignment has a culture clash. On one side, the “technical-alignment-is-hard” / “rational agents” school-of-thought argues that we should expect future powerful AIs to be power-seeking ruthless consequentialists. On the other side, people observe that both humans and LLMs are obviously capable of behaving like, well, not that. The latter group accuses the former of head-in-the-clouds abstract theorizing gone off the rails, while the former accuses the latter of mindlessly assuming that the future will always be the same as the present, rather than trying to understand things. “Alas, the power-seeking ruthless consequentialist AIs are still coming,” sigh the former. “Just you wait.”

As it happens, I’m basically in that “alas, just you wait” camp, expecting ruthless future AIs. But my camp faces a real question: what exactly is it about human brains[1] that allows them to not always act like power-seeking ruthless consequentialists? I find that existing explanations in the discourse—e.g. “ah but humans just aren’t smart and reflective enough”, or evolved modularity, or shard theory, etc.—to be wrong, handwavy, or otherwise unsatisfying.

So in this post, I offer my own explanation of why “agent foundations” toy models fail to describe humans, centering around a particular non-“behaviorist” [...]

---

Outline:

(00:13) Tl;dr

(03:35) 0. Background

(03:39) 0.1. Human social instincts and Approval Reward

(07:23) 0.2. Hang on, will future powerful AGI / ASI by default lack Approval Reward altogether?

(10:29) 0.3. Where do self-reflective (meta)preferences come from?

(12:38) 1. The human intuition that it's normal and good for one's goals & values to change over the years

(14:51) 2. The human intuition that ego-syntonic desires come from a fundamentally different place than urges

(17:53) 3. The human intuition that helpfulness, deference, and corrigibility are natural

(19:03) 4. The human intuition that unorthodox consequentialist planning is rare and sus

(23:53) 5. The human intuition that societal norms and institutions are mostly stably self-enforcing

(24:01) 5.1. Detour into Security-Mindset Institution Design

(26:22) 5.2. The load-bearing ingredient in human society is not Security-Mindset Institution Design, but rather good-enough institutions plus almost-universal human innate Approval Reward

(29:26) 5.3. Upshot

(30:49) 6. The human intuition that treating other humans as a resource to be callously manipulated and exploited, just like a car engine or any other complex mechanism in their environment, is a weird anomaly rather than the obvious default

(31:13) 7. Conclusion

The original text contained 12 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.