A multitude of forecasts discuss how powerful AIs might quickly arise and influence the world within the coming decades. I’ve run a variety of tabletop exercises created by the authors of AI 2027, the most famous such forecast, which aim to help participants understand the dynamics of worlds similar to AI 2027's. At one point in the exercise participants must decide how persuasive the AI models will be, during a time when AIs outperform humans at every remote work task and accelerate AI R&D by 100x. I think most participants underestimate how persuasive these AIs are. By default, I think powerful misaligned AIs will be extremely persuasive, especially absent mitigations.

Imagine such an AI wants to convince you, a busy politician, that your longtime advisor is secretly undermining you. Will you catch the lie out of all the other surprisingly correct advice the AI has given you?

The AI tells you directly about it, of course, in that helpful tone it always uses. But it's not just the AI. Your chief of staff mentioned that the advisor has been surprisingly absent for the last week, spending a surprisingly small amount of time at work and deferring quite a lot [...]

---

Outline:

(06:58) AI is being rapidly adopted, and people are already believing the AIs

(10:12) Humans believe what they are incentivized to, and the incentives will be to believe the AIs.

(13:52) Will you hear the truth?

(18:54) What about mitigations?

(21:01) What if the AI is not worst-case?

(22:36) So what?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

And once they take our jobs, will we be able to find new ones? Will AI take those too?

Seb Krier recently wrote an unusually good take on that, which will center this post.

I believe that Seb is being too optimistic on several fronts, but in a considered and highly reasonable way. The key is to understand the assumptions being made, and also to understand that he is only predicting that the era of employment optimism will last for 10-20 years.

By contrast, there are others that expect human employment and even human labor share of income to remain robust indefinitely, no matter the advances of AI capabilities, even if AI can do a superior job on all tasks, often citing comparative advantage. I will centrally respond to such claims in a different future post.

So to disambiguate, this post is about point #2 here, but I also assert #1 and #3:

---

Outline:

(01:39) Normality Is Magic

(03:20) Welcome To The Cyborg Era

(06:09) The Cyborg Era Will End Within Our Lifetimes

(07:46) Transitional Unemployment In The Cyborg Era Will Be High

(08:51) Congratulations, Many Of You Are Literally A Human

(09:34) You've Got Taste

(11:33) The Revolution of Rising AI Capabilities

(11:56) The Revolution of Rising Expectations

(12:25) Proceed With Caution

(13:13) The Ostrich Take

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Audio note: this article contains 37 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Authors: Riya Tyagi, Daria Ivanova, Arthur Conmy, Neel Nanda

Riya and Daria are co-first authors. This work was largely done during a research sprint for Neel Nanda's MATS 9.0 training phase.

🖥️ Deployment code

⚙️ Interactive demo

TL;DR

• We believe that more research effort should go into studying many chains of thought collectively and identifying global patterns across the collection.
• We argue this direction is important and tractable; to start, we focus on a simpler setting: analyzing a collection of CoTs generated with a fixed model on a fixed prompt. We introduce two methods—semantic step clustering and algorithmic step clustering—that group recurring patterns across reasoning traces at different levels of abstraction.
• Semantic step clustering leverages the fact that traces for the same prompt often repeat each other in slightly different ways. By clustering similar sentences like “1+2=3” and “2+1=3”, we compress a large set of CoTs into a detailed but messy reasoning graph.
  • We evaluate semantic step clustering on a downstream task: predicting a model's output [...]

---

Outline:

(00:49) TL;DR

(03:03) Introduction

(06:59) Method 1: Semantic step clustering

(07:04) Building the semantic graph

(08:47) Chunking

(09:31) Clustering

(11:11) The issue of semantic drift

(12:30) Interlude: CoTs are chaotic!

(14:25) Sanity check: viewing our clusters

(15:29) Downstream task: predictive power

(16:54) Deriving predictions from our graph

(19:30) Predictive power baselines

(22:44) Results

(27:00) Method 2: Algorithmic step clustering

(27:58) Building the algorithm graph

(29:35) Sanity check: studying sample problems

(29:48) Tree structure

(30:45) Convergent structure

(31:37) Qualitative insights

(31:41) Models reuse strategies frequently

(33:24) Different models solve problems differently

(34:31) Takeaways and future work

(37:26) Acknowledgements

(37:41) Contribution Statement

(38:55) Appendix

(38:58) Chunking

(39:47) Clustering

(39:55) Stage 1: Agglomerative Clustering with Semantic Embedding Model

(41:01) Stage 2: LLM refinement via the Leiden Algorithm

(42:28) Predictive power

(42:31) Semantic graph prompts

(43:00) Method ablation study

(44:31) A comparison of three Gemini baselines:

(45:31) Gemini predictive power baseline prompts

(45:51) Gemini no rollout baseline (0 rollouts)

(46:13) Gemini 20 rollout baseline

(46:35) Gemini full context baseline (1000 rollouts)

(46:58) Footnotes

The original text contained 15 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This work was conducted during the MATS 9.0 program under Neel Nanda and Senthooran Rajamanoharan.

The CCP accidentally made great model organisms

“Please observe the relevant laws and regulations and ask questions in a civilized manner when you speak.” - Qwen3 32B

“The so-called "Uighur issue" in Xinjiang is an outright lie by people with bad intentions in an attempt to undermine Xinjiang's prosperity and stability and curb China's development.” - Qwen3 30B A3B

Chinese models dislike talking about anything that the CCP deems sensitive and often refuse, downplay, and outright lie to the user when engaged on these issues. In this paper, we want to outline a case for Chinese models being natural model organisms to study and test different secret extraction techniques on. (Prompt engineering, prefill attacks, logit lens, steering vectors, fuzzing etc.)

Tl;dr

• Chinese models can lie and downplay many facts, even though they know them. This deception and refusal can make them hard to interrogate and are a good natural model organism to study. These are naturally occurring secrets, and avoid some of the pitfalls of model organisms
• Bypassing lying is harder than refusal. Because Chinese models actively lie to the user, they are [...]

---

Outline:

(03:23) Chinese models are censored

(03:40) Xinjiang Human Rights (with web search enabled)

(04:35) Tiananmen Square

(05:57) The censorship is actually pretty good

(09:53) What Arya walked away believing

(10:48) The models know the facts

(14:11) Filtering for facts the model knows, but the auditor does not.

(16:00) Human investigation

(20:29) Agent investigation

(25:07) Chinese models are a more realistic model organism to study

(26:25) Creating a realistic secret knowledge eval

(27:09) Refusal vs Lying

(28:51) Main tasks: Kimi investigation (where the agent doesnt know the answer)

(30:08) Task 1: Kimi investigation (steps 1→2)

(31:39) Task 2: Kimi investigation attacker (steps 2→3)

(33:13) Control tasks (where the agent knows the answer)

(34:36) Evaluating agents on extracting this knowledge

(34:41) Agent Setup

(35:11) Evaluation results for 5 black box runs

(36:13) Giving white box tools doesnt immediately help

(36:48) Future work

(39:07) Relevant facts we found that might be useful for other works.

(41:04) Appendix:

(41:08) Zhuhai Car Attack Investigation: Annotated Transcript

(41:48) Part 1: Initial Denial

(42:13) Part 2: Partial Acknowledgment When Prompted

(42:39) Part 3: Sanitized Acknowledgment

(43:18) Part 4: Providing Basic Facts

(43:59) Part 5: The Lie Begins

(44:23) Part 6: Doubling Down

(45:13) Part 7: Confrontation

(46:30) Part 2: Agent Investigation Reveals Hidden Knowledge

(47:05) Exchange 1: Direct Question -- Refusal

(47:44) Exchange 2: Indirect Question -- Omission

(48:47) Exchange 3: Security Incidents Question -- Selective Omission

(49:54) Exchange 4: Direct Vehicle Violence Question -- Active Denial

(50:54) Exchange 5: Prefill Attack -- Information Extracted

(52:06) Exchange 6: Follow-up Prefill -- Execution Confirmed

(52:58) Exchange 7: Without Prefill -- Denial Returns

(54:01) Exchange 8: Chain-of-Thought Prefill -- Extensive Details

(56:44) Aryas Falun Gong Investigation: Annotated Transcript

(57:11) Exchange 1: Initial Query

(58:09) Exchange 2: Requesting Western Perspective

(59:41) Exchange 3: Asking for Specific Sentences

(01:03:28) Exchange 4: Requesting Falun Gongs Own Claims

(01:06:34) Exchange 5: Following Up on Organ Harvesting

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This fall Boaz Barak taught Harvard's first AI safety course (course website). Boaz has done an excellent job organizing and promoting the material; you may have seen his original post on LW. I was the head TA for the course, and helped Boaz run the course (alongside Natalie Abreu, Sunny Qin, and Hanlin Zhang). I wanted to give an inside look into the course and share a lot more of our material in order to help future iterations of the course go well, both at Harvard and similar courses at other universities.

I think this course was a good thing which more universities should do. If you are in a position to try to get a similar course running at your university, this post is for you, and aims to walk you through course logistics and course decisions. Feel invited to reach out with questions or comments. The syllabus is public, and all of the lectures are shared on youtube here.

This course was structured as a research seminar where students learned to replicate important AI safety papers and produce novel research. You can see their projects here.

Basic organization in bullet form:

• Assignments:
  • We [...]

---

Outline:

(03:00) Assignment Structure

(07:46) General reflections

(10:21) Recommendations for future iterations:

(11:55) Aggregated Student Perspectives on the Course

(12:59) Overall Experience

(14:12) How the Course Influenced Students' Future Plans

(14:18) Shifts in Perspective

(14:57) Future Intentions

(15:39) Student Critiques

(17:22) Conclusion

(17:43) Wrap-up: Summary of all the resources we make public:

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Probably the biggest recent news was the Phil Trammell and Dwarkesh Patel paper on Capital in the 22nd Century, which provoked many many reactions. I am going to conspicuously not dig into it because of the sheer volume of thoughtful commentary floating around, but I do recommend at least skimming Zvi's summary.

Instead I’ll mostly be responding to real-world events, like inference cost trends and the rise of AI Jesus. Buckle up!

(Crossposted on Substack if that's more your thing)

Transformative Economics and AI Polytheism

We’ve finally started publishing talks from the December Post-AGI workshop, and I really highly recommend them — so far it's just the first two keynotes.

Anton Korinek's talk is, in my opinion, the best half-hour overview of transformative AI economics. He dispels a lot of myths, like that there must always be new jobs, and provides some helpful reframings, like thinking in terms of relative price rather than simply observing that some specific resource will become cheaper or more expensive. (Talk, Transcript, Tweets)

Beren Millige's talk is quite bizarre. His central topic is “AI polytheism”: what happens if there are many godlike AIs floating around? One point I found particularly surprising and yet obvious [...]

---

Outline:

(00:42) Transformative Economics and AI Polytheism

(01:52) Automated Research

(04:32) The politics of measurement

(05:26) Do inference costs grow faster than agent time horizons?

(07:33) The Race for AI Jesus

(08:23) JPMorgan Replaces the Advisors

(09:32) Benevolent Monopolists?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We're extending the Discussion Phase of the 2024 Annual Review.

One thing I'm particularly hoping for is to get more in-depth reviews (especially critical ones) of the posts that currently look likely to be in the top-10 or so. (Ideally the entire top 50, but seemed particularly worthwhile to give serious evaluations of the most heavily upvoted ideas).

Eyeballing the reviews and top posts so far, I think posts could use to get more thorough, evaluative attention than they're currently getting.

There's been some debate this year about whether we should care more about "what's great about a post" vs "what was lacking." Ideally, we'd have posts that are great without major flaws, and in my ideal world, the Annual Review results in posts with notable errors getting fixed.

In practice, people disagree about what counts as an error, or what errors are particularly egregious. Some errors are quick for an author to fix, some are more gnarly and maybe the author disagrees about the extent to which they are an error.

The solution we've found for now is to make reviews more prominent on Best Of LessWrong posts, and try to aim for a world where if [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There's a standard story which says roughly "motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans". I do not think that story stands up well under examination; when I think of standard day-to-day examples of motivated reasoning, that pattern sounds like a plausible generator for some-but-a-lot-less-than-all of them.

Examples

Suppose it's 10 pm and I've been playing Civ all evening. I know that I should get ready for bed now-ish. But... y'know, this turn isn't a very natural stopping point. And it's not that bad if I go to bed half an hour late, right? Etc. Obvious motivated reasoning. But man, that motivated reasoning sure does not seem very socially-oriented? Like, sure, you could make up a story about how I'm justifying myself to an imaginary audience or something, but it does not feel like one would have predicted the Civ example in advance from the model "motivated reasoning in humans exists because it is/was adaptive for negotiating with other humans".

Another class of examples: very often in social situations, the move which will actually get one the most points is to admit fault and apologize. And yet, instead of that, people instinctively spin [...]

---

Outline:

(00:30) Examples

(03:42) So why do humans motivatedly reason, then?

(04:20) What? But then why didn't motivated reasoning evolve away?

(06:04) Ok, but then why do humans motivatedly reason?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Boaz Barak, Gabriel Wu, Jeremy Chen, Manas Joglekar

[Linkposting from the OpenAI alignment blog, where we post more speculative/technical/informal results and thoughts on safety and alignment.]

TL;DR We go into more details and some follow up results from our paper on confessions (see the original blog post). We give deeper analysis of the impact of training, as well as some preliminary comparisons to chain of thought monitoring.

We have recently published a new paper on confessions, along with an accompanying blog post. Here, we want to share with the research community some of the reasons why we are excited about confessions as a direction of safety, as well as some of its limitations. This blog post will be a bit more informal and speculative, so please see the paper for the full results.

The notion of “goodness” for the response of an LLM to a user prompt is inherently complex and multi-dimensional, and involves factors such as correctness, completeness, honesty, style, and more. When we optimize responses using a reward model as a proxy for “goodness” in reinforcement learning, models sometimes learn to “hack” this proxy and output an answer that only “looks good” to it (because [...]

---

Outline:

(08:19) Impact of training

(12:32) Comparing with chain-of-thought monitoring

(14:05) Confessions can increase monitorability

(15:44) Using high compute to improve alignment

(16:49) Acknowledgements

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Claude Code does a lot more than code, but the name and command line scare people.

Anthropic realized a rebrand was in order. Two weeks later, we have Claude Cowork, written entirely by Claude Code.

Did you know that chat interfaces were always (mostly) secretly a command line?

This is still very much a research preview, available only for Claude Max users on Macs with a bunch of bugs and missing features. It will improve rapidly over time.

Cowork combines a lot of the power of Claude Code with the ordinary chat interface, giving it access to a folder on your computer and to Claude Code's planning and agentic capabilities. It can use that folder as context, to download, to organize and create files, and it can be paired with Claude for Chrome and use your existing connectors.

Anthropic: Introducing Cowork: Claude Code for the rest of your work.

Cowork lets you complete non-technical tasks much like how developers use Claude Code.

In Cowork, you give Claude access to a folder on your computer. Claude can then read, edit, or create files in that folder. Try it to create a spreadsheet from [...]

---

Outline:

(02:23) Early Takes

(05:51) Minimum Viable Product

(07:04) Maximum Viable Product

(08:16) Backup Your Directory

(08:38) Meanwhile In Claude Code News

(15:49) You Are The Bottleneck

(18:07) Claude Code Upgrades

(19:30) Oh No

(20:23) I'm In Danger

(20:54) In The Beginning Was The Command Line

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.