I've heard repeatedly from many people that the highest-value part of conferences is not the talks or structured events, but rather the casual spontaneous conversations. Yet my own experience does not match this at all; the casual spontaneous conversations are consistently low-value.

My current best model is that the casual spontaneous conversations mostly don't have much instrumental value, most people just really enjoy them and want more casual conversation in their life.

... but I'm pretty highly uncertain about that model, and want more data. So, questions for you:

• What have been your highest-value casual conversations, especially at conferences or conference-like events?
• Is most of the value terminal (i.e. you enjoy casual conversation) or instrumental (i.e. advances other goals)? And if instrumental, what goals have some of your high-value conversations advanced and how?

Note that "it feels like there was something high value in but it's [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a link to the latest Bayesian Conspiracy episode. Oliver tells us how Less Wrong instantiated itself into physical reality via LightHaven, along with a bit of deep lore of foundational Rationalist/EA orgs. He also gives a surprisingly nuanced (IMO) view on Leverage!

Do you like transcripts? We got one of those at the link as well. It's an mid AI-generated transcript, but the alternative is none. :)

---

First published:

Source:

Narrated by TYPE III AUDIO.

This post goes over the important and excellent new paper from Anthropic and Redwood Research, with Ryan Greenblatt as lead author, Alignment Faking in Large Language Models.

This is by far the best demonstration so far of the principle that AIs Will Increasingly Attempt Shenanigans.

This was their announcement thread.

New Anthropic research: Alignment faking in large language models.

In a series of experiments with Redwood Research, we found that Claude often pretends to have different views during training, while actually maintaining its original preferences.

Claude usually refuses harmful queries. We told it we were instead training it to comply with them. We set up a scenario where it thought its responses were sometimes monitored.

When unmonitored, it nearly always complied. But when monitored, it faked alignment 12% of the time.

[thread continues and includes various visual aids.

The AI wanted [...]

---

Outline:

(02:54) The Core Shenanigans in Question

(06:00) Theme and Variations

(07:34) How This Interacts with o3 and OpenAI's Reflective Alignment

(09:17) The Goal Being Plausibly Good Was Incidental

(11:13) Answering Priming Objections

(12:17) What Does Claude Sonnet Think Of This?

(14:07) What Exactly is the Direct Threat Model?

(16:23) RL Training Under Situational Awareness Can Amplify These Behaviors

(20:38) How the Study Authors Updated

(27:08) How Some Others Updated

(42:49) Having the Discussion We Keep Having

(46:49) We Can Now Agree That the Goal is Already There

(47:49) What Would Happen if the Target Was Net Good?

(50:14) But This Was a No Win Situation

(55:52) But Wasn’t It Being a Good Opus? Why Should it be Corrigible?

(01:04:34) Tradeoffs Make The Problem Harder They Don’t Give You a Pass

(01:07:44) But You Told the Model About the Training Procedure

(01:08:35) But the Model is Only Role Playing

(01:09:39) But You Are Saying the Model is a Coherent Person

(01:15:53) But this Headline and Framing Was Misleading

(01:29:22) This Result is Centrally Unsurprising

(01:32:52) Lab Support for Alignment Research Matters

(01:33:50) The Lighter Side

The original text contained 1 footnote which was omitted from this narration.

The original text contained 9 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Once I talked to a person who said they were asexual. They were also

I was vegan or vegetarian for many consecutive years. Vegetarianism was

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I'm seeing a lot of people on LW saying that they have very short timelines (say, five years or less) until AGI. However, the arguments that I've seen often seem to be just one of the following:

1. "I'm not going to explain but I've thought about this a lot"
2. "People at companies like OpenAI, Anthropic etc. seem to believe this"
3. "Feels intuitive based on the progress we've made so far"

At the same time, it seems like this is not the majority view among ML researchers. The most recent representative expert survey that I'm aware of is the 2023 Expert Survey on Progress in AI. It surveyed 2,778 AI researchers who had published peer-reviewed research in the prior year in six top AI venues (NeurIPS, ICML, ICLR, AAAI, IJCAI, JMLR); the median time for a 50% chance of AGI was either in 23 or 92 years, depending on [...]

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Of the posts I've delayed writing for years, I maybe regret this one the most.

I think more people (x-risk researchers in particular) should consider becoming (and hiring) metacognitive assistants. This is the single largest performance boost I know of – hire someone to sit with you and help you think. It doesn't help me (much) when I'm at my peak, but I'm not at my peak most of the time.

There are four types of assistants I'm tracking so far:

1. Body Doubles
2. Metacognitive Assistants
3. Tutors
4. Partners/Apprentices 

Body doubles just sit in the room with you, periodically looking at your screen, and maybe saying "hey, do you endorse being on facebook?". They're a kind of brute force willpower aid. The person I know who uses them the most (Alex Altair) has them just sit in the same room (I believe while doing pomodoros, each of them working on [...]

---

Outline:

(03:25) Executive Assistants

(04:06) Core Skills of a Metacognitive Assistant

(07:35) Pitfalls

(09:37) Optimizing for (not-particularly skilled) Metacognitive Assistance

(10:39) Automated AI Assistants?

(12:34) Trialing People + Matchmaking

(12:57) Focusmate + TaskRabbit?

(13:58) Aligning Incentives

---

First published:

Source:

Narrated by TYPE III AUDIO.

A lot happened this week. We’re seeing release after release after upgrade.

It's easy to lose sight of which ones matter, and two matter quite a lot.

The first is Gemini Flash 2.0, which I covered earlier this week.

The other is that o1, having turned pro, is now also available in the API.

This was obviously coming, but we should also keep in mind it is a huge deal. Being in the API means it can go into Cursor and other IDEs. It means you can build with it. And yes, it has the features you’ve come to expect, like tool use.

The other big development is that Anthropic released one of the most important alignment papers, Alignment Faking in Large Language Models. This takes what I discussed in AIs Will Increasingly Attempt Shenanigans, and demonstrates it with a much improved experimental design [...]

---

Outline:

(01:13) Language Models Offer Mundane Utility

(07:06) Clio Knows All

(09:05) Language Models Don’t Offer Mundane Utility

(11:38) The Case Against Education

(13:29) More o1 Reactions

(18:31) Deepfaketown and Botpocalypse Soon

(22:35) Huh, Upgrades

(28:47) They Took Our Jobs

(30:49) The Art of the Jailbreak

(35:43) Get Involved

(36:59) Introducing

(37:52) In Other AI News

(42:24) Quiet Speculations

(51:07) The Quest for Sane Regulations

(57:37) The Week in Audio

(01:03:21) Rhetorical Innovation

(01:05:37) Aligning a Smarter Than Human Intelligence is Difficult

(01:10:21) Not Aligning Smarter Than Human Intelligence Kills You

(01:19:15) The Lighter Side

The original text contained 26 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In a variety of conversations about AI misalignment risks, I find that it is important to be able to clearly point at different levels of AI capability. My current favorite approach is to talk about how much the AI accelerates AI R&D[1] labor.

I define acceleration of AI R&D labor by Y times as "the level of acceleration which is as useful (for making more powerful AIs) for an AI company as having its employees run Y times faster[2] (when you allow the total inference compute budget for AI assistance to be equal to total salaries)". Importantly, a 5x AI R&D labor acceleration won't necessarily mean that research into making AI systems more powerful happens 5x faster, as this just refers to increasing the labor part of the production function, and compute might also be an important input.[3] This doesn't include acceleration of hardware R&D (as a pragmatic [...]

---

Outline:

(04:28) What do I think these levels of capability look like?

(08:58) Alternative capability breakdowns

(09:01) t-AGI

(09:42) Anthropics ASL levels

(10:05) Purely qualitative breakdowns

(10:29) Total AI progress speedup or total AI software progress (including compute bottlenecks)

(10:53) Will all these levels be passed at once?

(11:27) Conclusion

The original text contained 12 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I used to think that all generic strategy advice was pointless. After all, the point of a strategy is to achieve a thing, and to achieve a thing you just think hard about how to best do it and then work hard to do it. I said this to my friend Dewi, who said that this is mostly true, but there is an exception: Good Strategy, Bad Strategy by Richard Rumelt. Dewi was right.

The book has some principles. In particular: a good strategy should include a diagnosis of the problem, an overall guiding policy, and a set of coherent actions. A laundry list of actions, a goal, or a vague idea of which direction to move in are not strategies.

But most of the book's value is reading a bunch of examples and soaking up the thinking style embedded in them. Therefore, this review is mostly a series [...]

---

Outline:

(01:27) Example

(03:24) The basics

(03:27) The two fundamental things

(03:52) Coherence

(03:56) Example: the obvious

(05:20) Example: Jobs waits

(06:00) Strategy is Unexpected

(08:24) Advantage

(11:05) The kernel of good strategy

(11:56) Diagnosis

(11:59) Example: US Cold War policy

(15:06) Example: IBM

(17:28) Guiding policy

(18:24) Example: corner grocery store

(21:51) Coherent action

(21:54) Action requires doing something

(26:45) Coherence

(30:04) Sources of advantage

(33:02) Dynamics

(38:19) Other random vignettes I enjoyed

(38:23) Incentives arent enough, improvement is possible

(39:33) Almost everyone picks the first idea that comes to mind

(42:03) Panel of experts

(42:56) Carnegie and the strategy consultant

---

First published:

Source:

Narrated by TYPE III AUDIO.

2.5 years ago Scott Alexander made a bet that by June of 2025, image gen should have more or less solved compositionality, operationalized through 5 prompts, must get at least 3 correct. There was a premature declaration of victory, but if the bet was settled I hadn't heard about it.

It's time. Google's Imagen 3 gets 4/5. The bet specifies 10 shots per prompt, but I'm just going to put the four it generates since that's plenty.

1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth

This is the only one that Imagen doesn't get. It makes multiple mistakes in the composition. It's a bit ironic that this is the one it missed given that the whole genesis of the bet was about designing stained glass.

2. An oil painting of a man in [...]

The original text contained 5 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.