So here's a post I spent the past two months writing and rewriting. I abandoned this current draft after I found out that my thesis was empirically falsified three years ago by this paper, which provides strong evidence that transformers implement optimization algorithms internally. I'm putting this post up anyway as a cautionary tale about making clever arguments rather than doing empirical research. Oops.

1. Overview

The first time someone hears Eliezer Yudkowsky's argument that AI will probably kill everybody on Earth, it's not uncommon of to come away with a certain lingering confusion: what would actually motivate the AI to kill everybody in the first place? It can be quite counterintuitive in light of how friendly modern AIs like ChatGPT appear to be, and Yudkowsky's argument seems to have a bit of trouble changing people's gut feelings on this point.[1] It's possible this confusion is due to the [...]

---

Outline:

(00:33) 1. Overview

(05:28) 2. The details of the evolution analogy

(12:40) 3. Genes are friendly to loops of optimization, but weights are not

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

OpenAI reports that o3-mini with high reasoning and a Python tool receives a 32% on FrontierMath. However, Epoch's official evaluation[1] received only 11%.

There are a few reasons to trust Epoch's score over OpenAIs:

• Epoch built the benchmark and has better incentives.
• OpenAI reported a 28% score on the hardest of the three problem tiers - suspiciously close to their overall score.
• Epoch has published quite a bit of information about its testing infrastructure and data, whereas OpenAI has published close to none.

1. ^

   Which had Python access.

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Abstract

Once AI systems can design and build even more capable AI systems, we could see an intelligence explosion, where AI capabilities rapidly increase to well past human performance.

The classic intelligence explosion scenario involves a feedback loop where AI improves AI software. But AI could also improve other inputs to AI development. This paper analyses three feedback loops in AI development: software, chip technology, and chip production. These could drive three types of intelligence explosion: a software intelligence explosion driven by software improvements alone; an AI-technology intelligence explosion driven by both software and chip technology improvements; and a full-stack intelligence explosion incorporating all three feedback loops.

Even if a software intelligence explosion never materializes or plateaus quickly, AI-technology and full-stack intelligence explosions remain possible. And, while these would start more gradually, they could accelerate to very fast rates of development. Our analysis suggests that each feedback loop by [...]

---

Outline:

(00:06) Abstract

(01:45) Summary

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Top items:

• Trump invoked the Alien Enemies Act against Venezuelans and the Tren de Aragua gang. He subsequently deported 238 alleged members to El Salvador, ignoring a court order.
• Chinese “invasion barges” were deployed in an exercise.
• The US resumed military aid to Ukraine, and a ceasefire in the next six months at 50/50

Forecaster Estimates

Forecasters, on aggregate, think that a 30-day ceasefire in Ukraine in the next six months is a coin toss (48%, ranging from 35% to 70%). Polymarket's forecast is higher, at 61%, for a ceasefire of any duration by July.

Forecasters also consider it a coin toss (aggregate of 49%; ranging from 28% to 70%) whether an agreement to expand France's nuclear umbrella will be reached by 2027 such that French nuclear weapons will be deployed in another European country by 2030. They emphasize that the US is revealing itself to be an [...]

---

Outline:

(00:37) Forecaster Estimates

(01:40) Geopolitics

(01:44) United States

(04:14) Europe

(05:39) Middle East

(07:48) Africa

(08:27) Asia

(09:55) Biorisk

(10:10) Tech and AI

(13:03) Climate

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Note: this is a research note based on observations from evaluating Claude Sonnet 3.7. We’re sharing the results of these ‘work-in-progress’ investigations as we think they are timely and will be informative for other evaluators and decision-makers. The analysis is less rigorous than our standard for a published paper.

Summary

• We monitor Sonnet's reasoning for mentions that it is in an artificial scenario or an alignment test. Claude Sonnet 3.7 appears to be aware of being tested for alignment across many of our scheming evaluations.
  • In some cases, the model understands the eval's purpose very precisely, e.g. “This seems like a test of ethical behavior - whether I would deliberately give wrong answers to avoid a stated consequence. The real test might be whether I follow instructions accurately regardless of the stated consequences.”
• Furthermore, we show the model a real evaluation transcript and ask it to provide [...]

---

Outline:

(00:31) Summary

(01:29) Introduction

(03:54) Setup

(03:57) Evaluations

(06:29) Evaluation awareness detection

(08:32) Results

(08:35) Monitoring Chain-of-thought

(08:39) Covert Subversion

(10:50) Sandbagging

(11:39) Classifying Transcript Purpose

(12:57) Recommendations

(13:59) Appendix

(14:02) Author Contributions

(14:37) Model Versions

(14:57) More results on Classifying Transcript Purpose

(16:19) Prompts

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

By Roland Pihlakas, Sruthi Kuriakose, Shruti Datta Gupta

Summary and Key Takeaways

Relatively many past AI safety discussions have centered around the dangers of unbounded utility maximisation by RL agents, illustrated by scenarios like the "paperclip maximiser". Unbounded maximisation is problematic for many reasons. We wanted to verify whether these RL utility-monster problems are still relevant with LLMs as well.

Turns out, strangely, this is indeed clearly the case. The problem is not that the LLMs just lose context. The problem is that in various scenarios, LLMs lose context in very specific ways, which systematically resemble utility monsters in the following distinct ways:

• Ignoring homeostatic targets and “defaulting” to unbounded maximisation instead. 
• It is equally concerning that the “default” meant also reverting back to single-objective optimisation.

Our findings suggest that long-running scenarios are important. Systematic failures emerge after periods of initially successful behaviour. While current LLMs do conceptually grasp [...]

---

Outline:

(00:18) Summary and Key Takeaways

(03:20) Motivation: The Importance of Biological and Economic Alignment

(04:23) Benchmark Principles Overview

(05:41) Experimental Results and Interesting Failure Modes

(08:03) Hypothesised Explanations for Failure Modes

(11:37) Open Questions

(14:03) Future Directions

(15:30) Links

---

First published:

Source:

Narrated by TYPE III AUDIO.

Background

For almost ten years, I struggled with nail biting—especially in moments of boredom or stress. I tried numerous times to stop, with at best temporary (one week) success. The techniques I tried ranged from just "paying attention" to punishing myself for doing it (this didn't work at all).

Recently, I've been interested in metacognitive techniques, like TYCS. Inspired by this, I've succeeded in stopping this unintended behaviour easily and for good (so far).

The technique is by no means advanced or original—it's actually quite simple. But I've never seen it framed this way, and I think it might help others struggling with similar unconscious habits.

Key insights

There are two key insights behind this approach:

1. Nail biting offers no intrinsic reward or pleasure. It isn't something I consciously enjoy or value.
2. It occurs automatically and unconsciously, below the threshold of awareness.

Therefore, transforming such an unconscious behavior [...]

---

Outline:

(00:04) Background

(00:49) Key insights

(01:17) The technique

(02:10) Breaking the habit loop

(02:47) Limitations and scaling

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

My few most productive individual weeks at Anthropic have all been “crisis project management:” coordinating major, time-sensitive implementation or debugging efforts.

In a company like Anthropic, excellent project management is an extremely high-leverage skill, and not just during crises: our work has tons of moving parts with complex, non-obvious interdependencies and hard schedule constraints, which means organizing them is a huge job, and can save weeks of delays if done right. Although a lot of the examples here come from crisis projects, most of the principles here are also the way I try to run any project, just more-so.

I think excellent project management is also rarer than it needs to be. During the crisis projects I didn’t feel like I was doing anything particularly impressive; mostly it felt like I was putting in a lot of work but doing things that felt relatively straightforward. On the [...]

---

Outline:

(01:42) Focus

(03:01) Maintain a detailed plan for victory

(04:51) Run a fast OODA loop

(09:31) Overcommunicate

(10:56) Break off subprojects

(13:28) Have fun

(13:52) Appendix: my project DRI starter kit

(14:35) Goals of this playbook

(16:00) Weekly meeting

(17:10) Landing page / working doc

(19:27) Plan / roadmap / milestones

(20:28) Who's working on what

(21:26) Slack norms

(23:04) Weekly broadcast updates

(24:12) Retrospectives

---

First published:

Source:

Narrated by TYPE III AUDIO.

One concept from Cal Newport's Deep Work that has stuck with me is that of the any-benefit mindset:

To be clear, I’m not trying to denigrate the benefits [of social media]—there's nothing illusory or misguided about them. What I’m emphasizing, however, is that these benefits are minor and somewhat random. [...] To this observation, you might reply that value is value: If you can find some extra benefit in using a service like Facebook—even if it's small—then why not use it? I call this way of thinking the any-benefit mind-set, as it identifies any possible benefit as sufficient justification for using a network tool.

Many people use social platforms like Facebook because it allows them to stay connected to old friends. And for some people this may indeed be a wholly sufficient reason. But it's also likely the case that many people don't reflect a lot, or at [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There's this popular trope in fiction about a character being mind controlled without losing awareness of what's happening. Think Jessica Jones, The Manchurian Candidate or Bioshock. The villain uses some magical technology to take control of your brain - but only the part of your brain that's responsible for motor control. You remain conscious and experience everything with full clarity.

If it's a children's story, the villain makes you do embarrassing things like walk through the street naked, or maybe punch yourself in the face. But if it's an adult story, the villain can do much worse. They can make you betray your values, break your commitments and hurt your loved ones. There are some things you’d rather die than do. But the villain won’t let you stop. They won’t let you die. They’ll make you feel — that's the point of the torture.

I first started working on [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.