This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. Many other people have talked about similar ideas, and I claim neither novelty nor credit.

Note that this reflects my median scenario for catastrophe, not my median scenario overall. I think there are plausible alternative scenarios where AI development goes very well.

When thinking about how AI could go wrong, the kind of story I’ve increasingly converged on is what I call “catastrophe through chaos.” Previously, my default scenario for how I expect AI to go wrong was something like Paul Christiano's “What failure looks like,” with the modification that scheming would be a more salient part of the story much earlier.

In contrast, “catastrophe through chaos” is much more messy, and it's much harder to point to a single clear thing that went wrong. The broad strokes of [...]

---

Outline:

(02:46) Parts of the story

(02:50) AI progress

(11:12) Government

(14:21) Military and Intelligence

(16:13) International players

(17:36) Society

(18:22) The powder keg

(21:48) Closing thoughts

---

First published:

Source:

Narrated by TYPE III AUDIO.

In this post, I discuss a number of critiques of Guaranteed Safe AI (GSAI) approaches I’ve encountered since releasing this position paper.

The critiques I’m responding to here are taken from a range of places, including personal conversations and social media. The two most substantive written/recorded critiques I'm aware of (and had in mind when writing this post) are Andrew Dickson's Limitations on Formal Verification for AI Safety and Zac Hatfield-Dodds's talk Formal Verification is Overrated. I’ve done my best to do justice in our depiction of those critiques, but I certainly welcome clarifications or pushback where I seem to have missed the original point of a given critique.

To be clear, the position I'm defending here (and will be adding more nuance to throughout this piece) is not that GSAI is definitely going to succeed. Nor that it is a solution to all of AI risk. What [...]

---

Outline:

(03:21) Acknowledgements

(03:43) 1. Brief summary of Towards Guaranteed Safe AI

(10:05) 2. Commonly-encountered critiques, and some responses

(10:17) 1. The proposed architecture won't work

(10:22) 1.1. You can't get a good enough world model

(19:42) 1.2. You can't get good enough guarantees

(26:43) 1.3. You can't get good enough safety specifications

(30:14) 2. Other approaches are a better use of time

(30:19) 2.1. Does not address the most serious risks

(44:04) 2.2. AI won't help (soon) enough

(48:20) 2.3. Lack of 'portability and independent verification of proof certificate

(49:48) 3. Conclusion

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The avalanche of DeepSeek news continues. We are not yet spending more than a few hours at a time in the singularity, where news happens faster than it can be processed. But it's close, and I’ve had to not follow a bunch of other non-AI things that are also happening, at least not well enough to offer any insights.

So this week we’re going to consider China, DeepSeek and r1 fully split off from everything else, and we’ll cover everything related to DeepSeek, including the policy responses to the situation, tomorrow instead.

This is everything else in AI from the past week. Some of it almost feels like it is from another time, so long ago.

I’m afraid you’re going to need to get used to that feeling.

Also, I went on Odd Lots to discuss DeepSeek, where I was and truly hope to again [...]

---

Outline:

(00:55) Language Models Offer Mundane Utility

(02:47) Language Models Don't Offer Mundane Utility

(05:43) Language Models Don't Offer You In Particular Mundane Utility

(10:49) (Don't) Feel the AGI

(12:36) Huh, Upgrades

(16:08) They Took Our Jobs

(21:30) Get Involved

(22:04) Introducing

(23:38) In Other AI News

(27:10) Hype

(29:56) We Had a Deal

(31:43) Quiet Speculations

(37:14) The Quest for Sane Regulations

(39:40) The Week in Audio

(39:51) Don't Tread on Me

(45:42) Rhetorical Innovation

(55:22) Scott Sumner on Objectivity in Taste, Ethics and AGI

(01:04:41) The Mask Comes Off (1)

(01:06:58) The Mask Comes Off (2)

(01:09:12) International AI Safety Report

(01:10:37) One Step at a Time

(01:14:12) Aligning a Smarter Than Human Intelligence is Difficult

(01:18:54) Two Attractor States

(01:26:51) You Play to Win the Game

(01:28:10) Six Thoughts on AI Safety

(01:35:53) AI Situational Awareness

(01:40:15) People Are Worried About AI Killing Everyone

(01:43:40) Other People Are Not As Worried About AI Killing Everyone

(01:44:24) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Not too long ago, synthetic biology was fashionable.

Then, in quick succession, we saw massive stock price collapses of three unicorn synbio startups in 2021-2023: Gingko Bioworks, Zymergen, and Amyris.

What went wrong? And can synthetic-biology (aka biomanufacturing) startups be successful at all?

Biomanufacturing Basics

Biomanufacturing generally refers to the industrial-scale production of materials from simple living things. In other words, extracts produced from cultures of yeast, bacteria, fungi, algae, or other types of cells, typically grown in bioreactor vats.1

Fermented foods and beverages like beer and koji are the oldest biomanufactured goods, and they played a pioneering role in the development of industrial biomanufacturing techniques. William Sealy Gossett, the statistician behind the “Student's t distribution” and the concept of “statistical significance” in experimental design, developed his techniques as the Head Brewer of the Guinness beer company.

Many other substances are routinely biomanufactured, from [...]

---

Outline:

(00:39) Biomanufacturing Basics

(04:29) Amyris: (Energy) Math is Hard, Let's Go Shopping!

(06:48) Zymergen: Yeast Ain't Everything

(09:43) Gingko Bioworks: You May Have To, Like, Actually Sell Things

(11:35) Takeaways

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Coauthored with Mark Kurzeja

A while back, we explored the “BiDPO” method for training steering vectors. In Gemini 1.5v1 Flash and Pro, BiDPO steering vectors boosted TruthfulQA scores by >10% while mostly retaining capabilities. When we updated to Gemini 1.5v2, prompt-based steering baselines became significantly stronger. BiDPO did not beat the stronger baselines, ending the project.

...

BiDPO seems effective and sample-efficient but does not currently exceed more standard baselines. It's hard to draw firm conclusions about BiDPO because TruthfulQA might not be measuring truthfulness ／factuality. However, we remain excited about DPO-driven Conditional Activation Steering, which has additional advantages—particularly for targeted loss mitigation.

This result is largely negative. I wanted to share it to increase scientific understanding around steering! We also conducted a postmortem on why the method stopped outperforming baselines and reflect on lessons learned.

I'd also like to note that @Ryan Greenblatt's skepticism [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

In case you have not heard, there have been some recent and not-so-recent killings allegedly by people who have participated in aspiring rationalist spaces.

I've put some links in a footnote for those who want the basic info.[1]

I believe many people are thinking about this and how to orient. People have questions like:

1. What has happened?
2. What is likely to happen?
3. What can be done to prevent more killings?
4. How to relate to all of the widespread interest (e.g. from journalists)?

I hereby demarcate this thread as a place for people to ask and answer questions about this topic.

Three virtues for this thread are (1) Taking care of yourself, (2) Courage, and (3) Informing.

Airlines have standard advice to put your own oxygen mask on first, before helping others. The reasoning being that if you don't help yourself, you won't then be able to help [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Many thinkers worth reading wrote in past centuries:

• 1600s: Bacon, Hobbes, Descartes, Spinoza, Locke, Leibniz
• 1700s: Berkeley, Hume, Rousseau, Smith, Kant, Burke, Bentham
• 1800s: Schopenhauer, Mill, Sidgwick

Some of them wrote in English and their texts are usually presented in the original, even when they use archaic spelling, punctuation, and style. (This is in contrast to foreign-language works which are often translated to a modern style.)

For example, in the past there were more uses of the comma, such as

• Before any relative clause:
  • Dickens: “The objects he had lately pursued, turned worthless beside her”
  • Melville: “This august dignity I treat of, is not the dignity of kings”
• Before any clause starting with “that” for any reason:
  • Austen: “It is a truth universally acknowledged, that…”
• After any clausal or compound subject as far as I can tell:
  • J Robertson: “Whoever is capable of forgetting [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Full version on arXiv | X

Executive summary

AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal. However, we argue that even an incremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial risk of eventual human disempowerment. This loss of human influence will be centrally driven by having more competitive machine alternatives to humans in almost all societal functions, such as economic labor, decision making, artistic creation, and even companionship.

A gradual loss of control of our own civilization might sound implausible. Hasn't technological disruption usually improved aggregate human welfare? We argue that the alignment of societal systems with human interests has been stable only because of the necessity of human participation for thriving economies, states, and [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a linkpost accompanying a new paper by UK AISI and Redwood Research. Please see the full paper for more details.

An illustration of the control evaluation that provides the core evidence used in our control safety case sketch. During the control evaluation, a red team constructs adversarial substitute models to cause unacceptable outcomes in a deployment proxy consisting of 2,000 AI research tasks, followed by extrapolation to a larger deployment setting.

Abstract. As LLM agents gain a greater capacity to cause harm, AI developers might increasingly rely on control measures such as monitoring to justify that they are safe. We sketch how developers could construct a “control safety case,” which is a structured argument that models are incapable of subverting control measures in order to cause unacceptable outcomes. As a case study, we sketch an argument that a hypothetical LLM agent deployed internally at an AI [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

"If China can't get millions of chips, we'll (at least temporarily) live in a unipolar world, where only the US and its allies have these models.

It's unclear whether the unipolar world will last, but there's at least the possibility that, because AI systems can eventually help make even smarter AI systems, a temporary lead could be parlayed into a durable advantage.

Thus, in this world, the US and its allies might take a commanding and long-lasting lead on the global stage.

Well-enforced export controls are the only thing that can prevent China from getting millions of chips, and are therefore the most important determinant of whether we end up in a unipolar or bipolar world."

From Anthropic CEO Dario Amodei's latest piece, On DeepSeek and Export Controls.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.