Subtitle: System cards are established but other approaches seem importantly better.

Here's a relatively important question regarding transparency requirements for AI companies: At which points in time should AI companies be required to disclose information? (While I focus on transparency, this question is also applicable to other safety-relevant requirements, and is applicable to norms around voluntary actions rather than requirements.)

A natural option would be to attach transparency requirements to the existing processes of pre-deployment testing and releasing a model card when a new model is released. As in, companies would be required to include the relevant information whenever they release a new model (likely in the model card). This is convenient because pre-deployment testing and model cards are already established norms in the AI industry, which makes it easier to attach something new to these existing processes rather than creating a new process.

However, I think attaching [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Subtitle: More thoughts on making deals with schemers.

These are some research notes on whether we could reduce AI takeover risk by cooperating with unaligned AIs. I think the best and most readable public writing on this topic is “Making deals with early schemers”, so if you haven't read that post, I recommend starting there.

These notes were drafted before that post existed, and the content is significantly overlapping. Nevertheless, these notes do contain several points that aren’t in that post, and I think it could be helpful for people who are (thinking about) working in this area.

Most of the novel content can be found in the sections on What do AIs want?, Payment structure, What I think should eventually happen, and the appendix BOTEC. (While the sections on “AI could do a lot to help us” and “Making credible promises” are mostly overlapping with “Making deals [...]

---

Outline:

(01:10) Summary

(06:18) AIs could do a lot to help us

(10:29) What do AIs want?

(11:25) Non-consequentialist AI

(12:11) Short-term offers

(13:54) Long-term offers

(15:44) Unified AIs with ~linear returns to resources

(17:10) Uncoordinated AIs with ~linear returns to resources

(18:43) AIs with diminishing returns to resources

(22:18) Making credible promises

(25:46) Payment structure

(28:57) What I think should eventually happen

(29:39) AI companies should make AIs short-term offers

(32:34) AI companies should make long-term promises

(33:14) AI companies should commit to some honesty policy and abide by it

(33:32) AI companies should be creative when trying to communicate with AIs

(36:02) Conclusion

(38:18) Acknowledgments

(38:34) Appendix: BOTEC

(43:44) Full BOTEC

(46:19) BOTEC structure

(48:33) What's the probability that our efforts would make the AIs decide to cooperate with us?

(53:56) There are multiple AIs

(56:55) How valuable would AI cooperation be?

(59:04) Discount for misalignment

(59:25) Discount for lack of understanding

(01:00:41) Putting it together

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Subtitle: When and why we should refrain from lying.

In the future, we might accidentally create AIs with ambitious goals that are misaligned with ours. But just because we don’t have the same goals doesn’t mean we need to be in conflict. We could also cooperate with each other and pursue mutually beneficial deals.

For previous discussion of this, you can read Making deals with early schemers. I recommend starting there before you read this post.

To facilitate this kind of cooperation, I would like us to be able to credibly negotiate with AIs, and credibly make promises about how we will help them if they help us. But it may be difficult to achieve high credibility in negotiations with AIs. AIs are in a very vulnerable epistemic position, because humans have a lot of control over AI memories and sensory inputs. Even worse: we have already developed [...]

---

Outline:

(04:17) What do we want from an honesty policy?

(06:40) Falsehoods that don't necessarily ruin credibility

(09:25) More complicated: Behavioral science

(12:29) The proposed policies

(12:43) No deception about deals

(19:27) Honesty string

(21:44) Compensation

(25:32) Communicating commitments

(27:01) Miscellaneous Q&A

(30:12) Conclusion

(30:53) Acknowledgments

(31:07) Appendix: How does no deception about deals apply to past experiments

(31:23) Sleeper agents

(31:52) Alignment faking

(33:08) Discovering language model behaviors

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Subtitle: AGI before 2029 now seems substantially less likely.

As I discussed in a prior post, I felt like there were some reasonably compelling arguments for expecting very fast AI progress in 2025 (especially on easily verified programming tasks). Concretely, this might have looked like reaching 8 hour 50% reliability horizon lengths on METR's task suite1 by now due to greatly scaling up RL and getting large training runs to work well. In practice, I think we've seen AI progress in 2025 which is probably somewhat faster than the historical rate (at least in terms of progress on agentic software engineering tasks), but not much faster. And, despite large scale-ups in RL and now seeing multiple serious training runs much bigger than GPT-4 (including GPT-5), this progress didn't involve any very large jumps.

The doubling time for horizon length on METR's task suite has been around 135 days [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Subtitle: To wit: LLM APIs, agent scaffolds, code review, and detection-and-response systems.

To prevent potentially misaligned LLM agents from taking actions with catastrophic consequences, you can try to monitor LLM actions - that is, try to detect dangerous or malicious actions, and do something about it when you do (like blocking the action, starting an investigation, …).1

But where in your infrastructure should your LLM monitor be implemented, and what team in your organization should own it? Historically, researchers have mostly focused on implementing monitoring inside the agent scaffold, which would mean that it would be owned by agent product teams. But actually there are also three other important places where you might put it, and I think these are comparable in importance to LLM monitors in agent scaffolds.

This is important for a few reasons:

1. This might inspire us to focus on developing better monitoring [...]

---

Outline:

(01:55) Monitors in agent scaffolds

(04:17) Monitors in cyber threat detection-and-response systems

(07:41) Monitors in LLM APIs

(10:32) Monitors in code reviews

(12:08) So where should you put the monitors?

(13:29) Appendix: Further subdivisions

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: To wit: LLM APIs, agent scaffolds, code review, and detection-and-response systems.

To prevent potentially misaligned LLM agents from taking actions with catastrophic consequences, you can try to monitor LLM actions - that is, try to detect dangerous or malicious actions, and do something about it when you do (like blocking the action, starting an investigation, …).1

But where in your infrastructure should your LLM monitor be implemented, and what team in your organization should own it? Historically, researchers have mostly focused on implementing monitoring inside the agent scaffold, which would mean that it would be owned by agent product teams. But actually there are also three other important places where you might put it, and I think these are comparable in importance to LLM monitors in agent scaffolds.

This is important for a few reasons:

1. This might inspire us to focus on developing better monitoring [...]

---

Outline:

(01:55) Monitors in agent scaffolds

(04:19) Monitors in cyber threat detection-and-response systems

(07:50) Monitors in LLM APIs

(10:44) Monitors in code reviews

(12:20) So where should you put the monitors?

(14:08) Appendix: Further subdivisions

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: Maybe we should (re)assess the case for relatively fast progress after the GPT-5 release..

Around the early o3 announcement (and maybe somewhat before that?), I felt like there were some reasonably compelling arguments for putting a decent amount of weight on relatively fast AI progress in 2025 (and maybe in 2026):

• Maybe AI companies will be able to rapidly scale up RL further because RL compute is still pretty low (so there is a bunch of overhang here); this could cause fast progress if the companies can effectively directly RL on useful stuff or RL transfers well even from more arbitrary tasks (e.g. competition programming)

• Maybe OpenAI hasn't really tried hard to scale up RL on agentic software engineering and has instead focused on scaling up single turn RL. So, when people (either OpenAI themselves or other people like Anthropic) scale up RL on [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Subtitle: Maybe we should (re)assess the case for relatively fast progress after the GPT-5 release..

Around the early o3 announcement (and maybe somewhat before that?), I felt like there were some reasonably compelling arguments for putting a decent amount of weight on relatively fast AI progress in 2025 (and maybe in 2026):

• Maybe AI companies will be able to rapidly scale up RL further because RL compute is still pretty low (so there is a bunch of overhang here); this could cause fast progress if the companies can effectively directly RL on useful stuff or RL transfers well even from more arbitrary tasks (e.g. competition programming)

• Maybe OpenAI hasn't really tried hard to scale up RL on agentic software engineering and has instead focused on scaling up single turn RL. So, when people (either OpenAI themselves or other people like Anthropic) scale up RL on [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Subtitle: It's like making challenging evals, but more constrained.

One of our main activities at Redwood is writing follow-ups to previous papers on control like the original and Ctrl-Z, where we construct a setting with a bunch of tasks (e.g. APPS problems) and a notion of safety failure (e.g. backdoors according to our specific definition), then play the adversarial game where we develop protocols and attacks on those protocols.

It turns out that a substantial fraction of the difficulty here is developing the settings, rather than developing the protocols and attacks. I want to explain why this is the case.

In order to do high-stakes control research, you need a dataset of tasks with the following properties:

• You can check whether the main task was completed successfully.

• There's some notion of safety failure that we consider unacceptable; we can check whether a safety failure occurred. [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Subtitle: It's like making challenging evals, but more constrained.

One of our main activities at Redwood is writing follow-ups to previous papers on control like the original and Ctrl-Z, where we construct a setting with a bunch of tasks (e.g. APPS problems) and a notion of safety failure (e.g. backdoors according to our specific definition), then play the adversarial game where we develop protocols and attacks on those protocols.

It turns out that a substantial fraction of the difficulty here is developing the settings, rather than developing the protocols and attacks. I want to explain why this is the case.

In order to do high-stakes control research, you need a dataset of tasks with the following properties:

• You can check whether the main task was completed successfully.

• There's some notion of safety failure that we consider unacceptable; we can check whether a safety failure occurred. [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.