Subtitle: If it transfers misalignment, we might get a misaligned model that's easier to incriminate. If it doesn’t, we might get a capable benign replacement model.

Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen:

1. Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform.

2. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., because it is less capable). If so, auditing the distilled model might give us indirect evidence of the teacher's misalignment.

In a previous post we discussed the second possibility and proposed distillation for incrimination techniques: distillation methods that we hoped would transfer misalignment without transferring the ability to fool audits. In this post, we discuss the first possibility, and propose distillation for capabilities techniques: distillation methods that we hope will transfer capabilities without transferring misalignment.

Thanks to Carlo Leonardo Attubato, Eric Gan, Aniket Chakravorty, Francis Rhys Ward, Anders Woodruff, Alex Mallen, Buck Shlegeris, Julian Stastny and [...]

---

Outline:

(02:07) Why distillation might transfer capabilities but not misalignment

(03:10) Ideas for implementing distillation for capabilities

(04:55) Distillation double bind: If distillation for incrimination fails, then distillation for capabilities is somewhat likely to succeed

(05:38) Why distillation for capabilities and incrimination might both fail

(05:44) Reason 1: Capabilities and misalignment might be too tightly linked

(08:29) Reason 2: Context-dependent misalignment

(09:51) Reason 3: If U became misaligned in the first place, why won't M become misaligned for that same reason?

(10:34) How we can empirically study distillation techniques

(11:35) Appendix: details on ideas for implementing distillation for capabilities

(12:02) Basic ideas

(14:17) Idea 1: Iteratively filter and distill

(15:14) Idea 2: Inoculate against misalignment

(16:08) Idea 3: Reduce cognitive slack

(16:47) Idea 4: Distill and audit

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: Models' no-CoT time horizon has doubled roughly every year.

by Francis Rhys Ward, Dewi Gould, Anders Cairns Woodruff et al. (see full author list at the end)

PAPER LINK

About a year ago, METR showed that the length of tasks frontier models can reliably complete doubles every few months. A related safety-relevant question is this: what length of tasks can models complete without any chain of thought (CoT)?

If models can do extensive reasoning without outputting any CoT, it would have implications for safety. Developers and deployment-time monitors couldn’t easily understand models’ motivations and catch dangerous planning. Models that reason substantially without a CoT might also drift further from human patterns of thought, since their reasoning is no longer constrained by text in the pretraining prior. As a result, they would be harder to understand and might be more likely to scheme.

Extending Ryan Greenblatt's research, we investigate this by measuring models’ ability to complete tasks without any CoT on a suite of 43 benchmarks spanning different domains. We compare AI reasoning ability to humans using the estimated 50% time horizon (TH)---the typical time taken for a human to perform a task that the LLM performs with [...]

---

Outline:

(02:28) Methods

(04:59) Results

(06:34) FAQ

(08:09) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: When is "increasing safety budget" a useful concept?

I often use what I’ll call the “safety-usefulness tradeoff model”, which is: developers face a tradeoff between “safety” and “usefulness” of an AI deployment, and the developer has only limited willingness or ability to sacrifice usefulness for the sake of safety. This model assumes that developers choose whether to take safety-relevant actions based on their cost efficiency, i.e., the marginal safety gain relative to the cost. However, that is not necessarily true. In this post, I spell out different stories for how developers choose what safety-relevant actions to take, in order to clarify when this model is relevant and how strategies for reducing AI risk are affected when its assumptions don’t hold.

The model suggests two ways a safety-concerned person can increase safety:

• Safety tech improvements: push out the Pareto frontier, so that any given level of usefulness reduction buys more safety than it would have previously.

• Safety budget increase: increase the extent to which the developer sacrifices usefulness for safety. On the cheaper end, this means implementing safety measures; on the more expensive end, it might mean refraining from training or deploying models whose risks [...]

---

Outline:

(04:17) Rushed reasonable developers

(06:08) Limited political will

(08:42) This model is unhelpful if developers don't trade efficiently between safety and usefulness

(12:59) Overall thoughts

(14:26) Appendix: Definitions of safety and usefulness in the rushed reasonable developer model

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We’ve just released a new paper: Retrying vs Resampling in AI Control. We revisit the resampling protocols introduced in Ctrl-Z with an up-to-date setting and much stronger models, and compare them against “retrying” protocols similar to Claude Code auto mode or Codex Auto-review.

Motivation

Roughly a year ago we released Ctrl-Z, the first paper to study control techniques for agents. A headline result of that paper was the performance of resample protocols – strategies that involve taking multiple i.i.d. samples from the model per step. But since Ctrl-Z, models have gotten much stronger, and we have built more sophisticated control settings to keep up. We wanted to answer the following questions:

How well do the results from Ctrl-Z hold up with better models and a better setting? Current high stakes control research is trying to learn by analogy about how to do control effectively in a real high stakes deployment during a real intelligence explosion. Findings about technique performance1 are going to have to generalize pretty far to be useful.

If the resample protocols from Ctrl-Z still work, what makes them work? One way we try to make our work more generalizable is by understanding the dynamics governing outcomes [...]

---

Outline:

(00:30) Motivation

(02:35) TL;DR Takeaways

(04:34) Methodology

(07:14) Differences from Ctrl-Z

(12:14) Are Retrying Protocols Exploitable?

(15:25) Cost and Latency of Resampling

(17:24) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We’d like to develop training techniques that work when applied to future misaligned AI systems. One strategy for studying proposed techniques is to test them on model organisms. However, model organisms built with common techniques are often fragile: we (and other researchers like Roger et al. and Ryd et al.) have observed them to stop misbehaving after untargeted training—training that doesn’t directly target the misbehavior. For example, we have observed that simple untargeted training methods like “train the model to talk like a pirate” is effective against many model organisms that we have created, including many replications of prior work like Hubinger et al., Greenblatt et al., and Ryd et al..

Fragile model organisms aren’t very useful for technique development: when a sophisticated technique succeeds on one, you can’t tell whether the technique is good or the model organism is weak. For instance, if you come up with some complicated technique for generating high-quality responses and find that SFTing on these removes the bad behavior, that may just be because almost any SFT would have removed it.

This post identifies factors that make model organisms more robust to untargeted training. Our main findings are:

1. Prompted model organisms [...]

---

Outline:

(03:11) Experimental setup

(06:12) Main results

(06:15) Result 1: Untargeted training removes misalignment from prompted model organisms without degrading other capabilities

(12:32) Result 2: FWFT model organisms are more robust than LoRA model organisms

(15:09) Result 3: Backdoor behavior and trigger affect robustness

(16:37) Result 4: Password locking makes model organisms less robust

(20:20) Conclusion

(21:29) Factors that don't help much

(21:54) 1: Training longer doesn't seem increase robustness past a certain point for our backdoors

(22:33) 2: CoT-distilling doesn't increase robustness for our backdoors

(23:07) 3: Using larger models doesn't substantially increase robustness for 2 of our backdoors

(24:44) 4: There aren't clear trends for what types of backdoor triggers and behaviors are most robust

(25:32) 5: SOAP Optimizer doesn't help much

(26:35) 6: Weight Decay doesn't help

(27:18) Appendix:

(27:21) Appendix 1: Pirate SFT is better than QA SFT

(27:45) Appendix 2: Are blue team methods wiping previous training?

(29:30) Appendix 3: Backdoor list

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: Full automation likely yields a one-time speed-up and higher returns from compute.

This is a somewhat technical note.

By “software-only singularity”, I mean that, after full automation of AI R&D, progress gets faster and faster due to smarter AIs driving increasingly fast rates of improvement in algorithms (overcoming diminishing returns), and that this lasts long enough to yield a large amount of progress (e.g. at least 4 years of progress in 1 year). The equivalent statement in jargon is: r is significantly greater than 1 (implying progress is getting faster and faster) and this remains the case for long enough to get large amounts of progress. For context, see How quick and big would a software intelligence explosion be?

Even without a “software-only singularity”, I think full automation of AI R&D probably greatly speeds up progress for two main reasons:

• You get a one-time speed up from automation and this speed up seems like it will be pretty large (even with r<1). See How quick and big would a software intelligence explosion be? for discussion and see the AI Futures Model for an end-to-end model that naturally incorporates this effect. Quantitatively, with my median [...]

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen:

1. Misalignment fails to transfer to the student. If so, we get a fairly capable benign model.

2. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., due to being less capable). If so, we might get indirect evidence about the teacher's misalignment by auditing the distilled model.

In this post, we will discuss the second possibility, which we call incrimination via distillation. Specifically, we propose distillation methods that we hope transfer misalignment without transferring the ability to fool audits, and discuss why these techniques might work or fail. In a future post, we discuss the first possibility, and what distillation methods should be used when aiming to create a capable benign model.

We’re excited for research that empirically tests and refines this approach; if successful, this technique could become a valuable part of alignment audits.

How incrimination via distillation works

Powerful misaligned AI models might not be auditable: they might pass alignment audits but still act on their misaligned drives when given the chance. [...]

---

Outline:

(01:32) How incrimination via distillation works

(02:45) How we propose implementing incrimination via distillation

(03:59) Auditability-preserving distillation

(05:11) Misalignment-targeted distillation

(06:34) Why incrimination via distillation might work

(08:17) Why incrimination via distillation might not work

(10:53) Conclusion

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment.

Risk reports commonly use pre-deployment alignment assessments to measure misalignment risk from an internally deployed AI. However, an AI that genuinely starts out with largely benign motivations can develop widespread dangerous motivations during deployment. I think this is the most plausible route to consistent adversarial misalignment in the near future. So, AI companies and evaluators should substantively incorporate it into risk analysis and planning.

In this post, I’ll briefly argue why, absent improved mitigations, this will probably soon become a reason why AI companies will be unable to convincingly argue against consistent adversarial misalignment (this risk will perhaps be even larger than risk of consistent adversarial misalignment arising from training). Then I’ll discuss how well current risk reports address it (the Claude Mythos risk report does a reasonable job; others don’t).

Thanks to Ryan Greenblatt, Alexa Pan, Charlie Griffin, Anders Cairns Woodruff, and Buck Shlegeris for feedback on drafts.

Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment

In some contexts, AIs might adopt misaligned goals, even if they were otherwise previously aligned. Because this misalignment can be rare, the AI might [...]

---

Outline:

(01:21) Deployment-time spread is the most plausible near-term route to consistent adversarial misalignment

(06:15) Company risk reports

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Subtitle: My median guess: it's as good as a crystal ball that sees 2.5 months into the future.

This post was drafted by Buck, and substantially edited by Anders. “I” refers to Buck. Thanks to Alex Mallen for comments.

People who work inside AI companies get access to information that I only get later or never. Quantitatively, how big a deal is this access?

Here's an operationalization of this. Consider the following two ways my knowledge could be augmented:

• I get a crystal ball that tells me all the information I would know n months in the future.

• I become an employee of a frontier AI company (like OpenAI or Anthropic), with access to all the private information I’d normally get from working at that company.

How big would n have to be for me to be indifferent between these two options, from the perspective of learning things that are helpful for making AI go well?

The answer is presumably different for me than for many readers, because I’m a reasonably well-connected researcher; I see published information and news from the rumor mill and I talk to researchers at frontier AI companies all the time. [...]

---

Outline:

(03:20) What do insiders know?

(04:34) Safety work and corporate attitudes

(05:54) Model capabilities

(07:27) Algorithms and architecture

(09:49) How will this change over time?

(12:27) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Last week, OpenAI staff shared an early draft of Investigating the consequences of accidentally grading CoT during RL with Redwood Research staff.

To start with, I appreciate them publishing this post. I think it is valuable for AI companies to be transparent about problems like these when they arise. I particularly appreciate them sharing the post with us early, discussing the issues in detail, and modifying it to address our most important criticisms.

I think it will be increasingly important for AI companies to have a policy of getting external feedback on the risks posed by their deployments, and in particular having some external accountability on whether they have adequate evidence to support their claims about the level of risk posed; as an example of this, see METR reviewing Anthropic's Sabotage Risk Report. We at Redwood Research are interested in participating in this kind of external review of evidence about safety. So I am taking this as an opportunity to try out writing this kind of review. If you work at a frontier AI company, please feel free to reach out if you’d like our review of similar documents.

My overall assessment is that I mostly agree with the [...]

---

Outline:

(01:35) Assessing the evidence that CoT training did not damage monitorability

(10:37) How much does this analysis rely on information that wasnt provided?

(12:09) Small amounts of RL training on CoT might not be more important than other sources of CoT unreliability

(13:20) AI companies will eventually need to learn not to make mistakes like this

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.