---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

Source:

---

Narrated by TYPE III AUDIO.

Recently, Anthropic released Opus 4 and said they couldn't rule out the model triggering ASL-3 safeguards due to the model's CBRN capabilities. That is, they say they couldn't rule out that this model had "the ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons" (quoting from Anthropic's RSP). More specifically, Anthropic is worried about the model's capabilities in assisting with bioweapons. (See footnote 3 here.)

Given this and results on Virology Capabilities Test, it seems pretty likely that various other AI companies have or will soon have models which can significantly help amateurs make bioweapons.[1] One relevant question is whether it would be bad if there were open-weight models above this capability threshold. Further, should people advocate for not releasing open-weight models above this capability level?

In this post, I'll discuss how I think about releasing [...]

---

Outline:

(02:45) Costs and benefits of open-weight models with these CBRN capabilities

(08:12) Implications of this cost-benefit situation

(11:39) When would my views on open weights change?

(14:32) Mitigations

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Edit on 08/06/2024: At least one person has pointed out that, at one point, giving hypertensives at night were also thought to matter, a now disproven idea. Someone also mentioned how many times the clinical trial information was altered during the study. I added in a section at the end to discuss this.

There's a really interesting phenomenon in the immunotherapy field that has been going on for what seems to be several years now, but was raised to me — a non-oncologist — via a viral Twitter thread of some work at ASCO25:

Translating the jargon: amongst the patients who received their immunotherapy infusion before 3pm (as opposed to after 3pm), their cancer stayed under control for longer (11.3 months vs. 5.7 months) and on median lived longer (at least[1] 23.2 months versus 16.4 months). A near 2x~ improvement in the most important metrics doing something [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

METR just made a lovely post detailing many examples they've found of reward hacks by frontier models. Unlike the reward hacks of yesteryear, these models are smart enough to know that what they are doing is deceptive and not what the company wanted them to do.

---

First published:

Source:

Linkpost URL:
https://metr.org/blog/2025-06-05-recent-reward-hacking/

---

Narrated by TYPE III AUDIO.

Disempowerment is on the fence, gets interpreted as either implying human extinction or being a good place. "Doom" tends to be ambiguous between disempowerment and extinction, as well as about when that outcome must be gauged. And many people currently feel both disempowered and OK, so see eutopia as similar to disempowerment, neither an example of "doom".

Arguments pointing to risk of human extinction run into the issue of people expecting disempowerment without extinction, when some of the same arguments would remain relevant if applied directly to disempowerment (including the moral arguments about extinction or disempowerment being a problem). And arguments pointing to desirability of establishing eutopia run into the issue of people expecting disempowerment to be approximately as good and in practice much more likely. When the distinctions between these levels of doom are not maintained, conflation makes it harder to meaningfully disagree.

Eutopia Without Disempowerment

---

Outline:

(01:06) Eutopia Without Disempowerment

(02:13) Yudkowskys Arguments and Disempowerment

(02:52) Noncentral Fallacy of Doom

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

AI companies claim that their models are safe on the basis of dangerous capability evaluations. OpenAI, Google DeepMind, and Anthropic publish reports intended to show their eval results and explain why those results imply that the models' capabilities aren't too dangerous.[1] Unfortunately, the reports mostly don't support the companies' claims. Crucially, the companies usually don't explain why they think the results, which often seem strong, actually indicate safety, especially for biothreat and cyber capabilities. (Additionally, the companies are undereliciting and thus underestimating their models' capabilities, and they don't share enough information for people on the outside to tell how bad this is.)

Bad explanation/contextualization

OpenAI biothreat evals: OpenAI says "several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold." It doesn't say how it concludes this (or what results [...]

---

Outline:

(00:54) Bad explanation/contextualization

(04:34) Dubious elicitation

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR We reproduce emergent misalignment (Betley et al. 2025) in Qwen2.5-Coder-32B-Instruct using single-layer LoRA finetuning, showing that tweaking even one layer can lead to toxic or insecure outputs. We then extract steering vectors from those LoRAs (with a method derived from the Mechanisms of Awareness blogpost) and use them to induce similarly misaligned behavior in an un-finetuned version of the same model.

We take the results to support two main claims:

1. Single-layer LoRAs are sufficient to induce emergent misalignment.
    
2. Steering vectors derived from those LoRAs can partially replicate their effects — showing strong correlation between direction and behavior, but not enough to suggest EM can be captured by a steering vector at one layer.

This may suggest that emergent misalignment is a distributed phenomenon: directional, but not reducible to any single layer or vector.

Reproducing Previous Results

For our intents in this post, we will be summarizing Betley [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.