Recently, Anthropic released Opus 4 and said they couldn't rule out the model triggering ASL-3 safeguards due to the model's CBRN capabilities. That is, they say they couldn't rule out that this model had "the ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons" (quoting from Anthropic's RSP). More specifically, Anthropic is worried about the model's capabilities in assisting with bioweapons. (See footnote 3 here.)
Given this and results on Virology Capabilities Test, it seems pretty likely that various other AI companies have or will soon have models which can significantly help amateurs make bioweapons.[1] One relevant question is whether it would be bad if there were open-weight models above this capability threshold. Further, should people advocate for not releasing open-weight models above this capability level?
In this post, I'll discuss how I think about releasing [...]
---
Outline:
(02:45) Costs and benefits of open-weight models with these CBRN capabilities
(08:12) Implications of this cost-benefit situation
(11:39) When would my views on open weights change?
(14:32) Mitigations
The original text contained 10 footnotes which were omitted from this narration.
---
First published:
June 9th, 2025
Source:
https://www.lesswrong.com/posts/TeF8Az2EiWenR9APF/when-is-it-important-that-open-weight-models-aren-t-released
---
Narrated by TYPE III AUDIO.
View all episodes
