Recently, Anthropic released Opus 4 and said they couldn't rule out the model triggering ASL-3 safeguards due to the model's CBRN capabilities. That is, they say they couldn't rule out that this model had "the ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons" (quoting from Anthropic's RSP). More specifically, Anthropic is worried about the model's capabilities in assisting with bioweapons. (See footnote 3 here.)
Given this and results on Virology Capabilities Test, it seems pretty likely that various other AI companies have or will soon have models which can significantly help amateurs make bioweapons.[1] One relevant question is whether it would be bad if there were open-weight models above this capability threshold. Further, should people advocate for not releasing open-weight models above this capability level?
In this post, I'll discuss how I think about releasing [...]
---
Outline:
(02:45) Costs and benefits of open-weight models with these CBRN capabilities
(08:12) Implications of this cost-benefit situation
(11:39) When would my views on open weights change?
(14:32) Mitigations
The original text contained 10 footnotes which were omitted from this narration.
---
First published:
Source:
---
Narrated by TYPE III AUDIO.
View all episodes
By LessWrongRecently, Anthropic released Opus 4 and said they couldn't rule out the model triggering ASL-3 safeguards due to the model's CBRN capabilities. That is, they say they couldn't rule out that this model had "the ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons" (quoting from Anthropic's RSP). More specifically, Anthropic is worried about the model's capabilities in assisting with bioweapons. (See footnote 3 here.)
Given this and results on Virology Capabilities Test, it seems pretty likely that various other AI companies have or will soon have models which can significantly help amateurs make bioweapons.[1] One relevant question is whether it would be bad if there were open-weight models above this capability threshold. Further, should people advocate for not releasing open-weight models above this capability level?
In this post, I'll discuss how I think about releasing [...]
---
Outline:
(02:45) Costs and benefits of open-weight models with these CBRN capabilities
(08:12) Implications of this cost-benefit situation
(11:39) When would my views on open weights change?
(14:32) Mitigations
The original text contained 10 footnotes which were omitted from this narration.
---
First published:
Source:
---
Narrated by TYPE III AUDIO.
26,469 Listeners
2,395 Listeners
7,953 Listeners
4,142 Listeners
89 Listeners
1,472 Listeners
9,207 Listeners
88 Listeners
426 Listeners
5,461 Listeners
15,321 Listeners
482 Listeners
121 Listeners
75 Listeners
461 Listeners