At last. Highlights: some details on Preparedness Framework evals + evals by METR and Apollo.

Preparedness framework evaluations

You should follow the link and read this section.

Brief comments:

• Cyber: the setup sounds good (but maybe substantially more powerful scaffolding/prompting is possible). Separately, I wish OpenAI shared the tasks (or a small random sample of them) or at least said more about where they came from. (Recall that DeepMind shared CTF tasks.)
• Bio uplift: GPT-4o clearly boosts users on biological threat creation tasks — OpenAI doesn't say that but shows a graph. (It continues to be puzzling that novices score similarly to experts.) (I kinda worry that this is the wrong threat model—most bio risk from near-future models comes from a process that looks pretty different from a bigger boost to users like these—but I don't have better ideas for evals.)
• Persuasion: unclear whether substantially [...]

---

Outline:

(00:16) Preparedness framework evaluations

(01:34) Third party assessments

(03:26) Context

---

First published:

Source:

Narrated by TYPE III AUDIO.