LessWrong (30+ Karma)

“GPT-4o System Card” by Zach Stein-Perlman


Listen Later

This is a link post.

At last. Highlights: some details on Preparedness Framework evals + evals by METR and Apollo.

Preparedness framework evaluations

You should follow the link and read this section.

Brief comments:

  • Cyber: the setup sounds good (but maybe substantially more powerful scaffolding/prompting is possible). Separately, I wish OpenAI shared the tasks (or a small random sample of them) or at least said more about where they came from. (Recall that DeepMind shared CTF tasks.)
  • Bio uplift: GPT-4o clearly boosts users on biological threat creation tasks — OpenAI doesn't say that but shows a graph. (It continues to be puzzling that novices score similarly to experts.) (I kinda worry that this is the wrong threat model—most bio risk from near-future models comes from a process that looks pretty different from a bigger boost to users like these—but I don't have better ideas for evals.)
  • Persuasion: unclear whether substantially [...]

---

Outline:

(00:16) Preparedness framework evaluations

(01:34) Third party assessments

(03:26) Context

---

First published:

August 8th, 2024

Source:

https://www.lesswrong.com/posts/nBtryzLzLnsHEiKRn/gpt-4o-system-card

---

Narrated by TYPE III AUDIO.

...more
View all episodesView all episodes
Download on the App Store

LessWrong (30+ Karma)By LessWrong


More shows like LessWrong (30+ Karma)

View all
The Daily by The New York Times

The Daily

112,843 Listeners

Astral Codex Ten Podcast by Jeremiah

Astral Codex Ten Podcast

130 Listeners

Interesting Times with Ross Douthat by New York Times Opinion

Interesting Times with Ross Douthat

7,214 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

531 Listeners

The Ezra Klein Show by New York Times Opinion

The Ezra Klein Show

16,223 Listeners

AI Article Readings by Readings of great articles in AI voices

AI Article Readings

4 Listeners

Doom Debates by Liron Shapira

Doom Debates

14 Listeners

LessWrong posts by zvi by zvi

LessWrong posts by zvi

2 Listeners