LessWrong (30+ Karma)

“The o1 System Card Is Not About o1” by Zvi


Listen Later

Or rather, we don’t actually have a proper o1 system card, aside from the outside red teaming reports. At all.

Because, as I realized after writing my first draft of this, the data here does not reflect the o1 model they released, or o1 pro?

I think what happened is pretty bad on multiple levels.

  1. The failure to properly communicate the information they did provide.
  2. The failure to provide the correct information.
  3. The failure, potentially, to actually test the same model they released, in many of the ways we are counting on to ensure the model is safe to release.
  4. The failure to properly elicit model capabilities. o1 scores unreasonably poorly on a large portion of the preparedness tests, to the point where I would not take the tests seriously as assessments of what o1 is capable of doing, even at [...]
  5. ---

    Outline:

    (02:18) Where Art Thou o1 System Card?

    (05:35) Introduction (Section 1)

    (06:01) Model Data and Training (Section 2)

    (06:13) Challenges and Evaluations (Section 3)

    (09:38) Jailbreak Evaluations (Section 3.1.2)

    (11:33) Regurgitation (3.1.3) and Hallucinations (3.1.4)

    (12:30) Fairness and Bias (3.1.5)

    (13:33) Jailbreaks Through Custom Developer Messages (3.2)

    (14:41) Chain of Thought Safety (3.3)

    (18:52) External Red Teaming Via Pairwise Safety Comparisons (3.4.1)

    (19:57) Jailbreak Arena (3.4.2)

    (20:25) Apollo Research (3.4.3) and the ‘Escape Attempts’

    (21:38) METR (3.4.4) and Autonomous Capability

    (25:22) Preparedness Framework Evaluations (Section 4)

    (27:47) Mitigations

    (30:27) Cybersecurity

    (31:22) Chemical and Biological Threats (4.5)

    (31:52) Radiological and Nuclear Threat Creation (4.6)

    (32:21) Persuasion (4.7)

    (32:49) Model Autonomy (4.8)

    (34:45) Multilingual Performance

    (34:55) Conclusion

    The original text contained 17 images which were described by AI.

    ---

    First published:

    December 13th, 2024

    Source:

    https://www.lesswrong.com/posts/HfigEyXddxkSGunKr/the-o1-system-card-is-not-about-o1

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

    ...more
    View all episodesView all episodes
    Download on the App Store

    LessWrong (30+ Karma)By LessWrong


    More shows like LessWrong (30+ Karma)

    View all
    Making Sense with Sam Harris by Sam Harris

    Making Sense with Sam Harris

    26,350 Listeners

    Conversations with Tyler by Mercatus Center at George Mason University

    Conversations with Tyler

    2,392 Listeners

    The Peter Attia Drive by Peter Attia, MD

    The Peter Attia Drive

    7,955 Listeners

    Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas by Sean Carroll | Wondery

    Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas

    4,128 Listeners

    ManifoldOne by Steve Hsu

    ManifoldOne

    87 Listeners

    Your Undivided Attention by Tristan Harris and Aza Raskin, The Center for Humane Technology

    Your Undivided Attention

    1,445 Listeners

    All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

    All-In with Chamath, Jason, Sacks & Friedberg

    8,909 Listeners

    Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

    Machine Learning Street Talk (MLST)

    88 Listeners

    Dwarkesh Podcast by Dwarkesh Patel

    Dwarkesh Podcast

    372 Listeners

    Hard Fork by The New York Times

    Hard Fork

    5,426 Listeners

    The Ezra Klein Show by New York Times Opinion

    The Ezra Klein Show

    15,326 Listeners

    Moonshots with Peter Diamandis by PHD Ventures

    Moonshots with Peter Diamandis

    466 Listeners

    No Priors: Artificial Intelligence | Technology | Startups by Conviction

    No Priors: Artificial Intelligence | Technology | Startups

    122 Listeners

    Latent Space: The AI Engineer Podcast by swyx + Alessio

    Latent Space: The AI Engineer Podcast

    76 Listeners

    BG2Pod with Brad Gerstner and Bill Gurley by BG2Pod

    BG2Pod with Brad Gerstner and Bill Gurley

    450 Listeners