We’ve just released BashArena, a new high-stakes control setting we think is a major improvement over the settings we’ve used in the past. In this post we’ll discuss the strengths and weaknesses of BashArena, and what we’ve learned about how to make settings for high stakes control research.

BashArena

Overview

BashArena is a control setting: a dataset designed to support the kind of adversarial game we studied in the original AI control paper and Ctrl-Z. In general, control settings require two types of tasks – main tasks and side tasks. Main tasks represent the work the AI is supposed to do, and side tasks represent security failures a malicious AI might cause. The red team constructs an attack policy that attempts to accomplish the side tasks, while the blue team constructs control protocols that detect and prevent attempts to achieve side tasks without hurting an honest policy's performance on main tasks.

We think BashArena offers several advantages over existing control settings:

1. More complex attacks. BashArena tasks support many different ways to attack and ways to disguise attacks as benign. Future AI deployments are likely to be more complex than any datasets we can currently create, so more [...]

---

Outline:

(00:27) BashArena

(00:30) Overview

(02:07) Main tasks

(04:23) Side tasks

(07:27) Weaknesses

(09:12) Lessons in Control Setting Design

(09:41) The core goal: insights that transfer

(11:13) What we already knew about how to design BashArena

(13:25) How BashArena addresses these challenges

(15:33) General principles for control setting design

(18:48) More specific lessons we learned while building BashArena

(19:13) Information design

(19:17) Side task test code should not reveal too much about attack surface

(20:44) Main task test code should be private

(22:00) Integration requirements are stricter than expected

(22:19) Partial credit

(24:39) Targeted deferral

(26:01) The gap

(26:57) We can (mostly) fix these integration problems by tying the blue team's hands

(27:39) Ban on dynamic permission management

(28:16) Limited side task knowledge during protocol design

(29:44) Remaining imperfection

(30:19) Closing thoughts

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.