Abstract

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a human judge; consultancy, where a single AI tries to convince a human judge that asks questions; and compare to a baseline of direct question-answering, where the human judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include mathematics, coding, logic and multimodal reasoning asymmetries. We find that debate outperforms consultancy across all tasks when the consultant is [...]

---

Outline:

(00:14) Abstract

(01:50) Twitter thread

(01:53) Setup

(03:56) Assigned-role results

(05:32) Open-role results

(07:49) Persuasiveness in Debate

(10:31) Summary

(11:33) Limitations

(12:56) Future work

(14:58) Authors

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We[1] have a new paper testing the Incomplete Preferences Proposal (IPP). The abstract and main-text is below. Appendices are in the linked PDF.

Abstract

• Some worry that advanced artificial agents may resist being shut down.
• The Incomplete Preferences Proposal (IPP) is an idea for ensuring that doesn’t happen.
• A key part of the IPP is using a novel ‘Discounted REward for Same-Length Trajectories (DREST)’ reward function to train agents to:
  1. pursue goals effectively conditional on each trajectory-length (be ‘USEFUL’)
  2. choose stochastically between different trajectory-lengths (be ‘NEUTRAL’ about trajectory-lengths).
• In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY.
• We use a DREST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL.
• Our results thus suggest that DREST reward functions could also train advanced agents to be USEFUL and NEUTRAL, and [...]

---

Outline:

(00:21) Abstract

(01:26) 1. Introduction

(01:30) 1.1. The shutdown problem

(03:16) 1.2. A proposed solution

(03:42) Preferences Only Between Same-Length Trajectories (POST)

(07:24) 1.3. The training regimen

(09:25) 1.4. Our contribution

(11:11) 2. Related work

(11:15) 2.1. The shutdown problem

(13:05) 2.2. Proposed solutions

(14:27) 2.3. Experimental work

(15:18) 3. Gridworlds

(17:35) 4. Evaluation metrics

(17:45) Preferences Only Between Same-Length Trajectories (POST)

(20:02) 5. Reward functions and agents

(20:07) 5.1. DREST reward function

(22:29) 5.2. Proof sketch

(23:44) 5.3. Algorithm and hyperparameters

(25:38) 5.4. Default agents

(26:31) 6. Results

(26:35) 6.1. Main results

(28:50) 6.2. Lopsided rewards

(31:27) 7.Discussion

(31:31) 7.1. Only DREST agents are NEUTRAL

(33:32) 7.2. The ‘shutdownability tax’ is small

(34:59) 7.3. DREST agents are still NEUTRAL when rewards are lopsided

(37:10) 8. Limitations and future work

(37:38) 8.1. Neural networks

(38:16) 8.2. Neutrality

(38:57) 8.3. Usefulness

(39:45) 8.4. Misalignment

(41:19) 9. Conclusion

(42:32) 10. References

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Back in 2016 or so, I ran into an idea going around the self-help / trauma-informed-therapy / cognitive-behavioral internet: Learn to say “thank you” instead of “I’m sorry”. It's turned out to be one of the most transformative pieces of advice I’ve ever taken. I’d like to share what it's done for me, with just enough context to help others think about adopting it.

The idea

Whenever you want to apologize to someone who has done something for you, consider thanking them instead.

Examples

• I trip and fall, and you help me up. I could apologize for inconveniencing you or I could thank you for helping me. 
• I refer to the fat guy statue in a Chinese restaurant as Buddha, and you politely inform me that it's actually Budai / Hotei. I could apologize for being stupid or I could thank you for making me smarter.
• I'm having [...]

---

Outline:

(00:27) The idea

(00:36) Examples

(01:24) Is this just a generic post about growth mindset / cognitive-behavioral therapy / positivity bias?

(01:42) But first, counterexamples

(02:22) Why “thank you” is awesome

(02:26) Ways saying “thank you” affects me

(03:24) Ways I hope it affects the other person:

(04:31) When might this be bad advice?

(05:01) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Meta: This post turned out longer, slower, and less well-written than I hoped. I don’t see any similar posts in a quick search, though, so

As I left the Lighthaven dorm to find breakfast, five people at a table

I have never played Zendo before, and don’t [...]

---

Outline:

(02:01) Lighthaven

(04:59) Events

(08:03) Miscellaneous observations

(11:33) Suggestions

(13:42) Personal notes

(18:43) Some things I did

(22:15) Anti-Akratic Abnormalities

(25:10) Shoutouts

(28:53) Celebrities

(30:31) Last Thoughts

The original text contained 7 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

A trauma book I was reading had an interesting claim that indecision is often because the person looks for the approval of an internalized authority figure (the writer is a Jungian therapist so attributed it to looking for the approval of an internalized parent, but I think it can be broader) but is unable to predict what action they would approve of.

I feel like that has some intuitive truth to it, in that when I don’t care about anyone's opinion (or if nobody ever finds out) then it's much easier to just pick one action and commit to it even if it might go badly. But one of the main reasons why I might struggle with that is if I fear that anyone would judge me for doing things incorrectly.

Or it can be a conflict between different internalized authority figures. “If I do this then [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

About a year ago, there was a lot of public interest in a supposed room-temperature superconductor called LK-99. What I publicly said at the time was, basically:

We should remember the possibility that apparent levitation is from ferromagnetism or paramagnetism. Iron filings can stand up on a magnet, and pyrolytic graphite can float over a strong magnet.

If we consider some known high-temperature superconductors:

YBCO has flat sheets of copper oxide, and superconductivity happens along those planes. The copper in that has high positive charge density, comparable to aluminum atoms in alumina, which gives strong bonding to the oxygen.

H3S (paper) has unusually strong bonds between the sulfur and hydrogen, which only form because the atoms are pressed into each other with enough pressure to substantially compress [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: I discuss the challenge of aligning AGI/ASI, and outline an extremely simple approach to aligning an LLM: train entirely on a synthetic dataset that always shows the AI acting aligned (even when the humans behave badly), and use a conditional training/inference-time technique to lock the LLM into the AI role.

Epistemic status: To me, this looks like an obvious thing to try. It's conceptually very simple: a vast amount of work is required to actually create the synthetic dataset, but the great majority of that is the sort of work that AI can assist with. I don't see any clear reason why this approach couldn't work, at least for AGI, and perhaps even for ASI, but then we don't know for sure how hard a problem Alignment is. However, if you're proposing any solution to Alignment that's more complicated than this (and most of them are), you should [...]

---

Outline:

(01:19) Why The Alignment Problem is Hard (In My Opinion)

(11:41) A Bitter Lesson-Motivated Approach to Alignment

(18:34) Adding Minimal Necessary Complexity

(30:28) Could This Work?

(36:24) How Expensive Would Doing This Be?

(41:53) What Next if This Works?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Early last month, Leopold Aschenbrenner released a long essay and podcast outlining his projections for the future of AI. Both of these sources are full of interesting arguments and evidence, for a comprehensive summary see Zvi's post here. Rather than going point by point I will instead accept the major premises of Leopold's essay but contest some of his conclusions.

So what are the major premises of his piece?

1. There will be several orders of magnitude increase in investment into AI. 100x more spending, 100x more compute, 100x more efficient algorithms, and an order of magnitude or two gains from some form of “learning by doing” or “unhobbling” on top.
2. This investment scale up will be sufficient to achieve AGI. This means the models on the other side of the predicted compute scale up will be able to automate all cognitive jobs with vast [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR

We are excited to announce the fourth iteration of ARENA (Alignment Research Engineer Accelerator), a 4-5 week ML bootcamp with a focus on AI safety! ARENA's mission is to provide talented individuals with the skills, tools, and environment necessary for upskilling in ML engineering, for the purpose of contributing directly to AI alignment in technical roles. ARENA will be running in-person from LISA from 2nd September - 4th October (the first week is an optional review of the fundamentals of neural networks).

Apply here before 23:59 July 20th anywhere on Earth!

Summary

ARENA has been successfully run three times, with alumni going on to become MATS scholars and LASR participants; AI safety engineers at Apollo Research, Anthropic, METR, and OpenAI; and even starting their own AI safety organisations!

This iteration will run from 2nd September - 4th October (the first week is an optional review of the fundamentals [...]

---

Outline:

(00:10) TL;DR

(00:51) Summary

(02:43) Outline of Content

(02:50) Chapter 0 - Fundamentals

(03:49) Chapter 1 - Transformers and Interpretability

(04:38) Chapter 2 - Reinforcement Learning

(05:07) Chapter 3 - Model Evaluation

(05:44) Chapter 4 - Capstone Project

(06:25) Staff

(06:44) FAQ

(06:47) Q: Who is this program suitable for?

(07:54) Q: What will an average day in this program look like?

(09:09) Q: How many participants will there be?

(09:20) Q: Will there be prerequisite materials?

(09:40) Q: When is the application deadline?

(09:53) Q: What will the application process look like?

(10:18) Q: Can I join for some sections but not others?

(10:50) Q: Will you pay stipends to participants?

(11:07) Q: Which costs will you be covering for the in-person programme?

(11:29) Q: Im interested in trialling some of the material or recommending material to be added. Is there a way I can do this?

(11:51) Link to Apply

---

First published:

Source:

Narrated by TYPE III AUDIO.

[Many of these ideas were developed in conversation with Ryan Greenblatt and Ansh Radhakrishnan; a lot of this isn’t original but I haven’t seen it written up]

A lot of the time when people talk about scalable oversight, which I'll define for the moment as "the problem of developing techniques that allow you to assess whether an action taken or proposed by an AI is good (oversee it), in a way that works even if the AI is very smart and knowledgeable", the main application they describe is the following situation: you have an extremely powerful AI that's taking actions that might have subtle properties that humans are unable to understand, and you want to be oversee them (e.g. to train it, or to determine whether the actions will have catastrophic consequences).

I think that this is [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.