This time around, we cover the Hanson/Alexander debates on the value of medicine, and otherwise we mostly have good news.

Technology Advances

Regeneron administers a single shot in a genetically deaf child's ear, and they can hear after a few months, n=2 so far.

Great news: An mRNA vaccine in early human clinical trials reprograms the immune system to attack glioblastoma, the most aggressive and lethal brain tumor. It will now proceed to Phase I. In a saner world, people would be able to try this now.

More great news, we have a cancer vaccine trial in the UK.

And we’re testing personalized mRNA BioNTech canner vaccines too.

US paying Moderna $176 million to develop a pandemic vaccine against bird flu.

We also have this claim that Lorlatinib jumps cancer PFS rates from 8% to 60%.

The GLP-1 Revolution

Early [...]

---

Outline:

(00:12) Technology Advances

(01:04) The GLP-1 Revolution

(04:46) Claims About Hansoninan Medicine

(18:36) Pricing

(19:17) Epistemics

(19:39) DEA Worse Than FDA

(21:19) Study Harder

(23:05) FDA Delenda Est

(24:51) Bioethics

(27:28) Covid

(29:21) Demons

(32:48) Genetics

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

by Anne Marthe van der Bles, Sander van der Linden, Alexandra L. J. Freeman, and David J. Spiegelhalter. (2020)

https://www.pnas.org/doi/pdf/10.1073/pnas.1913678117.

Summary: Numerically expressing uncertainty when talking to the public is fine. It causes people to be less confident in the number itself (as it should), but does not cause people to lose trust in the source of that number.

Uncertainty is inherent to our knowledge about the state of the world yet often not communicated alongside scientific facts and numbers. In the “posttruth” era where facts are increasingly contested, a common assumption is that communicating uncertainty will reduce public trust. However, a lack of systematic research makes it difficult to evaluate such claims.

Within many specialized communities, there are norms which encourage people to state numerical uncertainty when reporting a number. This is not often done when speaking to the public. The public might not understand what the uncertainty [...]

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Introduction

I happen to be in that happy stage in the research cycle where I ask for money so I can continue to work on things I think are important. Part of that means justifying what I want to work on to the satisfaction of the people who provide that money.

This presents a good opportunity to say what I plan to work on in a more layman-friendly way, for the benefit of LessWrong, potential collaborators, interested researchers, and funders who want to read the fun version of my project proposal

It also provides the opportunity for people who are very pessimistic about the chances I end up doing anything useful by pursuing this to have their say. So if you read this (or skim it), and have critiques (or just recommendations), I'd love to hear them! Publicly or privately.

So without further ado, in this post I will [...]

---

Outline:

(00:06) Introduction

(02:40) Reinforcement learning

(02:44) Agentic AIs vs Tool AIs

(04:54) Data walls

(07:05) Values

(07:49) Ok, but concretely what will you actually do?

(11:10) Call to action

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Editor's note: Somewhat after I posted this on my own blog, Max Chiswick cornered me at LessOnline / Manifest and gave me a whole new perspective on this topic. I now believe that there is a way to use poker to sharpen epistemics that works dramatically better than anything I had been considering. I hope to write it up—together with Max—when I have time. Anyway, I'm still happy to keep this post around as a record of my first thoughts on the matter, and because it's better than nothing in the time before Max and I get around to writing up our joint second thoughts.

As an epilogue to this story, Max and I are now running a beta test for a course on making AIs to play poker and other games. The course will a synthesis of our respective theories of pedagogy re [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TLDR: We build a comprehensive benchmark to measure situational awareness in LLMs. It consists of 16 tasks, which we group into 7 categories and 3 aspects of situational awareness (self-knowledge, situational inferences, and taking actions).

We test 19 LLMs and find that all perform above chance, including the pretrained GPT-4-base (which was not subject to RLHF finetuning). However, the benchmark is still far from saturated, with the top-scoring model (Claude-3.5-Sonnet) scoring 54%, compared to a random chance of 27.4% and an estimated upper baseline of 90.7%.

This post has excerpts from our paper, as well as some results on new models that are not in the paper.

Links: Twitter thread, Website (latest results + code), Paper

---

Outline:

(01:27) Abstract

(03:28) Introduction

(05:17) Motivation

(06:38) Benchmark

(08:45) Examples and prompts

(10:44) Results

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This summer, I’m supervising some research fellows through Cambridge's ERA AI Fellowship. The program started last week, and I’ve had conversations with about 6 fellows about their research projects & summer goals.

In this post, I’ll highlight a few pieces of advice I’ve found myself regularly giving to research fellows. This post reflects my own opinions and does not necessarily reflect the views of others at ERA.

Prioritize projects that have a clear target audience

Problem: One of the most common reasons why research products fail to add value is that they do not have a target audience. I think it can be easy to find a topic that is interesting/important, spend several months working on it, produce a 20-50 page paper, and then realize that you have no particular stakeholder(s) who find the work action-relevant.

Advice: Try to brainstorm what specific individuals you would want to [...]

---

Outline:

(00:32) Prioritize projects that have a clear target audience

(02:38) Be specific about your end products

(05:26) Take advantage of your network (and others’ networks)

(08:00) Miscellaneous

---

First published:

Source:

Narrated by TYPE III AUDIO.

Alice: A lot of people are talking about Singular Learning Theory. Do you know what it is?

Bob: I do. (pause) Kind of.

Alice: Well, I don't. Explanation time?

Bob: Uh, I'm not really an expert on it. You know, there's a lot of materials out there that--

Alice: that I realistically won't ever actually look at. Or, I've looked at them a little, but I still have basically no idea what's going on. Maybe if I watched a dozen hours of introductory lectures I'd start to understand it, but that's not currently happening.

What I really want is a short overview of what's going on. That's self-contained. And easy to follow. Aimed at a non-expert. And which perfectly answers any questions I might have. So, I thought I'd ask you!

Bob: Sorry, I'm actually really not--

Alice: Pleeeease?

[pause]

Bob: Ah, fine, I'll try.

So, you might [...]

---

Outline:

(02:47) 1. Theoretical foundations

(13:27) 2. Practical side

The original text contained 8 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Pantheon is an experimental LLM interface exploring a different type of human-AI interaction. We created this as a part of the cyborgism project, with the abstract motivation of augmenting the human ability to think by integrating human and AI generated thoughts.

How it works:

1. A human user “thinks out loud” by typing out their thoughts one at a time. This leaves a text trace of their stream of thought.
2. AI characters (called daemons) read this trace, and interact with the user by responding asynchronously with comments and questions.

The core distinguishing philosophy is that, while most apps are about a human prompting an AI to do useful mental work, Pantheon is the opposite. Here, AI does the prompting, and the goal is for the AI generated questions or comments to cause the human user to think in ways they would not have on their own. At worst, the [...]

---

Outline:

(01:33) How do I use Pantheon?

(02:10) What do the daemons see?

(03:00) Trees, branching, and sections

(03:51) How do I save my progress?

(04:03) What are “AI Suggestions”?

(04:17) What is “Ask AI”?

(04:58) Can I control which daemon responds?

(05:14) What is the target use case?

(05:57) How do I make my own daemons?

(08:04) How does this relate to Loom?

(08:58) Does Pantheon use base models?

(10:41) Is Pantheon open source?

(11:03) Should I be concerned about data privacy?

(11:24) Acknowledgments

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Abstract

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a human judge; consultancy, where a single AI tries to convince a human judge that asks questions; and compare to a baseline of direct question-answering, where the human judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include mathematics, coding, logic and multimodal reasoning asymmetries. We find that debate outperforms consultancy across all tasks when the consultant is [...]

---

Outline:

(00:14) Abstract

(01:50) Twitter thread

(01:53) Setup

(03:56) Assigned-role results

(05:32) Open-role results

(07:49) Persuasiveness in Debate

(10:31) Summary

(11:33) Limitations

(12:56) Future work

(14:58) Authors

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We[1] have a new paper testing the Incomplete Preferences Proposal (IPP). The abstract and main-text is below. Appendices are in the linked PDF.

Abstract

• Some worry that advanced artificial agents may resist being shut down.
• The Incomplete Preferences Proposal (IPP) is an idea for ensuring that doesn’t happen.
• A key part of the IPP is using a novel ‘Discounted REward for Same-Length Trajectories (DREST)’ reward function to train agents to:
  1. pursue goals effectively conditional on each trajectory-length (be ‘USEFUL’)
  2. choose stochastically between different trajectory-lengths (be ‘NEUTRAL’ about trajectory-lengths).
• In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY.
• We use a DREST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL.
• Our results thus suggest that DREST reward functions could also train advanced agents to be USEFUL and NEUTRAL, and [...]

---

Outline:

(00:21) Abstract

(01:26) 1. Introduction

(01:30) 1.1. The shutdown problem

(03:16) 1.2. A proposed solution

(03:42) Preferences Only Between Same-Length Trajectories (POST)

(07:24) 1.3. The training regimen

(09:25) 1.4. Our contribution

(11:11) 2. Related work

(11:15) 2.1. The shutdown problem

(13:05) 2.2. Proposed solutions

(14:27) 2.3. Experimental work

(15:18) 3. Gridworlds

(17:35) 4. Evaluation metrics

(17:45) Preferences Only Between Same-Length Trajectories (POST)

(20:02) 5. Reward functions and agents

(20:07) 5.1. DREST reward function

(22:29) 5.2. Proof sketch

(23:44) 5.3. Algorithm and hyperparameters

(25:38) 5.4. Default agents

(26:31) 6. Results

(26:35) 6.1. Main results

(28:50) 6.2. Lopsided rewards

(31:27) 7.Discussion

(31:31) 7.1. Only DREST agents are NEUTRAL

(33:32) 7.2. The ‘shutdownability tax’ is small

(34:59) 7.3. DREST agents are still NEUTRAL when rewards are lopsided

(37:10) 8. Limitations and future work

(37:38) 8.1. Neural networks

(38:16) 8.2. Neutrality

(38:57) 8.3. Usefulness

(39:45) 8.4. Misalignment

(41:19) 9. Conclusion

(42:32) 10. References

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.