Say that we have a set of options, such as (for example) wild animal

Say also that you have two axes along which you can score those

Assume that we (for some reason) can't convert between and compare those

Should you then pick an intervention that is a compromise on the two

One thing you might consider is the distribution of options

Intuitively, the distributions [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

The fun at OpenAI continues.

We finally have the details of how Leopold Aschenbrenner was fired, at least according to Leopold. We have a letter calling for a way for employees to do something if frontier AI labs are endangering safety. And we have continued details and fallout from the issues with non-disparagement agreements and NDAs.

Hopefully we can stop meeting like this for a while.

Due to jury duty and it being largely distinct, this post does not cover the appointment of General Paul Nakasone to the board of directors. I’ll cover that later, probably in the weekly update.

The Firing of Leopold Aschenbrenner

What happened that caused Leopold to leave OpenAI? Given the nature of this topic, I encourage getting the story from Leopold by following along on the transcript of that section of his appearance on the Dwarkesh Patel Podcast or [...]

---

Outline:

(00:43) The Firing of Leopold Aschenbrenner

(11:24) Daniel Kokotajlo Speaks and The Right to Warn

(15:07) The Right to Warn Letter

(18:48) Signed by (alphabetical order):

(19:34) Endorsed by (alphabetical order):

(34:17) You’ll Be Hearing From Our Lawyer

(35:29) Possession is Nine Tenths of the Law

(41:31) What I Can Tell You I Used To Not Be Able To Tell You

(47:50) Clarifying the Mission

(52:04) Sam Altman Told the SEC He Was Chairman of YC

(53:26) YC Has an Investment in OpenAI

(54:33) OpenAI is Hiring a Lot of Lobbyists

(55:05) OpenAI Says They Value Privacy

(55:58) Microsoft Went Around the Safety Board

(56:39) I Don’t Really Know What You Were Expecting

(57:12) Where Did Everybody Go?

(59:43) In Other OpenAI News

---

First published:

Source:

Narrated by TYPE III AUDIO.

Or: Towards Bayesian Natural Language Semantics In Terms Of Interoperable Mental Content

Or: Towards a Theory of Interoperable Semantics

You know how natural language “semantics” as studied in e.g. linguistics is kinda bullshit? Like, there's some fine math there, it just ignores most of the thing which people intuitively mean by “semantics”.

When I think about what natural language “semantics” means, intuitively, the core picture in my head is:

• I hear/read some words, and my brain translates those words into some kind of internal mental content.
• The mental content in my head somehow “matches” the mental content typically evoked in other peoples’ heads by the same words, thereby allowing us to communicate at all; the mental content is “interoperable” in some sense.

That interoperable mental content is “the semantics of” the words. That's the stuff we’re going to try to model.

The main goal of this post is [...]

---

Outline:

(01:13) But Why Though?

(02:34) Overview

(04:19) What's The Problem?

(04:22) Central Problem 1: How To Model The Magic Box?

(05:34) Subproblem: What's The Range Of Box Outputs?

(06:48) SubSubProblem: What Can Children Attach Words To?

(08:15) Summary So Far

(09:18) Central Problem 2: How Do Alice and Bob “Agree” On Semantics?

(11:41) Summary: Interoperable Semantics

(13:07) First (Toy) Model: Clustering + Naturality

(14:54) Equivalence Via Naturality

(17:02) A Quick Empirical Check

(22:22) Strengths and Shortcomings of This Toy Model

(24:59) Aside: What Does “Grounding In Spacetime Locality” Mean?

(26:22) Second (Toy) Model Sketch: Rigid Body Objects

(27:44) The Teacup

(33:46) Geometry and Trajectory Clusters

(36:27) Strengths and Shortcomings of This Toy Model

(39:07) Summary and Call To Action

(40:10) Call To Action

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This article contains more than fifty uses of logical or mathematical notation, so an audio narration would be too hard to follow. You'll find a link to the original text in the episode description.

---

First published:

Source:

Narrated by TYPE III AUDIO.

“Appetitive” and “Consummatory” are terms used in the animal behavior literature. I was was briefly confused when I first came across these terms (a year or two ago), because I’m most comfortable thinking in terms of brain algorithms, whereas these terms were about categories of behavior, and the papers I was reading didn’t spell out how the one is related to the other.

I’m somewhat embarrassed to write this because the thesis seems so extremely obvious to me now, and it's probably obvious to many other people too. So if you read the title of this post and were thinking “yeah duh”, then you already get it, and you can stop reading.

Definition of “appetitive” and “consummatory”

In animal behavior there's a distinction between “appetitive behaviors” and “consummatory behaviors”. Here's a nice description from Hansen et al. 1991 (formatting added, references omitted):

It is sometimes helpful to break down [...]

---

Outline:

(00:44) Definition of “appetitive” and “consummatory”

(02:55) How do “appetitive” and “consummatory” relate to brain algorithms?

(04:53) Caveats

(05:57) Example of a time when this came up for me: the MPOA→VTA pathway in parental behavior

---

First published:

Source:

Narrated by TYPE III AUDIO.

The room was cozy despite its size, with wood-lined walls reflecting the dim lighting. At one end, a stone fireplace housed a roaring fire; in the middle stood a huge oak table. The woman seated at the head of it rapped her gavel. “I hereby call to order the first meeting of the Parliamentary Subcommittee on Intergalactic Colonization. We’ll start with brief opening statements, for which each representative will be allocated one minute, including—”

“Oh, enough with the pomp, Victoria. It's just the four of us.” The representative for the Liberal Democrats waved his hand around the nearly-empty room.

Victoria sniffed. “It's important, Stuart. This is a decision that will have astronomical implications. And it's recorded, besides, so we should do things by the book. Carla, you’re up first.”

The woman at the end of the table stood with a smile. “Thank you, Victoria. I’m [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

MIRI updates

• MIRI Communications Manager Gretta Duleba explains MIRI's current communications strategy. We hope to clearly communicate to policymakers and the general public why there's an urgent need to shut down frontier AI development, and make the case for installing an “off-switch”. This will not be easy, and there is a lot of work to be done. Some projects we’re currently exploring include a new website, a book, and an online reference resource.
   
• Rob Bensinger argues, contra Leopold Aschenbrenner, that the US government should not race to develop artificial superintelligence. “If anyone builds it, everyone dies.” Instead, Rob outlines a proposal for the US to spearhead an international alliance to halt progress toward the technology.
   
• At the end of June, the Agent Foundations team, including Scott Garrabrant and others, will be parting ways with MIRI to continue their work as independent researchers. The team was [...]

---

Outline:

(00:07) MIRI updates

(02:46) News and links

---

First published:

Source:

Narrated by TYPE III AUDIO.

Produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort under the supervision of Evan Hubinger.

Acknowledgements: Thanks to Kyle Brady for his many contributions to this project.

This post argues that the performance elicited by fine-tuning an LLM on a task using a given prompt format does not usefully bound the level of performance observed when the same information is presented in a different structure. Thus, fine-tuned performance provides very little information about the best performance that would be achieved by a large number of actors fine-tuning models with random prompting schemes in parallel.

In particular, we find that we get much better results from fine-tuning gpt-3.5-turbo (ChatGPT 3.5) to play chess when the game so far is presented in a single block of SAN[1] than when the game so far is separated into a series of SAN moves presented as [...]

---

Outline:

(03:06) General Setting

(04:44) LLM Prompting is Very Brittle and Can Elicit a Range of Behaviors

(06:46) Experiment 1: Semantic Prompt Variations

(09:09) Experiment 2: Structural Prompt Variations

(11:31) LLM Fine-tuning is Brittle and Can Elicit a Range of Behaviors

(14:21) Conclusion

(16:16) What's next?

(16:47) Appendix: Prompt specifications

(17:01) Completions model:

(17:23) Chat model tournament notation:

(18:09) Chat model move-by-move notation:

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

In our new video, we talk about research on interpreting InceptionV1, a convolutional neural network. Researchers have been able to understand the function of neurons and channels inside the network and uncover visual processing algorithms by looking at the weights. The work on InceptionV1 is early but landmark mechanistic interpretability research, and it functions well as an introduction to the field. We also go into the rationale and goals of the field and mention some more recent research near the end. Our main source material is the circuits thread in the Distill journal and this article on feature visualization. The author of the script is Arthur Frost. I have included the script below, although I recommend watching the video since the script has been written with accompanying moving visuals in mind.

Intro

In 2018, researchers trained an AI to find out if people were at [...]

---

Outline:

(00:56) Intro

(07:16) Visualisation by Optimisation

(11:09) Circuits

(15:27) Polysemanticity

(17:00) Closing thoughts, and the past few years of interpretability

---

First published:

Source:

Narrated by TYPE III AUDIO.

And is it a good model for value learning in AI?

TLDR

Shard theory proposes a view of value formation where experiences lead to the creation of context-based ‘shards’ that determine behaviour. Here, we go over psychological and neuroscientific views of learning, and find that while shard theory's emphasis on context bears similarity to types of learning such as conditioning, it does not address top-down influences that may decrease the locality of value-learning in the brain.

What's Shard Theory (and why do we care)?

In 2022, Quintin Pope and Alex Turner posted ‘The shard theory of human values’, where they described their view of how experiences shape the value we place on things. They give an example of a baby who enjoys drinking juice, and eventually learns that grabbing at the juice pouch, moving around to find the juice pouch, and modelling where the [...]

---

Outline:

(00:08) TLDR

(00:38) What's Shard Theory (and why do we care)?

(02:01) A Crash Course on Human Learning

(16:30) Where does Shard Theory come in?

(24:52) Conclusions

(26:09) Next Steps

(26:34) Thanks

---

First published:

Source:

Narrated by TYPE III AUDIO.