MIRI updates:

• MIRI is shutting down the Visible Thoughts Project.
  • We originally announced the project in November of 2021. At the time we were hoping we could build a new type of data set for training models to exhibit more of their inner workings. MIRI leadership is pessimistic about humanity's ability to solve the alignment problem in time, but this was an idea that seemed relatively promising to us, albeit still a longshot.
  • We also hoped that the $1+ million bounty on the project might attract someone who could build an organization to build the data set. Many of MIRI's ambitions are bottlenecked on executive capacity, and we hoped that we might find individuals (and/or a process) that could help us spin up more projects without requiring a large amount of oversight from MIRI leadership.
  • Neither hope played out, and in the intervening time [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This article contains more than 100 uses of logical or mathematical notation, so an audio narration would be too hard to follow. You'll find a link to the original text in the episode description.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I stayed up too late collecting way-past-deadline papers and writing report cards. When I woke up at 6, this anxious email from one of my g11 Computer Science students was already in my Inbox.

Student: Hello Mr. Carle, I hope you've slept well; I haven't.

I've been seeing a lot of new media regarding how developed AI has become in software programming, most relevantly videos about NVIDIA's new artificial intelligence software developer, Devin.

Things like these are almost disheartening for me to see as I try (and struggle) to get better at coding and developing software. It feels like I'll never use the information that I learn in your class outside of high school because I can just ask an AI to write complex programs, and it will do it much faster than I would.

I'd like to know what your thoughts on this are. Do you think AI [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Ilya Sutskever and Jan Leike have resigned. It seems that John Schulman will lead Superalignment. Jakub Pachocki replaced Sutskever as Chief Scientist.

Reasons are unclear (as usual when safety people resign from OpenAI).

OpenAI announced Sutskever's resignation in a blogpost.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Caspar Oesterheld came up with two of the most important concepts in my field of work: Evidential Cooperation in Large Worlds and Safe Pareto Improvements. He also came up with a potential implementation of evidential decision theory in boundedly rational agents called decision auctions, wrote a comprehensive review of anthropics and how it interacts with decision theory which most of my anthropics discussions built on, and independently decided to work on AI some time late 2009 or early 2010.

Needless to say, I have a lot of respect for Caspar's work. I’ve often felt very confused about what to do in my attempts at conceptual research, so I decided to ask Caspar how he did his research. Below is my writeup from the resulting conversation.

How Caspar came up with surrogate goals

The process

• Caspar had spent six months FTE thinking about a specific bargaining problem [...]

---

Outline:

(00:54) How Caspar came up with surrogate goals

(00:59) The process

(02:44) Caspar's reflections on what was important during the process

(04:33) How Caspar came up with ECL

(04:38) The process

(07:02) Caspar's reflections on what was important during the process

(07:24) How Caspar came up with decision auctions

(07:29) The process

(09:44) How Caspar decided to work on superhuman AI in late 2009 or early 2010

(10:25) The process

(13:09) Caspar's reflections on what was important during the process

(14:30) General notes on his approach to research

(14:34) What does research concretely look like in his case?

(15:17) Research immersion

(16:36) Goal orientation vs. curiosity orientation

---

First published:

Source:

Narrated by TYPE III AUDIO.

This post outlines an efficient implementation of Edge Patching that massively outperforms common hook-based implementations. This implementation is available to use in my new library, AutoCircuit, and was first introduced by Li et al. (2023).

What is activation patching?

I introduce new terminology to clarify the distinction between different types of activation patching.

Node Patching

Node Patching (aka. “normal” activation patching) is when some activation in a neural network is altered from the value computed by the network to some other value. For example we could run two different prompts through a language model and replace the output of Attn 1 when the model is given some input 1 with the output of the head when the model is given some other input 2.

We will use the running example of a tiny, 1-layer transformer, but this approach generalizes to any transformer and any residual network.

All [...]

---

Outline:

(00:20) What is activation patching?

(00:30) Node Patching

(01:22) Edge Patching

(01:59) Path Patching

(03:21) Fast Edge Patching

(05:34) Performance Comparison

(06:19) Mask Gradients

(08:24) Appendix: Path Patching vs. Edge Patching

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.

RULESET

Each alien has a different amount of HP:

*Threat has no effect on combat directly - it's a measure of how threatening Earth considers each alien to be, which scales how many soldiers they send. (The war has been getting worse - early on, Earth sent on average ~1 soldier/4 Threat of aliens, but today it's more like 1 soldier/6 Threat. The wave you're facing has 41 Threat, Earth would send on average ~7 soldiers to it. Earth doesn't exercise much selection with weapons [...]

---

Outline:

(00:28) RULESET

(03:26) STRATEGY

(08:55) LEADERBOARD

(11:05) REFLECTION and FEEDBACK REQUEST

---

First published:

Source:

Narrated by TYPE III AUDIO.

A stance against student debt cancellation doesn’t rely on the assumptions of any single ideology. Strong cases against student debt cancellation can be made based on the fundamental values of any section of the political compass. In no particular order, here are some arguments against student debt cancellation from the perspectives of many disparate ideologies.

Equity and Fairness

Student debt cancellation is a massive subsidy to an already prosperous and privileged population. American college graduates have nearly double the income of high school graduates. African Americans are far underrepresented among degree holders compared to their overall population share.

Within the group of college graduates debt cancellation increases equity, but you can’t get around the fact that 72% of African Americans have no student debt because they never went to college. The tax base for debt cancellation will mostly come from rich white college graduates, but [...]

---

Outline:

(00:29) Equity and Fairness

(01:59) Effective Altruism

(02:54) Student Debt Crisis

(03:42) Fiscal Conservativism

(04:43) Democracy

(05:34) Cultural Conservatism

(06:24) Progress Studies

(07:03) Divestment

(07:32) Marxism

(08:27) Biden's Campaign

---

First published:

Source:

Narrated by TYPE III AUDIO.

Do you ever go to a lecture, follow it thinking it makes total sense, then look back at your notes later and realize it makes no sense? This used to happen to me, but I’ve learned how to use spaced repetition to fully avoid this if I want. I’m going to try to convey this method in this post.

Much of my understanding of how to create flashcards comes from “Using spaced repetition systems to see through a piece of mathematics” by Michael Nielsen and “How to write good prompts: using spaced repetition to create understanding” by Andy Matuschak, but I think my method falls in between both, in terms of abstraction. Finally, I want to credit Quantum Country for being an amazing example of flashcards created to develop intuition in users.

My method is more abstract than Michael Nielsen's approach, since it does not [...]

---

Outline:

(01:23) Minimize Inferential Distance on Flashcards

(02:25) Understanding Attention

(03:38) Examples

---

First published:

Source:

Narrated by TYPE III AUDIO.

As I note in the third section, I will be attending LessOnline at month's end at Lighthaven in Berkeley. If that is your kind of event, then consider going, and buy your ticket today before prices go up.

This month's edition was an opportunity to finish off some things that got left out of previous editions or where events have left many of the issues behind, including the question of TikTok.

Oh No

All of this has happened before. And all of this shall happen again.

Alex Tabarrok: I regret to inform you that the CDC is at it again.

Marc Johnson: We developed an assay for testing for H5N1 from wastewater over a year ago. (I wasn’t expecting it in milk, but I figured it was going to poke up somewhere.)

However, I was just on a call with the CDC and [...]

---

Outline:

(00:29) Oh No

(03:36) Oh No: Betting on Elections

(05:47) Oh Yeah: LessOnline

(07:21) Brief Explanations

(08:33) Patrick McKenzie Monthly

(11:15) Enemies of the People

(12:41) Oh Canada

(15:24) This is NPR

(20:10) Technology Advances

(21:17) TikTok on the Clock

(32:51) Antisocial Media

(34:51) Prosocial Media

(36:01) At the Movies

(37:27) Media Trustworthiness Rankings

(41:40) Government Working

(46:57) Florida Man Bans Lab-Grown Meat

(51:22) Crime and Punishment

(54:00) El Salvador

(58:02) Variously Effective Altruism

(01:02:37) While I Cannot Condone This

(01:06:38) Can Money Buy Happiness?

(01:08:16) Good News, Everyone

(01:09:07) Can’t Sleep Clowns Will Eat Me

(01:10:53) How Great are Great People?

(01:13:20) Gamers Gonna Game Game Game Game Game

(01:21:21) Sports Go Sports

(01:25:20) I Was Promised Flying Self-Driving Cars

(01:26:40) News You Can Use

(01:28:00) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.