I have liked music very much since I was a teenager. I spent many hours late at night in Soulseek chat rooms talking about and sharing music with my online friends. So, I tend to just have some music floating around in my head on any given day. But, I never learned to play any instrument, or use any digital audio software. It just didn't catch my interest.

My wife learned to play piano as a kid, so we happen to have a keyboard sitting around in our apartment. One day I was bored so I decided to just see whether I could figure out how to play some random song that I was thinking about right then. I found I was easily able to reconstitute a piano version of whatever melody I was thinking of, just by brute-forcing which notes were which, given a lot of patience. So [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Summary:

We think a lot about aligning AGI with human values. I think it's more likely that we’ll try to make the first AGIs do something else. This might intuitively be described as trying to make instruction-following (IF) or do-what-I-mean-and-check (DWIMAC) be the central goal of the AGI we design. Adopting this goal target seems to improve the odds of success of any technical alignment approach. This goal target avoids the hard problem of specifying human values in an adequately precise and stable way, and substantially helps with goal misspecification and deception by allowing one to treat the AGI as a collaborator in keeping it aligned as it becomes smarter and takes on more complex tasks.

This is similar but distinct from the goal targets of prosaic alignment efforts. Instruction-following is a single goal target that is [...]

---

Outline:

(01:35) Overview/Intuition

(01:39) How to use instruction-following AGI as a collaborator in alignment

(03:02) Instruction-following is safer than value alignment in a slow takeoff

(05:00) Relation to existing alignment approaches

(08:45) DWIMAC as goal target - more precise definition

(11:25) Intuition: a good employee follows instructions as they were intended

(14:30) Alignment difficulties reduced:

(14:34) Learning from examples is not precise enough to reliably convey alignment goals

(15:11) Solving ethics well enough to launch sovereign AGI is hard.

(15:37) Alignment difficulties remaining or made worse:

(15:42) Deceptive alignment is possible, and interpretability work does not seem on track to fully address this.

(17:24) Power remains in the hands of humans

(19:33) Well that just sounds like slavery with extra steps

(20:19) Maximizing goal following my be risky

(21:25) Conclusion

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

[Reminder: I am an internet weirdo with no medical credentials]

A few months ago, I published some crude estimates of the power of nitric oxide nasal spray to hasten recovery from illness, and speculated about what it could do prophylactically. While working on that piece a nice man on Twitter alerted me to the fact that humming produces lots of nasal nitric oxide. This post is my very crude model of what kind of anti-viral gains we could expect from humming.

I’ve encoded my model at Guesstimate. The results are pretty favorable (average estimated impact of 66% reduction in severity of illness), but extremely sensitive to my made-up numbers. Efficacy estimates go from ~0 to ~95%, depending on how you feel about publication bias, what percent of Enovid's impact can be credited to nitric oxide, and humming's relative effect. Given how made up speculative some [...]

---

Outline:

(01:20) Math

(01:23) Estimating the impact of Enovid

(02:33) How much nitric oxide does Enovid release?

(03:36) How much nitric oxide does humming move into the nose?

(04:25) Estimating the impact of humming

(06:08) How to hum

(07:52) Are there downsides?

(08:24) The barest scraps of other evidence

(09:24) Reasons to disbelieve

(10:23) Conclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.

MIRI updates:

• MIRI is shutting down the Visible Thoughts Project.
  • We originally announced the project in November of 2021. At the time we were hoping we could build a new type of data set for training models to exhibit more of their inner workings. MIRI leadership is pessimistic about humanity's ability to solve the alignment problem in time, but this was an idea that seemed relatively promising to us, albeit still a longshot.
  • We also hoped that the $1+ million bounty on the project might attract someone who could build an organization to build the data set. Many of MIRI's ambitions are bottlenecked on executive capacity, and we hoped that we might find individuals (and/or a process) that could help us spin up more projects without requiring a large amount of oversight from MIRI leadership.
  • Neither hope played out, and in the intervening time [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This article contains more than 100 uses of logical or mathematical notation, so an audio narration would be too hard to follow. You'll find a link to the original text in the episode description.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I stayed up too late collecting way-past-deadline papers and writing report cards. When I woke up at 6, this anxious email from one of my g11 Computer Science students was already in my Inbox.

Student: Hello Mr. Carle, I hope you've slept well; I haven't.

I've been seeing a lot of new media regarding how developed AI has become in software programming, most relevantly videos about NVIDIA's new artificial intelligence software developer, Devin.

Things like these are almost disheartening for me to see as I try (and struggle) to get better at coding and developing software. It feels like I'll never use the information that I learn in your class outside of high school because I can just ask an AI to write complex programs, and it will do it much faster than I would.

I'd like to know what your thoughts on this are. Do you think AI [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Ilya Sutskever and Jan Leike have resigned. It seems that John Schulman will lead Superalignment. Jakub Pachocki replaced Sutskever as Chief Scientist.

Reasons are unclear (as usual when safety people resign from OpenAI).

OpenAI announced Sutskever's resignation in a blogpost.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Caspar Oesterheld came up with two of the most important concepts in my field of work: Evidential Cooperation in Large Worlds and Safe Pareto Improvements. He also came up with a potential implementation of evidential decision theory in boundedly rational agents called decision auctions, wrote a comprehensive review of anthropics and how it interacts with decision theory which most of my anthropics discussions built on, and independently decided to work on AI some time late 2009 or early 2010.

Needless to say, I have a lot of respect for Caspar's work. I’ve often felt very confused about what to do in my attempts at conceptual research, so I decided to ask Caspar how he did his research. Below is my writeup from the resulting conversation.

How Caspar came up with surrogate goals

The process

• Caspar had spent six months FTE thinking about a specific bargaining problem [...]

---

Outline:

(00:54) How Caspar came up with surrogate goals

(00:59) The process

(02:44) Caspar's reflections on what was important during the process

(04:33) How Caspar came up with ECL

(04:38) The process

(07:02) Caspar's reflections on what was important during the process

(07:24) How Caspar came up with decision auctions

(07:29) The process

(09:44) How Caspar decided to work on superhuman AI in late 2009 or early 2010

(10:25) The process

(13:09) Caspar's reflections on what was important during the process

(14:30) General notes on his approach to research

(14:34) What does research concretely look like in his case?

(15:17) Research immersion

(16:36) Goal orientation vs. curiosity orientation

---

First published:

Source:

Narrated by TYPE III AUDIO.

This post outlines an efficient implementation of Edge Patching that massively outperforms common hook-based implementations. This implementation is available to use in my new library, AutoCircuit, and was first introduced by Li et al. (2023).

What is activation patching?

I introduce new terminology to clarify the distinction between different types of activation patching.

Node Patching

Node Patching (aka. “normal” activation patching) is when some activation in a neural network is altered from the value computed by the network to some other value. For example we could run two different prompts through a language model and replace the output of Attn 1 when the model is given some input 1 with the output of the head when the model is given some other input 2.

We will use the running example of a tiny, 1-layer transformer, but this approach generalizes to any transformer and any residual network.

All [...]

---

Outline:

(00:20) What is activation patching?

(00:30) Node Patching

(01:22) Edge Patching

(01:59) Path Patching

(03:21) Fast Edge Patching

(05:34) Performance Comparison

(06:19) Mask Gradients

(08:24) Appendix: Path Patching vs. Edge Patching

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.

RULESET

Each alien has a different amount of HP:

*Threat has no effect on combat directly - it's a measure of how threatening Earth considers each alien to be, which scales how many soldiers they send. (The war has been getting worse - early on, Earth sent on average ~1 soldier/4 Threat of aliens, but today it's more like 1 soldier/6 Threat. The wave you're facing has 41 Threat, Earth would send on average ~7 soldiers to it. Earth doesn't exercise much selection with weapons [...]

---

Outline:

(00:28) RULESET

(03:26) STRATEGY

(08:55) LEADERBOARD

(11:05) REFLECTION and FEEDBACK REQUEST

---

First published:

Source:

Narrated by TYPE III AUDIO.