Epistemic status -- sharing rough notes on an important topic because I don't think I'll have a chance to clean them up soon.

Summary

Suppose a human used AI to take over the world. Would this be worse than AI taking over? I think plausibly:

• In expectation, human-level AI will better live up to human moral standards than a randomly selected human. Because:
  • Humans fall far short of our moral standards.
  • Current models are much more nice, patient, honest and selfless than humans.
    • Though human-level AI will have much more agentic training for economic output, and a smaller fraction of HHH training, which could make them less nice.
  • Humans are "rewarded" for immoral behaviour more than AIs will be
    • Humans evolved under conditions where selfishness and cruelty often paid high dividends, so evolution often "rewarded" such behaviour. And similarly, during lifetime learning we [...]

---

Outline:

(00:12) Summary

(02:20) AGI is nicer than humans in expectation

(04:53) Conditioning on AI actually seizing power

(08:18) Conditioning on the human actually seizing power

(09:31) Other considerations

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[This is an interim report and continuation of the work from the research sprint done in MATS winter 7 (Neel Nanda's Training Phase)]

Try out binary masking for a few residual saes in this colab notebook: [Github Notebook] [Colab Notebook]

TL;DR: 

We propose a novel approach to:

1. Scaling SAE Circuits to Large Models: By placing sparse autoencoders only in the residual stream at intervals, we find circuits in models as large as Gemma 9B without requiring SAEs to be trained for every transformer layer.
2. Finding Circuits: We develop a better circuit finding algorithm. Our method optimizes a binary mask over SAE latents, which proves significantly more effective than existing thresholding-based methods like Attribution Patching or Integrated Gradients.

Our discovered circuits paint a clear picture of how Gemma does a given task, with one circuit achieving 95% faithfulness with <20 total latents. This minimality lets us quickly understand the [...]

---

Outline:

(00:35) TL;DR:

(01:35) 1 Introduction

(04:30) 2 Background

(04:33) 2.1 SAEs

(05:19) 2.2 Circuits

(07:04) 2.3 Problems with Current Sparse Feature Interpretability Approaches

(07:10) 2.3.1 Scalability

(07:44) 2.3.2 Independent Scoring of Nodes

(08:25) 2.3.3 Error Nodes

(10:01) 3 Our Approach

(10:29) 3.1 Solving Scalability: Circuits with few residual SAEs

(11:21) 3.2 Solving Independent Scoring: Masking

(12:08) 3.3 Error nodes

(12:25) 4 Results

(12:29) 4.1 Setup

(13:49) 4.2 Performance Recovery

(15:00) 4.2.1 Code Output Prediction:

(16:48) 4.2.2 Subject Verb Agreement (SVA):

(17:42) 4.2.3 IOI:

(18:55) 4.3 Completeness

(22:22) 4.4 Mask Stability

(23:07) 5 Case Study: Code Output Prediction

(25:09) 6. Conclusions

(27:45) 7. Future Research and Ideas

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

MATS currently has more people interested in being mentors than we are able to support—for example, for the Winter 2024-25 Program, we received applications from 87 prospective mentors who cumulatively asked for 223 scholars[1] (for a cohort where we expected to only accept 80 scholars). As a result, we need some process for how to choose which researchers to take on as mentors and how many scholars to allocate each. Our desiderata for the process are as follows:

• We want to base our decisions on expert opinions of the quality of various research directions.
• We want the above opinions to be sourced from a range of perspectives with the AI existential safety field, to incorporate multiple perspectives on alignment research and other research areas we think are important, such as AI governance and policy, AI security, and other approaches for addressing AI catastrophic risk.
• We want [...]

---

Outline:

(00:04) Introduction

(02:17) Mentor selection

(08:04) Mentor demographics

(10:55) Scholar demographics

(13:57) Acknowledgements

The original text contained 3 footnotes which were omitted from this narration.

The original text contained 11 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Response to Discursive Games, Discursive Warfare

The discursive distortions you discuss serve two functions:

1 Narratives can only serve as effective group identifiers by containing fixed elements that deviate from what naive reason would think. In other words, something about the shared story has to be a costly signal of loyalty, and therefore a sign of a distorted map. An undistorted map would be advantageous for anyone regardless of group membership; a distorted map is advantageous only for people using it as an identifying trait. Commercial mapmakers will sometimes include phantom towns so that they (and courts) can distinguish competitors who plagiarized their work from competitors who independently mapped the same terrain. Point deer make horse can catalyze the formation of a faction because it reduces motive ambiguity in a way that "point deer make deer" could not.

"Not Invented Here" dynamics are part of this. To occupy territory [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: Making claims or demands about/into other people's internal states, rather than about your state or observable external states, predictably ties people in knots—instead: only make claims about your own experience or observables. This lets the other control the copy of them that's in the shared context.[1]

Non-Violent Communication rapidly grew into a world-spanning movement due to its remarkable ability to avert and dissolve conflict. The NVC book has a whole bunch of examples of communication procedures and practical tips. Here I'll try and convey the core insights more rapidly by focusing on the underlying principles I've extracted.[2]

Background models

People have self-models. People have models of other people. Conversations form shared context, which syncs with both of those. Some ways of conversing work better than others.

Syncing back an incorrect model from conversational context can disrupt or harm one's self-model, so dissonance between your self-model and others' models [...]

---

Outline:

(00:45) Background models

(01:13) Common collision in conversational context (Cartoons start here )

(02:32) Just talking about yourself is good, actually

(03:06) 2x2 (Purpose x Domain)

(04:09) Observables (Cartoons #2)

(05:15) Summary (slightly spaced repetition)

(06:05) Appendix: Speculation on sometimes spectacular successes

The original text contained 16 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

SPOILER WARNING: Extensive spoilers for XX.

XX was recommended by one of the comments on my Sci-Fi micro-reviews, so I read it. I can't seem to find any critical reviews of the book online so I'm going to while away a few hours of my life in a misguided attempt to reveal that the emperor has no clothes.

Typography

First let's dispense with XX's most distinguishing feature, then we'll get down to brass tacks. Rian Hughes plays around a huge amount with typography and tries to push the edges of text as a medium. This is an interesting quirk, and is sometimes appreciated, but sometimes makes the novel actively hostile to the reader. Reading through twenty pages of this quickly gets tiring:

Overall, I don't think it added that much to the story, but I guess some people would like it 🤷.

Length And Pacing

This book is long [...]

---

Outline:

(00:45) Typography

(01:28) Length And Pacing

(01:49) Ideas

(02:34) The Oxbow:

(04:44) Signal Compression

(06:44) Moving the grid

(07:49) Sentient Galaxies

(08:27) The signal as a virus

(09:12) Alternative physics

(09:45) Conclusion

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There's a conventional narrative by which the pre-20th century aristocracy was the “old corruption” where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a professional civil service after more enlightened thinkers prevailed. Orwell writes in 1941 (emphasis mine):

For long past there had been in England an entirely functionless class, living on money that was invested they hardly knew where, the ‘idle rich’, the people whose photographs you can look at in the Tatler and the Bystander, always supposing that you want to. The existence of these people was by any standard unjustifiable. They were simply parasites, less useful to society than his fleas are to a dog.

But this misses a key point about the pre-industrial world: it was difficult to monitor performance or even measure things in general. A modern bureaucracy can track hours worked, tasks completed, goods received [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

The Sun is the most nutritious thing that's reasonably close. It's only 8 light-minutes away, yet contains the vast majority of mass within 4 light-years of the Earth. The next-nearest star, Proxima Centauri, is about 4.25 light-years away.

By "nutritious", I mean it has a lot of what's needed for making computers: mass-energy. Seth Lloyd's "Ultimate physical limits to computation" imagines an "ultimate laptop" which is the theoretically best computer that is 1 kilogram of mass, contained in 1 liter. He notes a limit to calculations per second that is proportional to the energy of the computer, which is mostly locked in its mass (E = mc²). He also notes that computer memory is limited linearly by energy, and that energy expenditure can be reduced a lot with reversible computation.

So, you need energy to make computers out of (much more than you need energy to power [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This post reflects my personal opinion and not necessarily that of other members of Apollo Research or any of the people acknowledged below. Thanks to Jarrah Bloomfield, Lucius Bushnaq, Marius Hobbhahn, Axel Højmark, and Stefan Heimersheim for comments/discussions.

I find that people in the AI/AI safety community have not considered many of the important implications that security in AI companies has on catastrophic risks.

In this post, I’ve laid out some of these implications:

• AI companies are a long way from state-proof security
• Implementing state-proof security will slow down safety (and capabilities) research a lot
• Sabotage is sufficient for catastrophe
• What will happen if timelines are short?
• Security level matters, even if you’re not robust to top cyber operations

AI companies are a long way from state-proof security

I’m of course not the first one to make this claim (e.g. see Aschenbrenner). But it bears repeating.

Last year [...]

---

Outline:

(00:56) AI companies are a long way from state-proof security

(05:36) Implementing state-proof security will slow down safety (and capabilities) research a lot

(08:36) Sabotage is sufficient for catastrophe

(11:24) What will happen if timelines are short?

(14:34) Security level matters, even if you're not robust to top cyber operations

(15:21) Advice to frontier AI companies

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

tl;dr: LLMs rapidly improving at software engineering and math means lots of projects are better off as Google Docs until your AI agent intern can implement them.

Implementation keeps getting cheaper

Writing research code has gotten a lot faster over the past few years. Since 2021 and OpenAI Codex, new models and tools such as Cursor built around them have saved myself more and more time on coding every year.

This trend is accelerating fast: AI agents using Claude-3.5-Sonnet and o1-preview can do tasks that take ML researchers up to 2 hours of coding. This is without considering newer models such as o3, which do 70% on SWE-bench out of the box.
Yet this progress remains somewhat concentrated in implementation: progress on “soft” skills like idea generation has, as far as I can tell, been slower.

I’ve come to believe that, if you work in technical [...]

---

Outline:

(00:18) Implementation keeps getting cheaper

(01:33) When later means better

(03:01) Did I do well in 2024?

(07:29) Themes for temporally privileged work

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.