This is a plan for how ASI could be relatively safely developed.

Abstract: A plan that puts all frontier model companies on a unified schedule of model training, evaluation and approval, with regulatory compliance promoted through market access. This aims to combine (most of) the economic benefits of unrestricted competition but with more safety, (most of) the time-to-think benefits of AI pauses but with better compliance incentives, and (most of) the central oversight of a Manhattan project but with more freedom and pluralism.

Background

It is based on the following worldview, though not all are cruxes:

1. Rushing to ASI by default leads to deceptive and misaligned ASI, with catastrophic consequences for humanity.
2. A lot of alignment work will be empirical and requires access to (near) cutting edge models to work with.
3. A lot of progress is driven by increased compute and it is possible to measure compute in [...]

---

Outline:

(00:39) Background

(02:23) The Plan

(04:57) Advantages

(07:18) Potential problems

(08:11) Do the frontier training runs have to be simultaneous?

(08:57) Quis custodiet ipsos custodes?

(10:19) How does this work internationally?

(12:25) To be determined

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

-I can be a nightmare conference attendee: I tend to ask nitpicky questions and apply a dose of skepticism to a speaker's claims which is healthy in doing one's own research, but probably not optimal when everyone else is trying to follow a talk. I'm working on being better at this, but for now I blame my background.

There is one nitpick that comes up again and again. In fact in one conference I brought it up so often that Jake Mendel coined a term for it: "Dmitry's koan".

In koan form, the nitpick is as follows:

There is no such thing as interpreting a neural network. There is only interpreting a neural network at a given scale of precision.

On its face, this observation is true but a bit banal. Indeed there are two extremes:

1. At the "less precise" extreme, you can claim you have interpreted [...]

---

Outline:

(04:13) Elucidating the spectrum of precision

(04:17) Step 1: coming to terms with imprecision

(10:57) Step 2: Factoring in the memorization-generalization spectrum

(18:31) Natural scale and natural degradation

(18:56) Sometimes reconstruction loss is not the point

(20:10) Degradation as a dial

(27:49) Natural scale

(30:31) Natural degradation

(35:08) Possible issues

(36:08) Experiment suggestions

The original text contained 12 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

There's one week left in the Discussion Phase of the 2023 Review. Here's a few things you might want to spend that time doing, in escalating order of effort:

Review three posts, without stressing much about the details. Go the Quick Review[1] page, pick some posts you remember and had opinions about. Reread them, and write a review.

Think about which posts should be in the Best of LessWrong. Go to the Advanced Review[2] page, look at which posts are currently in the top 50, and at the posts that aren't in the top 50 which you think should be. Write reviews arguing for posts you think are underappreciated.

Think big picture, longform thoughts about the conversations from 2023. After reviewing several posts, write a top level post with the 2023 Longform Review tag, exploring multiple posts from 2023 and how they tie together with today's thinking.

The Basic [...]

---

Outline:

(00:59) The Basic Ask: Reviews worth highlighting

(02:05) Specific prompts for reviewing:

(02:34) Prompts for Self-Reviews

(03:35) What should be in the Best of LessWrong?

(04:20) Longform Reflection

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Dwarkesh Patel again interviewed Tyler Cowen, largely about AI, so here we go.

Note that I take it as a given that the entire discussion is taking place in some form of an ‘AI Fizzle’ and ‘economic normal’ world, where AI does not advance too much in capability from its current form, in meaningful senses, and we do not get superintelligence [because of reasons]. It's still massive additional progress by the standards of any other technology, but painfully slow by the ‘AGI is coming soon’ crowd.

That's the only way I can make the discussion make at least some sense, with Tyler Cowen predicting 0.5%/year additional RGDP growth from AI. That level of capabilities progress is a possible world, although the various elements stated here seem like they are sometimes from different possible worlds.

I note that this conversation was recorded prior to o3 and all [...]

---

Outline:

(02:01) AI and Economic Growth

(02:28) Cost Disease

(09:03) The Lump of Intelligence Fallacy

(10:56) The Efficient Market Hypothesis is False

(13:05) Not Sending Your Best People

(20:49) Energy as the Bottleneck

(22:31) The Experts are Wrong But Trust Them Anyway

(25:41) AI as Additional Population

(26:51) Opposition to AI as the Bottleneck

(29:10) China as Existence Proof for Rapid Growth

(29:56) Second Derivatives

(31:05) Talent and Leadership

(33:26) Adapting to the Age of AI

(35:37) Identifying Alpha

(37:39) Old Man Yells at Crowd

(41:59) Some Statements for Everyone to Ponder

(43:13) No Royal Road to Wisdom

(47:29) Concluding Thoughts

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic's Alignment Science team conducts technical research aimed at mitigating the risk of catastrophes caused by future advanced AI systems, such as mass loss of life or permanent loss of human control. A central challenge we face is identifying concrete technical work that can be done today to prevent these risks. Future worlds where our research matters—that is, worlds that carry substantial catastrophic risk from AI—will have been radically transformed by AI development. Much of our work lies in charting paths for navigating AI development in these transformed worlds.

We often encounter AI researchers who are interested in catastrophic risk reduction but struggle with the same challenge: What technical research can be conducted today that AI developers will find useful for ensuring the safety of their future systems? In this blog post we share some of our thoughts on this question.

To create this post, we asked Alignment Science [...]

---

Outline:

(02:36) Evaluating capabilities

(04:16) Evaluating alignment

(06:03) Understanding model cognition

(09:01) Understanding how a models persona affects its behavior and how it generalizes out-of-distribution

(10:25) Chain-of-thought faithfulness

(12:00) AI control

(13:12) Behavioral monitoring

(15:11) Activation monitoring

(17:01) Anomaly detection

(18:42) Scalable oversight

(20:14) Improving oversight despite systematic, exploitable errors in the oversight signal

(22:24) Recursive oversight

(23:50) Weak-to-strong and easy-to-hard generalization

(27:16) Honesty

(28:45) Adversarial robustness

(29:54) Realistic and differential benchmarks for jailbreaks

(32:15) Adaptive defenses

(33:38) Miscellaneous

(33:49) Unlearning dangerous information and capabilities

(34:53) Learned governance for multi-agent alignment

(36:36) Acknowledgements

---

First published:

Source:

Narrated by TYPE III AUDIO.

Epistemic status -- sharing rough notes on an important topic because I don't think I'll have a chance to clean them up soon.

Summary

Suppose a human used AI to take over the world. Would this be worse than AI taking over? I think plausibly:

• In expectation, human-level AI will better live up to human moral standards than a randomly selected human. Because:
  • Humans fall far short of our moral standards.
  • Current models are much more nice, patient, honest and selfless than humans.
    • Though human-level AI will have much more agentic training for economic output, and a smaller fraction of HHH training, which could make them less nice.
  • Humans are "rewarded" for immoral behaviour more than AIs will be
    • Humans evolved under conditions where selfishness and cruelty often paid high dividends, so evolution often "rewarded" such behaviour. And similarly, during lifetime learning we [...]

---

Outline:

(00:12) Summary

(02:20) AGI is nicer than humans in expectation

(04:53) Conditioning on AI actually seizing power

(08:18) Conditioning on the human actually seizing power

(09:31) Other considerations

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[This is an interim report and continuation of the work from the research sprint done in MATS winter 7 (Neel Nanda's Training Phase)]

Try out binary masking for a few residual saes in this colab notebook: [Github Notebook] [Colab Notebook]

TL;DR: 

We propose a novel approach to:

1. Scaling SAE Circuits to Large Models: By placing sparse autoencoders only in the residual stream at intervals, we find circuits in models as large as Gemma 9B without requiring SAEs to be trained for every transformer layer.
2. Finding Circuits: We develop a better circuit finding algorithm. Our method optimizes a binary mask over SAE latents, which proves significantly more effective than existing thresholding-based methods like Attribution Patching or Integrated Gradients.

Our discovered circuits paint a clear picture of how Gemma does a given task, with one circuit achieving 95% faithfulness with <20 total latents. This minimality lets us quickly understand the [...]

---

Outline:

(00:35) TL;DR:

(01:35) 1 Introduction

(04:30) 2 Background

(04:33) 2.1 SAEs

(05:19) 2.2 Circuits

(07:04) 2.3 Problems with Current Sparse Feature Interpretability Approaches

(07:10) 2.3.1 Scalability

(07:44) 2.3.2 Independent Scoring of Nodes

(08:25) 2.3.3 Error Nodes

(10:01) 3 Our Approach

(10:29) 3.1 Solving Scalability: Circuits with few residual SAEs

(11:21) 3.2 Solving Independent Scoring: Masking

(12:08) 3.3 Error nodes

(12:25) 4 Results

(12:29) 4.1 Setup

(13:49) 4.2 Performance Recovery

(15:00) 4.2.1 Code Output Prediction:

(16:48) 4.2.2 Subject Verb Agreement (SVA):

(17:42) 4.2.3 IOI:

(18:55) 4.3 Completeness

(22:22) 4.4 Mask Stability

(23:07) 5 Case Study: Code Output Prediction

(25:09) 6. Conclusions

(27:45) 7. Future Research and Ideas

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

MATS currently has more people interested in being mentors than we are able to support—for example, for the Winter 2024-25 Program, we received applications from 87 prospective mentors who cumulatively asked for 223 scholars[1] (for a cohort where we expected to only accept 80 scholars). As a result, we need some process for how to choose which researchers to take on as mentors and how many scholars to allocate each. Our desiderata for the process are as follows:

• We want to base our decisions on expert opinions of the quality of various research directions.
• We want the above opinions to be sourced from a range of perspectives with the AI existential safety field, to incorporate multiple perspectives on alignment research and other research areas we think are important, such as AI governance and policy, AI security, and other approaches for addressing AI catastrophic risk.
• We want [...]

---

Outline:

(00:04) Introduction

(02:17) Mentor selection

(08:04) Mentor demographics

(10:55) Scholar demographics

(13:57) Acknowledgements

The original text contained 3 footnotes which were omitted from this narration.

The original text contained 11 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Response to Discursive Games, Discursive Warfare

The discursive distortions you discuss serve two functions:

1 Narratives can only serve as effective group identifiers by containing fixed elements that deviate from what naive reason would think. In other words, something about the shared story has to be a costly signal of loyalty, and therefore a sign of a distorted map. An undistorted map would be advantageous for anyone regardless of group membership; a distorted map is advantageous only for people using it as an identifying trait. Commercial mapmakers will sometimes include phantom towns so that they (and courts) can distinguish competitors who plagiarized their work from competitors who independently mapped the same terrain. Point deer make horse can catalyze the formation of a faction because it reduces motive ambiguity in a way that "point deer make deer" could not.

"Not Invented Here" dynamics are part of this. To occupy territory [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: Making claims or demands about/into other people's internal states, rather than about your state or observable external states, predictably ties people in knots—instead: only make claims about your own experience or observables. This lets the other control the copy of them that's in the shared context.[1]

Non-Violent Communication rapidly grew into a world-spanning movement due to its remarkable ability to avert and dissolve conflict. The NVC book has a whole bunch of examples of communication procedures and practical tips. Here I'll try and convey the core insights more rapidly by focusing on the underlying principles I've extracted.[2]

Background models

People have self-models. People have models of other people. Conversations form shared context, which syncs with both of those. Some ways of conversing work better than others.

Syncing back an incorrect model from conversational context can disrupt or harm one's self-model, so dissonance between your self-model and others' models [...]

---

Outline:

(00:45) Background models

(01:13) Common collision in conversational context (Cartoons start here )

(02:32) Just talking about yourself is good, actually

(03:06) 2x2 (Purpose x Domain)

(04:09) Observables (Cartoons #2)

(05:15) Summary (slightly spaced repetition)

(06:05) Appendix: Speculation on sometimes spectacular successes

The original text contained 16 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.