From AI scientist to AI research fleet

Research automation is here (1, 2, 3). We saw it coming and planned ahead, which puts us ahead of most (4, 5, 6). But that foresight also comes with a set of outdated expectations that are holding us back. In particular, research automation is not just about “aligning the first AI scientist”, it's also about the institution-building problem of coordinating the first AI research fleets.

Research automation is not about developing a plug-and-play “AI scientist”. Transformative technologies are rarely straightforward substitutes for what came before. The industrial revolution was not about creating mechanical craftsmen but about deconstructing craftsmen into assembly lines of specialized, repeatable tasks. Algorithmic trading was not just about creating faster digital traders but about reimagining traders as fleets of bots, quants, engineers, and other specialists. AI-augmented science will not just be about creating AI “scientists.”

Why? New technologies come [...]

---

Outline:

(00:04) From AI scientist to AI research fleet

(05:05) Recommendations

(05:28) Individual practices

(06:22) Organizational changes

(07:27) Community-level actions

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I'm behind on a couple of posts I've been planning, but am trying to post something every day if possible. So today I'll post a cached fun piece on overinterpreting a random data phenomenon that's tricked me before.

Recall that a random walk or a "drunkard's walk" (as in the title) is a sequence of vectors _x_1, x_2, ldots_ in some _mathbb{R}^n_ such that each _x_k_ is obtained from _x_{k-1}_ by adding noise. Here is a picture of a 1D random walk as a function of time:

A random walk is the "null hypothesis" for any data with memory. If you are looking at some learning process that updates state to state with some degree of stochasticity, seeing a random walk means that your update steps are random and you're not in fact learning. If you graph some collection of activations from layer to layer of a transformer [...]

The original text contained 4 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We have contact details and can send emails to 1500 students and former students who've received hard-cover copies of HPMOR (and possibly Human Compatible and/or The Precipice) because they've won international or Russian olympiads in maths, computer science, physics, biology, or chemistry.

This includes over 60 IMO and IOI medalists.

This is a pool of potentially extremely talented people, many of whom have read HPMOR.

I don't have the time to do anything with them, and people in the Russian-speaking EA community are all busy with other things.

The only thing that ever happened was an email sent to some kids still in high school about the Atlas Fellowship, and a couple of them became fellows.

I think it could be very valuable to alignment-pill these people. I think for most researchers who understand AI x-risk well enough and can speak Russian, even manually going through IMO and IOI [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a plan for how ASI could be relatively safely developed.

Abstract: A plan that puts all frontier model companies on a unified schedule of model training, evaluation and approval, with regulatory compliance promoted through market access. This aims to combine (most of) the economic benefits of unrestricted competition but with more safety, (most of) the time-to-think benefits of AI pauses but with better compliance incentives, and (most of) the central oversight of a Manhattan project but with more freedom and pluralism.

Background

It is based on the following worldview, though not all are cruxes:

1. Rushing to ASI by default leads to deceptive and misaligned ASI, with catastrophic consequences for humanity.
2. A lot of alignment work will be empirical and requires access to (near) cutting edge models to work with.
3. A lot of progress is driven by increased compute and it is possible to measure compute in [...]

---

Outline:

(00:39) Background

(02:23) The Plan

(04:57) Advantages

(07:18) Potential problems

(08:11) Do the frontier training runs have to be simultaneous?

(08:57) Quis custodiet ipsos custodes?

(10:19) How does this work internationally?

(12:25) To be determined

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

-I can be a nightmare conference attendee: I tend to ask nitpicky questions and apply a dose of skepticism to a speaker's claims which is healthy in doing one's own research, but probably not optimal when everyone else is trying to follow a talk. I'm working on being better at this, but for now I blame my background.

There is one nitpick that comes up again and again. In fact in one conference I brought it up so often that Jake Mendel coined a term for it: "Dmitry's koan".

In koan form, the nitpick is as follows:

There is no such thing as interpreting a neural network. There is only interpreting a neural network at a given scale of precision.

On its face, this observation is true but a bit banal. Indeed there are two extremes:

1. At the "less precise" extreme, you can claim you have interpreted [...]

---

Outline:

(04:13) Elucidating the spectrum of precision

(04:17) Step 1: coming to terms with imprecision

(10:57) Step 2: Factoring in the memorization-generalization spectrum

(18:31) Natural scale and natural degradation

(18:56) Sometimes reconstruction loss is not the point

(20:10) Degradation as a dial

(27:49) Natural scale

(30:31) Natural degradation

(35:08) Possible issues

(36:08) Experiment suggestions

The original text contained 12 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

There's one week left in the Discussion Phase of the 2023 Review. Here's a few things you might want to spend that time doing, in escalating order of effort:

Review three posts, without stressing much about the details. Go the Quick Review[1] page, pick some posts you remember and had opinions about. Reread them, and write a review.

Think about which posts should be in the Best of LessWrong. Go to the Advanced Review[2] page, look at which posts are currently in the top 50, and at the posts that aren't in the top 50 which you think should be. Write reviews arguing for posts you think are underappreciated.

Think big picture, longform thoughts about the conversations from 2023. After reviewing several posts, write a top level post with the 2023 Longform Review tag, exploring multiple posts from 2023 and how they tie together with today's thinking.

The Basic [...]

---

Outline:

(00:59) The Basic Ask: Reviews worth highlighting

(02:05) Specific prompts for reviewing:

(02:34) Prompts for Self-Reviews

(03:35) What should be in the Best of LessWrong?

(04:20) Longform Reflection

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Dwarkesh Patel again interviewed Tyler Cowen, largely about AI, so here we go.

Note that I take it as a given that the entire discussion is taking place in some form of an ‘AI Fizzle’ and ‘economic normal’ world, where AI does not advance too much in capability from its current form, in meaningful senses, and we do not get superintelligence [because of reasons]. It's still massive additional progress by the standards of any other technology, but painfully slow by the ‘AGI is coming soon’ crowd.

That's the only way I can make the discussion make at least some sense, with Tyler Cowen predicting 0.5%/year additional RGDP growth from AI. That level of capabilities progress is a possible world, although the various elements stated here seem like they are sometimes from different possible worlds.

I note that this conversation was recorded prior to o3 and all [...]

---

Outline:

(02:01) AI and Economic Growth

(02:28) Cost Disease

(09:03) The Lump of Intelligence Fallacy

(10:56) The Efficient Market Hypothesis is False

(13:05) Not Sending Your Best People

(20:49) Energy as the Bottleneck

(22:31) The Experts are Wrong But Trust Them Anyway

(25:41) AI as Additional Population

(26:51) Opposition to AI as the Bottleneck

(29:10) China as Existence Proof for Rapid Growth

(29:56) Second Derivatives

(31:05) Talent and Leadership

(33:26) Adapting to the Age of AI

(35:37) Identifying Alpha

(37:39) Old Man Yells at Crowd

(41:59) Some Statements for Everyone to Ponder

(43:13) No Royal Road to Wisdom

(47:29) Concluding Thoughts

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Anthropic's Alignment Science team conducts technical research aimed at mitigating the risk of catastrophes caused by future advanced AI systems, such as mass loss of life or permanent loss of human control. A central challenge we face is identifying concrete technical work that can be done today to prevent these risks. Future worlds where our research matters—that is, worlds that carry substantial catastrophic risk from AI—will have been radically transformed by AI development. Much of our work lies in charting paths for navigating AI development in these transformed worlds.

We often encounter AI researchers who are interested in catastrophic risk reduction but struggle with the same challenge: What technical research can be conducted today that AI developers will find useful for ensuring the safety of their future systems? In this blog post we share some of our thoughts on this question.

To create this post, we asked Alignment Science [...]

---

Outline:

(02:36) Evaluating capabilities

(04:16) Evaluating alignment

(06:03) Understanding model cognition

(09:01) Understanding how a models persona affects its behavior and how it generalizes out-of-distribution

(10:25) Chain-of-thought faithfulness

(12:00) AI control

(13:12) Behavioral monitoring

(15:11) Activation monitoring

(17:01) Anomaly detection

(18:42) Scalable oversight

(20:14) Improving oversight despite systematic, exploitable errors in the oversight signal

(22:24) Recursive oversight

(23:50) Weak-to-strong and easy-to-hard generalization

(27:16) Honesty

(28:45) Adversarial robustness

(29:54) Realistic and differential benchmarks for jailbreaks

(32:15) Adaptive defenses

(33:38) Miscellaneous

(33:49) Unlearning dangerous information and capabilities

(34:53) Learned governance for multi-agent alignment

(36:36) Acknowledgements

---

First published:

Source:

Narrated by TYPE III AUDIO.

Epistemic status -- sharing rough notes on an important topic because I don't think I'll have a chance to clean them up soon.

Summary

Suppose a human used AI to take over the world. Would this be worse than AI taking over? I think plausibly:

• In expectation, human-level AI will better live up to human moral standards than a randomly selected human. Because:
  • Humans fall far short of our moral standards.
  • Current models are much more nice, patient, honest and selfless than humans.
    • Though human-level AI will have much more agentic training for economic output, and a smaller fraction of HHH training, which could make them less nice.
  • Humans are "rewarded" for immoral behaviour more than AIs will be
    • Humans evolved under conditions where selfishness and cruelty often paid high dividends, so evolution often "rewarded" such behaviour. And similarly, during lifetime learning we [...]

---

Outline:

(00:12) Summary

(02:20) AGI is nicer than humans in expectation

(04:53) Conditioning on AI actually seizing power

(08:18) Conditioning on the human actually seizing power

(09:31) Other considerations

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[This is an interim report and continuation of the work from the research sprint done in MATS winter 7 (Neel Nanda's Training Phase)]

Try out binary masking for a few residual saes in this colab notebook: [Github Notebook] [Colab Notebook]

TL;DR: 

We propose a novel approach to:

1. Scaling SAE Circuits to Large Models: By placing sparse autoencoders only in the residual stream at intervals, we find circuits in models as large as Gemma 9B without requiring SAEs to be trained for every transformer layer.
2. Finding Circuits: We develop a better circuit finding algorithm. Our method optimizes a binary mask over SAE latents, which proves significantly more effective than existing thresholding-based methods like Attribution Patching or Integrated Gradients.

Our discovered circuits paint a clear picture of how Gemma does a given task, with one circuit achieving 95% faithfulness with <20 total latents. This minimality lets us quickly understand the [...]

---

Outline:

(00:35) TL;DR:

(01:35) 1 Introduction

(04:30) 2 Background

(04:33) 2.1 SAEs

(05:19) 2.2 Circuits

(07:04) 2.3 Problems with Current Sparse Feature Interpretability Approaches

(07:10) 2.3.1 Scalability

(07:44) 2.3.2 Independent Scoring of Nodes

(08:25) 2.3.3 Error Nodes

(10:01) 3 Our Approach

(10:29) 3.1 Solving Scalability: Circuits with few residual SAEs

(11:21) 3.2 Solving Independent Scoring: Masking

(12:08) 3.3 Error nodes

(12:25) 4 Results

(12:29) 4.1 Setup

(13:49) 4.2 Performance Recovery

(15:00) 4.2.1 Code Output Prediction:

(16:48) 4.2.2 Subject Verb Agreement (SVA):

(17:42) 4.2.3 IOI:

(18:55) 4.3 Completeness

(22:22) 4.4 Mask Stability

(23:07) 5 Case Study: Code Output Prediction

(25:09) 6. Conclusions

(27:45) 7. Future Research and Ideas

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.