LessWrong (30+ Karma)

Personally, I suspect the alignment problem is hard. But even if it turns out to be easy, survival may still require getting at least the absolute basics right; currently, I think we're mostly failing even at that.

Early discussion of AI risk often focused on debating the viability of various elaborate safety schemes humanity might someday devise—designing AI systems to be more like “tools” than “agents,” for example, or as purely question-answering oracles locked within some kryptonite-style box. These debates feel a bit quaint now, as AI companies race to release agentic models they barely understand directly onto the internet.

But a far more basic failure, from my perspective, is that at present nearly all AI company staff—including those tasked with deciding whether new models are safe to build and release—are paid substantially in equity, the value of which seems likely to decline if their employers stop building and [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

“A Reader's Manifesto” is a July 2001 Atlantic piece by B.R. Myers that I've returned to many times. He complains about the inaccessible pretension of the highbrow literary fiction of his day. The article is mostly a long list of critiques of various quotes/passages from well-reviewed books by famous authors. It's hard to accuse him of cherry-picking since he only targets passages that reviewers singled out as unusually good.

Some of his complaints are dumb but the general idea is useful: authors try to be “literary” by (1) avoiding a tightly-paced plot that could evoke “genre fiction” and (2) trying to shoot for individual standout sentences that reviewers can praise, using a shotgun approach where many of the sentences are banal or just don’t make sense.

Here are some excerpts of his complaints. Bolding is always mine.

The “Writerly” Style

He complains that critics now dismiss too much good [...]

---

Outline:

(00:57) The “Writerly” Style

(03:53) 4 Types of Bad Prose

(04:10) “Evocative” Prose

(08:12) “Muscular” Prose

(11:15) “Edgy” Prose

(19:11) “Spare” Prose

(21:37) Some Prose He Thinks Is Good

---

First published:

Source:

Narrated by TYPE III AUDIO.

Diego Caples ([email protected])

Rob Neuhaus ([email protected])

Introduction

In principle, neuron activations in a transformer-based language model residual stream should be about the same scale. In practice, however the dimensions unexpectedly widely vary in scale. Mathematical theories of the transformer architecture do not predict this. They expect rotational equivariance within a model, where one dimension is no more important than any other. Is there something wrong with our reasonably informed intuitions of how transformers work? What explains these outlier channels?

Previously, Anthropic researched the existence of these privileged basis dimensions (dimensions more important / larger than expected) and ruled out several causes. By elimination, they reached the hypothesis that per-channel normalization in the Adam optimizer was the cause of privileged basis. However, they did not prove this was the case.

We conclusively show that Adam causes outlier channels / privileged basis within the transformer residual stream. When replacing the Adam [...]

---

Outline:

(00:17) Introduction

(02:06) Background

(02:09) Recommended Reading

(02:20) More About Anthropic's Work

(02:54) Adam vs SGD, and Rotational Equivariance

(03:52) Kurtosis

(04:43) TinyStories

(05:14) Experiments

(05:17) Replicating Outlier Channels at Small Scale

(06:03) Training an LM with SGD

(07:09) Conclusions

(08:37) Future Research

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

ARC has released a paper on Backdoor defense, learnability and obfuscation in which we study a formal notion of backdoors in ML models. Part of our motivation for this is an analogy between backdoors and deceptive alignment, the possibility that an AI system would intentionally behave well in training in order to give itself the opportunity to behave uncooperatively later. In our paper, we prove several theoretical results that shed some light on possible mitigations for deceptive alignment, albeit in a way that is limited by the strength of this analogy.

In this post, we will:

Thanks to [...]

---

Outline:

(01:05) Backdoors and deceptive alignment

(03:03) Static backdoor detection

(04:41) Dynamic backdoor detection

(06:51) Statistical possibility of dynamic backdoor detection

(09:02) Implications for deceptive alignment

(10:39) Computational impossibility of dynamic backdoor detection

(11:36) Implications for deceptive alignment

(12:54) Conclusion

The original text contained 8 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Note: this began life as a "Quick Takes" comment, but it got pretty long, so I figured I might as well convert it to a regular post.]

In LM training, every token provides new information about "the world beyond the LM" that can be used/"learned" in-context to better predict future tokens in the same window.

But when text is produced by autoregressive sampling from the same LM, it is not informative in the same way, at least not to the same extent[1]. Thus, sampling inevitably produces a distribution shift.

I think this is one of the reasons why it's (apparently) difficult to get instruction-tuned / HH-tuned models to report their uncertainty and level of competence accurately, rather than being overconfident.

(I doubt this is a novel point, I just haven't seen it spelled out explicitly before, and felt like doing so.)

Imagine that you read the following (as the [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Epistemic status: I think something like this confusion is happening often. I'm not saying these are the only differences in what people mean by "AGI alignment".

Summary: 

Value alignment is better but probably harder to achieve than personal intent alignment to the short-term wants of some person(s). Different groups and people tend to primarily address one of these alignment targets when they discuss alignment. Confusion abounds.

One important confusion is an assumption that the type of AI defines the alignment target: strong goal-directed AGI must be value aligned or misaligned, while personal intent alignment is only viable for relatively weak AI. I think this assumption is important but false.

While value alignment is categorically better, intent alignment seems easier, safer, and more appealing in the short term, so AGI project leaders are likely to try it.[1]

Overview

Clarifying what people mean by alignment should dispel some illusory [...]

---

Outline:

(00:18) Summary:

(01:04) Overview

(03:57) Another unstated divide

(06:14) Personal intent alignment for full ASI: can I have your goals?

(07:24) Conclusions

The original text contained 6 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This part 2 in a 3-part sequence summarizes my book (see part 1 here), The Darwinian Trap. The book aims to popularize the concept of multipolar traps and establish them as a broader cause area. If you find this series intriguing and want to spread the word and learn more:

1. Share this post with others on X or other social media platforms. 
2. Pre-order the book here.
3. Sign up for my mailing list here before September 24 for a 20% chance to win a free hardcover copy of the book (it takes 5 seconds). 
4. Contact me at [email protected] if you have any input or ideas. 

In Part 1, I introduced the concept of a Darwinian demon—selection pressures that drive agents to harm others for personal gain. I also argued that the game theory of our evolutionary fitness landscape, with its limited resources, often favors defection over cooperation within populations. [...]

---

Outline:

(02:55) Major Evolutionary Transitions and Cooperation

(06:09) The Fragility of Life Hypothesis

(11:09) Exhibit A: Kamikaze mutants

(14:17) Exhibit C: Complex life in the universe is likely extremely rare

(16:20) Exhibit D: Life is constantly on the precipice

(18:06) Only Humanity Can Save Life from Itself

The original text contained 5 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Emmett Shear on twitter asked:

I think SB 1047 has gotten much better from where it started. It no longer appears actively bad. But can someone who is pro-SB 1047 explain the specific chain of causal events where they think this bill becoming law results in an actual safer world? What's the theory?

And I realized that AFAICT no one has concisely written up what the actual story for SB 1047 is supposed to be.

This is my current understanding. Other folk here

The bill isn't sufficient on it's own, but it's not regulation for regulation's sake because it's specifically a piece of the regulatory machine I'd ultimately want built.

Right now, it mostly solidifies the safety processes that existing orgs have voluntarily committed to. But, we are pretty lucky that they voluntarily committed to them, and we don't have any guarantee that they'll stick with them in [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

(this is an expanded, edited version of an x.com post)

It is easy to interpret Eliezer Yudkowsky's main goal as creating a friendly AGI. Clearly, he has failed at this goal and has little hope of achieving it. That's not a particularly interesting analysis, however. A priori, creating a machine that makes things ok forever is not a particularly plausible objective. Failure to do so is not particularly informative.

So I'll focus on a different but related project of his: executable philosophy. Quoting Arbital:

Two motivations of "executable philosophy" are as follows:

---

First published:

Source:

Narrated by TYPE III AUDIO.

Michael Dickens has read the research and performed two self-experiments on whether consuming caffeine builds up tolerance, and if yes, how quickly.

First literature review:

What if instead of taking caffeine every day, you only take it intermittently—say, once every 3 days? How often can most people take caffeine without developing a tolerance?

The scientific literature on this question is sparse. Here's what I found:

Second literature review:

If you [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.