I'm aware of good arguments that this scenario isn't inevitable, but it still seems frighteningly likely even if we solve technical alignment.

TL;DR:

1. If we solve alignment, it will probably be used to create AGI that follows human orders.
2. If takeoff is slow-ish, a pivotal act (preventing more AGIs from being developed) will be difficult.
3. If no pivotal act is performed, RSI-capable AGI proliferates. This creates an n-way non-iterated Prisoner's Dilemma where the first to attack, wins.
4. Disaster results.

The first AGIs will probably be aligned to take orders

People in charge of AGI projects like power. And by definition, they like their values somewhat better than the aggregate values of all of humanity. It also seems like there's a pretty strong argument that Instruction-following AGI is easier than value aligned AGI. In the slow-ish takeoff we expect, this alignment target seems to allow for error-correcting alignment [...]

---

Outline:

(00:43) The first AGIs will probably be aligned to take orders

(01:27) The first AGI probably wont perform a pivotal act

(02:30) So RSI-capable AGI may proliferate until a disaster occurs

(03:51) Counterarguments

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Meta: This post is a relatively rough dump of some recent research thoughts; it's not one of our more polished posts, in terms of either clarity or rigor. You’ve been warned.

The Interoperable Semantics post and the Solomonoff Inductor Walks Into A Bar post each tackled the question of how different agents in the same world can coordinate on an ontology, so that language can work at all given only a handful of example usages of each word (similar to e.g. children learning new words). Both use natural latents as a central mathematical tool - one in a Bayesian probabilistic framework, the other in a minimum description length framework. Both focus mainly on nouns, i.e. interoperable-across-minds clusters of “objects” in the environment.

… and the two propose totally different models. In one, the interoperability of cluster labels (i.e. nouns) follows from natural latent conditions over different features of each [...]

---

Outline:

(01:45) The Problem

(03:43) Two Previous Models: Naturality Over Objects vs Features

(07:34) Referents of Adjectives?

(10:49) One Implication: Adjectives Tend To Be Less Natural (Convergent) Than Nouns?

(12:38) How This Fits Into The Broader Gameplan

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

SB 1047 has been amended once more, with both strict improvements and big compromises. I cover the changes, and answer objections to the bill, in my extensive Guide to SB 1047. I follow that up here with reactions to the changes and some thoughts on where the debate goes from here. Ultimately, it is going to come down to one person: California Governor Gavin Newsom.

All of the debates we’re having matter to the extent they influence this one person. If he wants the bill to become law, it almost certainly will become law. If he does not want that, then it won’t become law, they never override a veto and if he makes that intention known then it likely wouldn’t even get to his desk. For now, he's not telling.

Table of Contents

---

Outline:

(00:52) Language Models Offer Mundane Utility

(04:39) Language Models Don’t Offer Mundane Utility

(06:49) Deepfaketown and Botpocalypse Soon

(08:00) The Art of the Jailbreak

(15:34) Get Involved

(16:33) Introducing

(17:44) In Other AI News

(20:27) Quiet Speculations

(27:51) SB 1047: Nancy Pelosi

(35:30) SB 1047: Anthropic

(39:59) SB 1047: Reactions to the Changes

(53:41) SB 1047: Big Picture

(55:45) The Week in Audio

(58:32) Rhetorical Innovation

(59:37) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:55) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 94 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

This post walks through the math for a theorem. It's intended to be a reference post, which we’ll link back to as-needed from future posts. The question which first motivated this theorem for us was: “Redness of a marker seems like maybe a natural latent over a bunch of parts of the marker, and redness of a car seems like maybe a natural latent over a bunch of parts of the car, but what makes redness of the marker ‘the same as’ redness of the car? How are they both instances of one natural thing, i.e. redness? (or ‘color’?)”. But we’re not going to explain in [...]

---

Outline:

(02:22) Trivial Version

(03:10) Nontrivial Version

(03:40) Redundancy

(05:01) Mediation

---

First published:

Source:

Narrated by TYPE III AUDIO.

Note: This post is co-authored with Stacy Li, a PhD student at Berkeley studying aging biology! Highly appreciate all her help in writing, editing, and fact-checking my understanding!

Introduction

The last time I read about aging research deeply was around 2021. The general impression I was getting was that aging research was increasingly more and more funded (good!). Unfortunately, none of the money led to actionable or useful insights (bad).

Over time, you get slightly burnt out by all the negative news.

After getting a job in biotech, I kept a hazy eye on the subject but mostly tuned out of it entirely. But, especially today, I am curious: how has the aging field progressed in the last few years? Since 2021, what has changed?

In this post, I’ll share a list of immediate questions about the state of affairs in aging research, and the [...]

---

Outline:

(00:31) Introduction

(01:30) Did the therapeutic focus on sirtuins amount to much?

(01:35) Background

(03:55) Answer

(05:48) Have the longevity-focused research institutions/grant programs/startups led to anything significant?

(05:56) Background

(07:37) Answer

(09:22) Has cellular reprogramming yielded anything useful?

(09:40) Background

(16:05) Answer

(20:24) What's the state of biological clocks?

(20:28) Background

(24:14) Answer

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Some code for this post can be found here.

Space tethers take the old, defunct space elevator concept and shorten it. Rockets can fly up to a dangling hook in the sky and then climb to a higher orbit. If the tether rotates, it can act like a catapult, providing a significant boost in a location where providing thrust is expensive. Kurzgesagt has a nice explainer and ToughSF has a great piece explaining the mechanics and some applications.

Tethers make it cheaper to explore space, but how much cheaper? Let's look at the benefits.

Tether materials and characteristic velocity

The key performance metric for the tether material is the characteristic velocity:

_V_c = sqrt frac{2 T}{rho}_

Where T is the tensile strength and rho is the density.

The stronger and lighter the material is, the faster the tether [...]

---

Outline:

(00:40) Tether materials and characteristic velocity

(01:58) Delta V and fuel savings

(06:06) Tether mass ratio and fuel costs

(07:51) Numerical examples

(12:05) Conclusion

(14:54) Appendix

(14:57) Tether materials

(18:41) Why tether material doesn’t matter too much

(21:09) Single stage to orbit is silly

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

tl;dr: We compute the evolution of the local learning coefficient (LLC), a proxy for model complexity, for an algorithmic transformer. The LLC decreases as the model learns more structured solutions, such as head specialization.

This post is structured in three main parts, (1) a summary, giving an overview of the main results, (2) the Fine Print, that delves into various cross-checks and details and (3) Discussion and Conclusions.

Structure Formation in Algorithmic Transformers

In this work we study the development of simple algorithmic transformers, which are transformers that learn to perform algorithmic tasks. A major advantage of this setup is that we can control several (hyper)parameters, such as the complexity of the training data and network architecture. This allows us to do targeted experiments studying the impacts of these parameters on the learning dynamics. The main tool we use to study the development is the Local Learning Coefficient [...]

---

Outline:

(00:37) Structure Formation in Algorithmic Transformers

(02:17) 1-Head Model

(04:01) 2-Head Model

(06:03) The Fine Print

(06:30) 2-Head Model: Re-loaded

(06:35) Removing Layer Norm

(08:05) Removing Weight Decay

(09:33) Removing both Layer Norm and Weight Decay

(11:02) Adding Noise

(12:26) Number of vocabulary regions

(13:52) Adding More Heads

(16:14) Other algorithmic transformers

(16:41) Related work

(17:21) Discussion and Conclusions

The original text contained 12 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I've recently seen several articles here that were clearly generated or heavily assisted by AI. They are all dreadful. They are verbose, they are full of "on the one hand" and "on the other", they never make any assertion without also making room for the opposite, and end with "conclusions" that say nothing. Please do not do this.

————

I wrote the above, unassisted, and it states my own views. To illustrate the point I then had GPT-4o rewrite it. Here was the prompt:

I have recently seen a substantial increase in the proportion of postings to a certain online forum that were clearly generated by AI, although not marked as such. The writing is verbose and meandering, full of "on the one hand" and "on the other", never making an assertion without also suggesting its opposite, sounding oh-so-measured and even-handed that it ends up saying nothing at all. [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

the giga press

Tesla decided to use large aluminum castings ("gigacastings") for the frame of many of its vehicles, including the Model Y and Cybertruck. This approach and the "Giga Press" used for it have been praised by many articles and youtube videos, repeatedly called revolutionary and a key advantage.

Most cars today are made by stamping steel sheets and spot welding them together with robotic arms. Here's video of a Honda factory. But that's outdated: gigacasting is the future! BYD is still welding stamped steel sheets together, and that's why it can't compete on price with Tesla. Hold on, it seems...BYD prices are actually lower than Tesla's? Much lower? Oh, and Tesla is no longer planning single unitary castings for future vehicles?

I remember reading analysis from a couple people with car manufacturing experience, concluding that unitary cast aluminum bodies could [...]

---

Outline:

(00:06) the giga press

(01:51) aluminum die casting

(03:35) aluminum vs steel

(05:25) this shouldnt be a thing

(08:29) why was this a thing

---

First published:

Source:

Narrated by TYPE III AUDIO.

We wanted to share a recap of our recent outputs with the AF community. Below, we fill in some details about what we have been working on, what motivated us to do it, and how we thought about its importance. We hope that this will help people build off things we have done and see how their work fits with ours.

Who are we?

We’re the main team at Google DeepMind working on technical approaches to existential risk from AI systems. Since our last post, we’ve evolved into the AGI Safety & Alignment team, which we think of as AGI Alignment (with subteams like mechanistic interpretability, scalable oversight, etc.), and Frontier Safety (working on the Frontier Safety Framework, including developing and running dangerous capability evaluations). We’ve also been growing since our last post: by 39% last year [...]

---

Outline:

(00:32) Who are we?

(01:32) What have we been up to?

(02:16) Frontier Safety

(02:38) FSF

(04:05) Dangerous Capability Evaluations

(05:12) Mechanistic Interpretability

(08:54) Amplified Oversight

(09:23) Theoretical Work on Debate

(10:32) Empirical Work on Debate

(11:37) Causal Alignment

(12:47) Emerging Topics

(14:57) Highlights from Our Collaborations

(17:07) What are we planning next?

---

First published:

Source:

Narrated by TYPE III AUDIO.