Epistemic status: I think something like this confusion is happening often. I'm not saying these are the only differences in what people mean by "AGI alignment".

Summary: 

Value alignment is better but probably harder to achieve than personal intent alignment to the short-term wants of some person(s). Different groups and people tend to primarily address one of these alignment targets when they discuss alignment. Confusion abounds.

One important confusion is an assumption that the type of AI defines the alignment target: strong goal-directed AGI must be value aligned or misaligned, while personal intent alignment is only viable for relatively weak AI. I think this assumption is important but false.

While value alignment is categorically better, intent alignment seems easier, safer, and more appealing in the short term, so AGI project leaders are likely to try it.[1]

Overview

Clarifying what people mean by alignment should dispel some illusory [...]

---

Outline:

(00:18) Summary:

(01:04) Overview

(03:57) Another unstated divide

(06:14) Personal intent alignment for full ASI: can I have your goals?

(07:24) Conclusions

The original text contained 6 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This part 2 in a 3-part sequence summarizes my book (see part 1 here), The Darwinian Trap. The book aims to popularize the concept of multipolar traps and establish them as a broader cause area. If you find this series intriguing and want to spread the word and learn more:

1. Share this post with others on X or other social media platforms. 
2. Pre-order the book here.
3. Sign up for my mailing list here before September 24 for a 20% chance to win a free hardcover copy of the book (it takes 5 seconds). 
4. Contact me at [email protected] if you have any input or ideas. 

In Part 1, I introduced the concept of a Darwinian demon—selection pressures that drive agents to harm others for personal gain. I also argued that the game theory of our evolutionary fitness landscape, with its limited resources, often favors defection over cooperation within populations. [...]

---

Outline:

(02:55) Major Evolutionary Transitions and Cooperation

(06:09) The Fragility of Life Hypothesis

(11:09) Exhibit A: Kamikaze mutants

(14:17) Exhibit C: Complex life in the universe is likely extremely rare

(16:20) Exhibit D: Life is constantly on the precipice

(18:06) Only Humanity Can Save Life from Itself

The original text contained 5 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Emmett Shear on twitter asked:

I think SB 1047 has gotten much better from where it started. It no longer appears actively bad. But can someone who is pro-SB 1047 explain the specific chain of causal events where they think this bill becoming law results in an actual safer world? What's the theory?

And I realized that AFAICT no one has concisely written up what the actual story for SB 1047 is supposed to be.

This is my current understanding. Other folk here

The bill isn't sufficient on it's own, but it's not regulation for regulation's sake because it's specifically a piece of the regulatory machine I'd ultimately want built.

Right now, it mostly solidifies the safety processes that existing orgs have voluntarily committed to. But, we are pretty lucky that they voluntarily committed to them, and we don't have any guarantee that they'll stick with them in [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

(this is an expanded, edited version of an x.com post)

It is easy to interpret Eliezer Yudkowsky's main goal as creating a friendly AGI. Clearly, he has failed at this goal and has little hope of achieving it. That's not a particularly interesting analysis, however. A priori, creating a machine that makes things ok forever is not a particularly plausible objective. Failure to do so is not particularly informative.

So I'll focus on a different but related project of his: executable philosophy. Quoting Arbital:

Two motivations of "executable philosophy" are as follows:

---

First published:

Source:

Narrated by TYPE III AUDIO.

Michael Dickens has read the research and performed two self-experiments on whether consuming caffeine builds up tolerance, and if yes, how quickly.

First literature review:

What if instead of taking caffeine every day, you only take it intermittently—say, once every 3 days? How often can most people take caffeine without developing a tolerance?

The scientific literature on this question is sparse. Here's what I found:

Second literature review:

If you [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Recently, Nathan Young and I wrote about arguments for AI risk and put them on the AI Impacts wiki. In the process, we ran a casual little survey of the American public regarding how they feel about the arguments, initially (if I recall) just because we were curious whether the arguments we found least compelling would also fail to compel a wide variety of people.

The results were very confusing, so we ended up thinking more about this than initially intended and running four iterations total. This is still a small and scrappy poll to satisfy our own understanding, and doesn’t involve careful analysis or error checking. But I’d like to share a few interesting things we found. Perhaps someone else wants to look at our data more carefully, or run more careful surveys about parts of it.

In total we surveyed around 570 people across 4 [...]

The original text contained 5 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Technological Richter scale is introduced about 80% of the way through Nate Silver's new book On the Edge.

A full review is in the works (note to prediction markets: this post alone does NOT on its own count as a review, but this counts as part of a future review), but this concept seems highly useful, stands on its own and I want a reference post for it. Nate skips around his chapter titles and timelines, so why not do the same here?

Defining the Scale

Nate Silver, On the Edge (location 8,088 on Kindle): The Richter scale was created by the physicist Charles Richter in 1935 to quantify the amount of energy released by earthquakes.

It has two key features that I’ll borrow for my Technological Richter Scale (TRS). First, it is logarithmic. A magnitude 7 earthquake is actually ten times more powerful [...]

---

Outline:

(00:32) Defining the Scale

(07:15) The Big Disagreement About Future Generative AI

(09:39) Just Think of the Potential

(11:06) A Perfect 10

(13:19) Some Arguments Against Transformational AI

(19:06) Brief Notes on Arguments Transformational AI Will Turn Out Fine

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Would a universal basic income (UBI) work? What would it do?

Many people agree July's RCT on giving people a guaranteed income, and its paper from Eva Vivalt, Elizabeth Rhodes, Alexander W. Bartik, David E. Broockman and Sarah Miller was, despite whatever flaws it might have, the best data we have so far on the potential impact of UBI. There are many key differences from how UBI would look if applied for real, but this is the best data we have.

This study was primarily funded by Sam Altman, so whatever else he may be up to, good job there. I do note that my model of ‘Altman several years ago’ is more positive than mine of Altman now, and past actions like this are a lot of the reason I give him so much benefit of the doubt.

They do not agree on what conclusions [...]

---

Outline:

(02:47) RTFP (Read the Paper): Core Design

(06:23) Headline Effects

(10:27) Are Those Good Results?

(21:24) Expectations

(22:47) Work

(25:37) Additional Reactions

(30:35) UBI as Complement or Substitute

(32:37) On UBI in General

(34:04) The Future May Be Different

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I submitted this review to the 2024 ACX book review contest, but it didn't make the cut, so I'm putting it here instead for posterity.

Conspiracy theories are fun because of how they make everything fit together, and scratch the unbearable itch some of us get when there are little details of a narrative that just don’t make sense. The problem is they tend to have a few issues, like requiring one to posit expansive perfectly coordinated infosecurity, demanding inaccessible or running contrary to existing evidence, and generally making you look weird for believing them.

We can get our connecting-the-dots high while avoiding social stigma and epistemic demerits by instead foraging in the verdant jungle of “new conceptual frameworks for intractable debates.” Arguments about gender tend to devolve, not just for lack of a shared conceptual framework, but because the dominant frameworks used by both defenders and [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Intro

In April 2024, my colleague and I (both affiliated with Peking University) conducted a survey involving 510 students from Tsinghua University and 518 students from Peking University—China's two top academic institutions. Our focus was on their perspectives regarding the frontier risks of artificial intelligence.

In the People's Republic of China (PRC), publicly accessible survey data on AI is relatively rare, so we hope this report provides some valuable insights into how people in the PRC are thinking about AI (especially the risks). Throughout this post, I’ll do my best to weave in other data reflecting the broader Chinese sentiment toward AI.

For similar research, check out The Center for Long-Term Artificial Intelligence, YouGov, Monmouth University, The Artificial Intelligence Policy Institute, and notably, a poll conducted by Rethink Priorities, which closely informed our survey design.

You can read the full report published in the Jamestown Foundation's China Brief [...]

The original text contained 11 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.