November 22, 2025

Abstract advice to researchers tackling the difficult core problems of AGI alignment

Listen Later

15 minutes

Crosspost from my blog.

This some quickly-written, better-than-nothing advice for people who want to make progress on the hard problems of technical AGI alignment.

Background assumptions

The following advice will assume that you're aiming to help solve the core, important technical problem of desigining AGI that does stuff humans would want it to do.

This excludes everything that isn't about minds and designing minds and so on; so, excluding governance, recruiting, anything social, fieldbuilding, fundraising, whatever. (Not saying those are unimportant; just, this guide is not about that.)

I don't especially think you should try to do that. It's very hard, and it's more important that AGI capabilities research gets stopped. I think it's so hard that human intelligence amplification is a better investment.

However, many people say that they want to help with technical AI safety. If you're mainly looking to get a job, this is not the guide for you. This guide is only aimed at helping you help solve the important parts of the problem, which is a very very neglected task among people who say they want to help with technical AI safety generally.

[...]

---

Outline:

(00:21) Background assumptions

(02:29) Dealing with deference

(04:44) Sacrifices

(06:28) True doubt

(07:27) Iterative babble and prune

(08:43) Learning to think

(09:22) Grappling with the size of minds

(10:05) Zooming

(11:05) Generalize a lot

(12:51) Notes to mentors

(13:59) Object level stuff

---

First published:

November 22nd, 2025

Source:

https://www.lesswrong.com/posts/rZQjk7T6dNqD5HKMg/abstract-advice-to-researchers-tackling-the-difficult-core

---

Narrated by TYPE III AUDIO.

...more

View all episodes

View all episodes

Download on the App Store

Download on the App Store

Get it on Google Play

LessWrong (30+ Karma)

By LessWrong

November 22, 2025

Abstract advice to researchers tackling the difficult core problems of AGI alignment

Listen Later

15 minutes

Crosspost from my blog.

This some quickly-written, better-than-nothing advice for people who want to make progress on the hard problems of technical AGI alignment.

Background assumptions

The following advice will assume that you're aiming to help solve the core, important technical problem of desigining AGI that does stuff humans would want it to do.

This excludes everything that isn't about minds and designing minds and so on; so, excluding governance, recruiting, anything social, fieldbuilding, fundraising, whatever. (Not saying those are unimportant; just, this guide is not about that.)

I don't especially think you should try to do that. It's very hard, and it's more important that AGI capabilities research gets stopped. I think it's so hard that human intelligence amplification is a better investment.

However, many people say that they want to help with technical AI safety. If you're mainly looking to get a job, this is not the guide for you. This guide is only aimed at helping you help solve the important parts of the problem, which is a very very neglected task among people who say they want to help with technical AI safety generally.

[...]

---

Outline:

(00:21) Background assumptions

(02:29) Dealing with deference

(04:44) Sacrifices

(06:28) True doubt

(07:27) Iterative babble and prune

(08:43) Learning to think

(09:22) Grappling with the size of minds

(10:05) Zooming

(11:05) Generalize a lot

(12:51) Notes to mentors

(13:59) Object level stuff

---

First published:

November 22nd, 2025

Source:

https://www.lesswrong.com/posts/rZQjk7T6dNqD5HKMg/abstract-advice-to-researchers-tackling-the-difficult-core

---

Narrated by TYPE III AUDIO.

...more

More shows like LessWrong (30+ Karma)

Making Sense with Sam Harris by Sam Harris

Making Sense with Sam Harris

26,332 Listeners

Conversations with Tyler by Mercatus Center at George Mason University

Conversations with Tyler

2,452 Listeners

The Peter Attia Drive by Peter Attia, MD

The Peter Attia Drive

8,579 Listeners

Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas by Sean Carroll | Wondery

Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas

4,183 Listeners

ManifoldOne by Steve Hsu

ManifoldOne

93 Listeners

Your Undivided Attention by The Center for Humane Technology, Tristan Harris, Daniel Barcay and Aza Raskin

Your Undivided Attention

1,598 Listeners

All-In with Chamath, Jason, Sacks & Friedberg by All-In Podcast, LLC

All-In with Chamath, Jason, Sacks & Friedberg

9,932 Listeners

Machine Learning Street Talk (MLST) by Machine Learning Street Talk (MLST)

Machine Learning Street Talk (MLST)

95 Listeners

Dwarkesh Podcast by Dwarkesh Patel

Dwarkesh Podcast

501 Listeners

Hard Fork by The New York Times

Hard Fork

5,518 Listeners

The Ezra Klein Show by New York Times Opinion

The Ezra Klein Show

15,938 Listeners

Moonshots with Peter Diamandis by PHD Ventures

Moonshots with Peter Diamandis

546 Listeners

No Priors: Artificial Intelligence | Technology | Startups by Conviction

No Priors: Artificial Intelligence | Technology | Startups

131 Listeners

Latent Space: The AI Engineer Podcast by swyx + Alessio

Latent Space: The AI Engineer Podcast

93 Listeners

BG2Pod with Brad Gerstner and Bill Gurley by BG2Pod

BG2Pod with Brad Gerstner and Bill Gurley

467 Listeners