Every now and then, some AI luminaries

• (1) propose that the future of powerful AI will be reinforcement learning agents—an algorithm class that in many ways has more in common with MuZero (2019) than with LLMs; and
• (2) propose that the technical problem of making these powerful future AIs follow human commands and/or care about human welfare—as opposed to, y’know, the Terminator thing—is a straightforward problem that they already know how to solve, at least in broad outline.

I agree with (1) and strenuously disagree with (2).

The last time I saw something like this, I responded by writing: LeCun's “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem.

Well, now we have a second entry in the series, with the new preprint book chapter “Welcome to the Era of Experience” by reinforcement learning pioneers David Silver & Richard Sutton.

The authors propose that “a new generation [...]

---

Outline:

(04:39) 1. What's their alignment plan?

(08:00) 2. The plan won't work

(08:04) 2.1 Background 1: Specification gaming and goal misgeneralization

(12:19) 2.2 Background 2: The usual agent debugging loop, and why it will eventually catastrophically fail

(15:12) 2.3 Background 3: Callous indifference and deception as the strong-default, natural way that era of experience AIs will interact with humans

(16:00) 2.3.1 Misleading intuitions from everyday life

(19:15) 2.3.2 Misleading intuitions from today's LLMs

(21:51) 2.3.3 Summary

(24:01) 2.4 Back to the proposal

(24:12) 2.4.1 Warm-up: The specification gaming game

(29:07) 2.4.2 What about bi-level optimization?

(31:13) 2.5 Is this a solvable problem?

(35:42) 3. Epilogue: The bigger picture--this is deeply troubling, not just a technical error

(35:51) 3.1 More on Richard Sutton

(40:52) 3.2 More on David Silver

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’ve read at least a few hundred blog posts, maybe upwards of a thousand. Agreeing with Gavin Leech, I believe I’ve gained from essays more than any other medium. I’m an intellectually curious college student with a strong interest in both the practical and philosophical. This blog post provides a snippet into the most valuable blog posts I’ve read on the topic of productivity. I encourage you to scan what I’ve gotten valuable out of the blog posts and read deeper if interesting.

• On systems - Living a life of zero willpower by Neel Nanda. This post inspired me to sign up for Focusmate. I can confidently say that Focusmate (1) has increased my productivity by >20% in a lasting way and (2) houses more than half of my social life. I’ve met many of my closest friends through the app. The other advice [...]

---

First published:

Source:

Linkpost URL:
https://parconley.com/my-favorite-productivity-blog-posts/

Narrated by TYPE III AUDIO.

Converting to a for-profit model would undermine the company's founding mission to ensure AGI "benefits all of humanity," argues new letter

This is the full text of a post from Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

Don’t become a for-profit.

That's the blunt message of a recent letter signed by more than 30 people, including former OpenAI employees, prominent civil-society leaders, legal scholars, and Nobel laureates, including AI pioneer Geoffrey Hinton and former World Bank chief economist Joseph Stiglitz.

Obsolete obtained the 25-page letter, which was sent last Thursday to the attorneys general (AGs) of California and Delaware, two officials with the power to block the deal.

Made [...]

---

Outline:

(00:13) Converting to a for-profit model would undermine the companys founding mission to ensure AGI benefits all of humanity, argues new letter

(02:10) Nonprofit origins

(05:16) Nobel opposition

(06:32) Contradictions

(09:18) Justifications

(12:22) No sale price can compensate

(13:55) An institutional test

(15:42) Appendix: Quotes from OpenAI's leaders over the years

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I love o3. I’m using it for most of my queries now.

But that damn model is a lying liar. Who lies.

This post covers that fact, and some related questions.

o3 Is a Lying Liar

The biggest thing to love about o3 is it just does things. You don’t need complex or multi-step prompting, ask and it will attempt to do things.

Ethan Mollick: o3 is far more agentic than people realize. Worth playing with a lot more than a typical new model. You can get remarkably complex work out of a single prompt.

It just does things. (Of course, that makes checking its work even harder, especially for non-experts.)

Teleprompt AI: Completely agree. o3 feels less like prompting and more like delegating. The upside is wild- but yeah, when it just does things, tracing the logic (or spotting hallucinations) becomes [...]

---

Outline:

(00:33) o3 Is a Lying Liar

(04:53) All This Implausible Lying Has Implications

(06:50) Misalignment By Default

(10:27) Is It Fixable?

(15:06) Just Don't Lie To Me

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

tl;dr: Even if we can't solve alignment, we can solve the problem of catching and fixing misalignment.

If a child is bowling for the first time, and they just aim at the pins and throw, they’re almost certain to miss. Their ball will fall into one of the gutters. But if there were beginners’ bumpers in place blocking much of the length of those gutters, their throw would be almost certain to hit at least a few pins. This essay describes an alignment strategy for early AGI systems I call ‘putting up bumpers’, in which we treat it as a top priority to implement and test safeguards that allow us to course-correct if we turn out to have built or deployed a misaligned model, in the same way that bowling bumpers allow a poorly aimed ball to reach its target.

To do this, we'd aim to build [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

to follow up my philantropic pledge from 2020, i've updated my philanthropy page with the 2024 results.

in 2024 my donations funded $51M worth of endpoint grants (plus $2.0M in admin overhead and philanthropic software development). this comfortably exceeded my 2024 commitment of $42M (20k times $2100.00 — the minimum price of ETH in 2024).

this also concludes my 5-year donation pledge, but of course my philanthropy continues: eg, i’ve already made over $4M in endpoint grants in the first quarter of 2025 (not including 2024 grants that were slow to disburse), as well as pledged at least $10M to the 2025 SFF grant round.

---

First published:

Source:

Linkpost URL:
https://jaan.info/philanthropy/#2024-results

Narrated by TYPE III AUDIO.

Guillaume Blanc has a piece in Works in Progress (I assume based on his paper) about how France's fertility declined earlier than in other European countries, and how its power waned as its relative population declined starting in the 18th century. In 1700, France had 20% of Europe's population (4% of the whole world population). Kissinger writes in Diplomacy with respect to the Versailles Peace Conference:

Victory brought home to France the stark realization that revanche had cost it too dearly, and that it had been living off capital for nearly a century. France alone knew just how weak it had become in comparison with Germany, though nobody else, especially not America, was prepared to believe it ...

Though France's allies insisted that its fears were exaggerated, French leaders knew better. In 1880, the French had represented 15.7 percent of Europe's population. By 1900, that [...]

---

First published:

Source:

Linkpost URL:
https://arjunpanickssery.substack.com/p/to-understand-history-keep-former

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The European AI Office is currently writing the rules for how general-purpose AI (GPAI) models will be governed under the EU AI Act.

The are explicitly asking for feedback on how to interpret and operationalize key obligations under the AI Act.

This includes the thresholds for systemic risk, the definition of GPAI, how to estimate training compute, and when downstream fine-tuners become legally responsible.

Why this matters for AI Safety:

The largest labs (OpenAI, Anthropic, Google DeepMind) have already expressed willigness to sign on to the Codes of Practice voluntarily.

These codes will become the de facto compliance baseline, and potentially a global reference point.

So far, AI safety perspectives are severely underrepresented.

Input is urgently needed to ensure the guidelines reflect concerns around misalignment, loss of control, emergent capabilities, robust model evaluation, and the need for interpretability audits.

Key intervention points include [...]

---

Outline:

(00:43) Why this matters for AI Safety:

(02:05) Purpose of this post:

(03:04) TL;DR

(03:08) What the GPAI Codes of Practice will actually Regulate

(04:49) What AI safety researchers should weigh in on:

(05:39) 1. Content of the Guidelines

(06:22) 2. What counts as a General-Purpose AI Model ?

(07:13) 2.1 Conditions for Sufficient Generality and Capabilities

(09:23) 2.2 Differentiation Between Distinct Models and Model Versions

(10:30) Why this matters for AI Safety:

(11:35) 3. What counts as a Provider of a General-Purpose AI Model ?

(12:38) 3.1 What Triggers Provider Status?

(13:13) Downstream Modifiers as Providers

(14:08) 3.2 GPAI with Systemic Risk: Stricter Thresholds

(15:14) 4. Exemptions for Open-Source models

(16:24) 5. Estimating Compute: The First Scalable Safety Trigger

(17:22) 5.1 How to Estimate Compute

(18:13) 5.2 What Counts Toward Cumulative Compute

(19:16) 6. Other Legal & Enforcement details

---

First published:

Source:

Narrated by TYPE III AUDIO.

Joel Z. Leibo [1], Alexander Sasha Vezhnevets [1], William A. Cunningham [1, 2], Sébastien Krier [1], Manfred Diaz [3], Simon Osindero [1]

[1] Google DeepMind, [2] University of Toronto, [3] Mila Québec AI Institute

Disclaimer: These are our own opinions; they do not represent the views of Google DeepMind as a whole or its broader community of safety researchers.

Beyond Alignment: The Patchwork Quilt of Human Coexistence

"We pragmatists think of moral progress as more like sewing together a very large, elaborate, polychrome quilt, than like getting a clearer vision of something true and deep." – Richard Rorty (2021) pg. 141

Quite a lot of thinking in AI alignment, particularly within the rationalist tradition, implicitly or explicitly appears to rest upon something we might call an 'Axiom of Rational Convergence'. This is the powerful idea that under sufficiently ideal epistemic conditions – ample time, information, reasoning ability, freedom [...]

---

Outline:

(00:47) Beyond Alignment: The Patchwork Quilt of Human Coexistence

(14:41) The Allure of the Clearer Vision: Alignment as Mistake Theory

(21:29) Potential Blind Spots in the Pursuit of Universal Alignment

(28:32) The Stitches That Bind: Conventions, Sanctions, and Norms

(33:50) Navigating Context and Building a Pluralistic AI Ecosystem

(41:34) Collective Flourishing: The Ever-Growing Quilt

(43:33) The Astronomer and the Tailor: Seeing Clearly vs. Sewing Well

(48:29) References

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Outline:

(01:41) You Better Mechanize

(10:32) Superintelligence Eventually

(11:38) Please Review This Podcast

(11:55) They Won't Take Our Jobs Yet

(13:04) They Took Our (Travel Agent) Jobs

(17:54) The Case Against Intelligence

(25:27) Intelligence Explosion

(27:23) Explosive Economic Growth

(30:25) Wowie on Alignment and the Future

(33:52) But That's Good Actually

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.