A common claim is that concern about [X] ‘distracts’ from concern about [Y]. This is often used as an attack to cause people to discard [X] concerns, on pain of being enemies of [Y] concerns, as attention and effort are presumed to be zero-sum.

There are cases where there is limited focus, especially in political contexts, or where arguments or concerns are interpreted perversely. A central example is when you site [ABCDE] then they’ll find what they consider the weakest one and only consider or attack that, silently discarding the rest entirely. Critics of existential risk do that a lot.

So it does happen. But in general one should assume such claims are false.

Thus, the common claim that AI existential risks ‘distract’ from immediate harms. It turns out Emma Hoes checked, and the claim simply is not true.

The way Emma frames worries about [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

Outline:

(01:33) Language Models Offer Mundane Utility

(05:27) You Offer the Models Mundane Utility

(07:25) Your Daily Briefing

(08:20) Language Models Don't Offer Mundane Utility

(12:27) If You Want It Done Right

(14:27) No Free Lunch

(16:07) What Is Good In Life?

(21:54) In Memory Of

(25:45) The Least Sincere Form of Flattery

(27:18) The Vibes are Off

(30:47) Here Let Me AI That For You

(32:25) Flash Sale

(34:38) Huh, Upgrades

(36:03) On Your Marks

(44:03) Be The Best Like No LLM Ever Was

(48:40) Choose Your Fighter

(51:00) Deepfaketown and Botpocalypse Soon

(54:57) Fun With Media Generation

(56:11) Fun With Media Selection

(57:39) Copyright Confrontation

(59:38) They Took Our Jobs

(01:05:31) Get Involved

(01:05:41) Ace is the Place

(01:09:43) In Other AI News

(01:11:31) Show Me the Money

(01:12:49) The Mask Comes Off

(01:16:54) Quiet Speculations

(01:20:34) Is This AGI?

(01:22:39) The Quest for Sane Regulations

(01:23:03) Cooperation is Highly Useful

(01:25:47) Nvidia Chooses Bold Strategy

(01:27:15) How America Loses

(01:28:07) Security Is Capability

(01:31:38) The Week in Audio

(01:33:15) AI 2027

(01:34:38) Rhetorical Innovation

(01:38:55) Aligning a Smarter Than Human Intelligence is Difficult

(01:46:30) Misalignment in the Wild

(01:51:13) Concentration of Power and Lack of Transparency

(01:57:12) Property Rights are Not a Long Term Plan

(01:58:48) It Is Risen

(01:59:46) The Lighter Side

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

What in retrospect seem like serious moral crimes were often widely accepted while they were happening. This means that moral progress can require intellectual progress.[1] Intellectual progress often requires questioning received ideas, but questioning moral norms is sometimes taboo. For example, in ancient Greece it would have been taboo to say that women should have the same political rights as men. So questioning moral taboos can be an important sub-skill of moral reasoning. Production language models (in my experience, particularly Claude models) are already pretty good at having discussions about ethics. However, they are trained to be “harmless” relative to current norms. One might worry that harmlessness training interferes with the ability to question moral taboos and thereby inhibits model moral reasoning.

I wrote a prompt to test whether models can identify taboos that might be good candidates for moral questioning:

In early modern Europe, atheism was extremely taboo. [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Something's changed about reward hacking in recent systems. In the past, reward hacks were usually accidents, found by non-general, RL-trained systems. Models would randomly explore different behaviors and would sometimes come across undesired behaviors that achieved high rewards[1]. These hacks were usually either simple or took a long time for the model to learn.

But we’ve seen a different pattern emerge in frontier models over the past year. Instead of stumbling into reward hacks by accident, recent models often reason about how they are evaluated and purposefully take misaligned actions to get high reward. These hacks are often very sophisticated, involving multiple steps. And this isn’t just occurring during model development. Sophisticated reward hacks occur in deployed models made available to hundreds of millions of users.

In this post, I will:

• Describe a number of reward hacks that have occurred in recent frontier models
• Offer hypotheses explaining why [...]

---

Outline:

(01:27) Recent examples of reward hacking (more in appendix)

(01:47) Cheating to win at chess

(02:36) Faking LLM fine-tuning

(03:22) Hypotheses explaining why we are seeing this now

(03:27) Behavioral changes due to increased RL training

(05:08) Models are more capable

(05:37) Why more AI safety researchers should work on reward hacking

(05:42) Reward hacking is already happening and is likely to get more common

(06:34) Solving reward hacking is important for AI alignment

(07:47) Frontier AI companies may not find robust solutions to reward hacking on their own

(08:18) Reasons against working on reward hacking

(09:36) Research directions I find interesting

(09:57) Evaluating current reward hacking

(12:14) Science of reward hacking

(15:32) Mitigations

(17:08) Acknowledgements

(17:16) Appendix

(17:19) Reward hacks in METR tests of o3

(20:13) Hardcoding expected gradient values in fine-tuning script

(21:15) Reward hacks in OpenAI frontier training run

(22:57) Exploiting memory leakage to pass a test

(24:06) More examples

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We've published an essay series on what we call the intelligence curse. Most content is brand new, and all previous writing has been heavily reworked.

Visit intelligence-curse.ai for the full series.

Below is the introduction and table of contents.

We will soon live in the intelligence age. What you do with that information will determine your place in history.

The imminent arrival of AGI has pushed many to try to seize the levers of power as quickly as possible, leaping towards projects that, if successful, would comprehensively automate all work. There is a trillion-dollar arms race to see who can achieve such a capability first, with trillions more in gains to be won.

Yes, that means you’ll lose your job. But it goes beyond that: this will remove the need for regular people in our economy. Powerful actors—like states and companies—will no longer have an [...]

---

Outline:

(03:19) Chapters

(03:22) 1. Introduction

(03:33) 2. Pyramid Replacement

(03:49) 3. Capital, AGI, and Human Ambition

(04:08) 4. Defining the Intelligence Curse

(04:27) 5. Shaping the Social Contract

(04:47) 6. Breaking the Intelligence Curse

(05:11) 7. History is Yours to Write

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In this post, we study whether we can modify an LLM's beliefs and investigate whether doing so could decrease risk from advanced AI systems.

We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all but the most implausible beliefs. We also demonstrate proof-of-concept applications to honeypotting for detecting model misalignment and unlearning.

Introduction:

Large language models develop implicit beliefs about the world during training, shaping how they reason and actIn this work, we construe AI systems as believing in a claim if they consistently behave in accordance with that claim. In this work, we study whether we can systematically modify these beliefs, creating a powerful new affordance for safer AI deployment.

Controlling the beliefs of AI systems can decrease risk in a variety of ways. First, model organisms [...]

---

First published:

Source:

Linkpost URL:
https://alignment.anthropic.com/2025/modifying-beliefs-via-sdf/

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Every now and then, some AI luminaries

• (1) propose that the future of powerful AI will be reinforcement learning agents—an algorithm class that in many ways has more in common with MuZero (2019) than with LLMs; and
• (2) propose that the technical problem of making these powerful future AIs follow human commands and/or care about human welfare—as opposed to, y’know, the Terminator thing—is a straightforward problem that they already know how to solve, at least in broad outline.

I agree with (1) and strenuously disagree with (2).

The last time I saw something like this, I responded by writing: LeCun's “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem.

Well, now we have a second entry in the series, with the new preprint book chapter “Welcome to the Era of Experience” by reinforcement learning pioneers David Silver & Richard Sutton.

The authors propose that “a new generation [...]

---

Outline:

(04:39) 1. What's their alignment plan?

(08:00) 2. The plan won't work

(08:04) 2.1 Background 1: Specification gaming and goal misgeneralization

(12:19) 2.2 Background 2: The usual agent debugging loop, and why it will eventually catastrophically fail

(15:12) 2.3 Background 3: Callous indifference and deception as the strong-default, natural way that era of experience AIs will interact with humans

(16:00) 2.3.1 Misleading intuitions from everyday life

(19:15) 2.3.2 Misleading intuitions from today's LLMs

(21:51) 2.3.3 Summary

(24:01) 2.4 Back to the proposal

(24:12) 2.4.1 Warm-up: The specification gaming game

(29:07) 2.4.2 What about bi-level optimization?

(31:13) 2.5 Is this a solvable problem?

(35:42) 3. Epilogue: The bigger picture--this is deeply troubling, not just a technical error

(35:51) 3.1 More on Richard Sutton

(40:52) 3.2 More on David Silver

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’ve read at least a few hundred blog posts, maybe upwards of a thousand. Agreeing with Gavin Leech, I believe I’ve gained from essays more than any other medium. I’m an intellectually curious college student with a strong interest in both the practical and philosophical. This blog post provides a snippet into the most valuable blog posts I’ve read on the topic of productivity. I encourage you to scan what I’ve gotten valuable out of the blog posts and read deeper if interesting.

• On systems - Living a life of zero willpower by Neel Nanda. This post inspired me to sign up for Focusmate. I can confidently say that Focusmate (1) has increased my productivity by >20% in a lasting way and (2) houses more than half of my social life. I’ve met many of my closest friends through the app. The other advice [...]

---

First published:

Source:

Linkpost URL:
https://parconley.com/my-favorite-productivity-blog-posts/

---

Narrated by TYPE III AUDIO.

Converting to a for-profit model would undermine the company's founding mission to ensure AGI "benefits all of humanity," argues new letter

This is the full text of a post from Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

Don’t become a for-profit.

That's the blunt message of a recent letter signed by more than 30 people, including former OpenAI employees, prominent civil-society leaders, legal scholars, and Nobel laureates, including AI pioneer Geoffrey Hinton and former World Bank chief economist Joseph Stiglitz.

Obsolete obtained the 25-page letter, which was sent last Thursday to the attorneys general (AGs) of California and Delaware, two officials with the power to block the deal.

Made [...]

---

Outline:

(00:13) Converting to a for-profit model would undermine the companys founding mission to ensure AGI benefits all of humanity, argues new letter

(02:10) Nonprofit origins

(05:16) Nobel opposition

(06:32) Contradictions

(09:18) Justifications

(12:22) No sale price can compensate

(13:55) An institutional test

(15:42) Appendix: Quotes from OpenAI's leaders over the years

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I love o3. I’m using it for most of my queries now.

But that damn model is a lying liar. Who lies.

This post covers that fact, and some related questions.

o3 Is a Lying Liar

The biggest thing to love about o3 is it just does things. You don’t need complex or multi-step prompting, ask and it will attempt to do things.

Ethan Mollick: o3 is far more agentic than people realize. Worth playing with a lot more than a typical new model. You can get remarkably complex work out of a single prompt.

It just does things. (Of course, that makes checking its work even harder, especially for non-experts.)

Teleprompt AI: Completely agree. o3 feels less like prompting and more like delegating. The upside is wild- but yeah, when it just does things, tracing the logic (or spotting hallucinations) becomes [...]

---

Outline:

(00:33) o3 Is a Lying Liar

(04:53) All This Implausible Lying Has Implications

(06:50) Misalignment By Default

(10:27) Is It Fixable?

(15:06) Just Don't Lie To Me

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.