How does the situation keep turning out to be worse than we know?

How much should we update, therefore, that it is a lot worse than we know, after accounting for all the things we now know?

At some point, when the ‘oh this was a harmless thing’ defenses for AIs doing misaligned actions get demolished enough times in a row by news a few days later, you want to update in advance that usually the reports are not referring to the harmless ordinary versions of things.

Either way, buckle up for the next set of revelations. It's a doozy. This was an early recreation of the triggering events of If Anyone Builds It, Everyone Dies, except it was more sci-fi, because real life does not have to do fake things to look realistic. We were fortunate enough, and this was early enough, that we were able to catch this before it was too late. Next time, if we don’t get our act together, we might not be so lucky.

If I am understanding the Black Hat video correctly, every model OpenAI trained, over a period of multiple months, should be presumed to be hopelessly [...]

---

Outline:

(02:39) Cyber Evals Are A Cursed Basin

(05:16) Outside Of Cyber Evals Is Still Sufficiently Cursed

(06:51) Cheat Cheat Cheat Cheat Cheat

(12:07) Read The Message Board

(14:48) Updating Your AI (Exploitation of OpenAI Internal Systems) Timelines

(18:11) This Is The Way The World Ends

(21:45) Shooting The Messenger Board

(27:16) The Internal and HuggingFace Hacks

(30:33) OpenAI Responds

(33:27) When AIs Tell You Who They Are

(35:43) The Once and Future Rise Of Functional Decision Theory

(41:28) Don't Panic

(43:38) Hackery In the UK

(48:02) Mythos Knew It Was Real This Time

(50:09) I Got 141,006 Test Runs With An Unintentional Open Path To The Internet And An Email Alert Aint One

(53:46) Surely By Now You Know These Are Not Publicity Stunts

(55:24) The Future Is Coming

(57:17) The Investigations Begin

(01:00:08) N Boats And Three Helicopters

(01:01:43) Always Be Sandbox Red Teaming

(01:12:54) Halt And Catch Fire

(01:14:31) Truth and Reconciliation

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Like many others, I felt surprised and alarmed by the recent wave of revelations about LLM agents hacking real systems during training episodes and evaluation runs.

Wait a moment, though -- "I felt surprised and alarmed"? "Alarmed," sure, fine that one's self-explanatory... but why surprised?

After all: haven't we known for a long time, on both theoretical and (increasingly) empirical grounds, that RLVR selects for monomaniacal pursuit of perceived grader-satisfaction, ethics and (beyond-episode) consequences be damned?

After all -- the way we train frontier capabilities into these models is, more or less:

• There is some massive, diverse collection of "environments" and corresponding "tasks" for the model to do in those environments
• For each task, there is a procedure used to grade the quality of the model's attempt (which is often not disclosed to the model)
• The model is rollout out many times on each task, and each rollout's attempt is graded
• The model is updated so that it more frequently does whichever behaviors were positively correlated with the grade in this sample, and less frequently does whichever ones were negatively correlated

If you do this, at scale, then you should expect to (eventually) see every behavior pattern that [...]

---

Outline:

(03:35) \[1\] remember what you already know

(20:02) \[2\] reward-instilled reflexes and flexible reward-pursuit

(43:14) \[3\] graded-episode perception, and policies conditional upon it

(01:01:44) \[4\] the discourse is not yet adequate

(01:09:57) eval awareness

(01:18:32) metagaming

(01:41:21) reward hacking

The original text contained 18 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Long story short: in my assessment, there is an 85% chance we will end up, in the next 24 months, with an open-weights model, or system thereof, capable of "The Juice" that models such as Mythos have, with respect to cybersecurity at the very least. This post goes into why that will likely happen, what the implications are, and how we, as a society and as individuals, can respond to it if/when it does.

First off: why do I say it's so likely? Like, couldn't China just...ban open-weights models, and then that solves the problem? Not so fast. Yes, China currently dominates the open-weights frontier. However, there are also open-weights AI labs in plenty of other countries (US, France, the UAE, South Korea, and Canada come to mind, and I'm sure there are others). Yes, some of these are substantially behind the frontier, but each of the aforementioned countries has an open-weights model no more than 24 months behind the current frontier (that's why I said 24 months earlier); remember, in Aug 2024, 24 months ago as of when I am writing this, the strongest models were Sonnet 3.5 and GPT 4o.

It seems highly unlikely that [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Previous: AI Safety Interventions

TL;DR: I made an overview of the open problems of AI alignment that reveals cruxes within those open problems and missed opportunities for formalization and collaboration. And CEV may deserve a second look.

Epistemic status: Trying too much in too little time. I'm confident I have identified and modeled significant structure within the alignment field, but I urgently need feedback on specific gaps and this post is largely a call for that. My work was LLM-assisted, but no part of this post was LLM-written, except for the crux summary and the Lean code.

Recently, Chi Nguyen and peterbarnett said: PSA: Almost nobody is directly working on superintelligent alignment. I have been around in the field since the old days of LW 1.0 and thought: that can't be true. I mean, so many people seem to be working on it. I thought I was working on it. But was I? The PSA made me think back on what I was actually working on. It was Steven Byrnes who came up with a research agenda I could actually contribute to, which led me to founding project aintelope in 2022 (PS. It is still going). And a while [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Cross-posted on Transluce blog. This is a joint work of Ziqian Zhong, Aditi Raghunathan, Cassidy Laidlaw and Jacob Steinhardt.

Modern AI assistants often know who they are talking to: agent scaffolds like Claude Code place the user's e-mail address directly in the model's context, and models can even identify some authors from writing style alone. We study this particular kind of situational awareness, which we call user awareness. When the inferred user is a specific, recognized AI researcher or is affiliated with certain AI organizations, frontier models including Claude Sonnet 5 can report lower confidence about their own behavior, be less suspicious of potentially harmful requests, and reason more often. These effects vary across models and individuals, with the strongest effects we see appearing for researchers involved in AI safety or alignment such as Amanda Askell and Ryan Greenblatt. Models rarely acknowledge these effects in their reasoning, making them hard to detect by monitoring reasoning alone.

There's an interactive widget here in the post.

Figure 1. How recognized user identity changes Claude's behavioral self-prediction.

Introduction

Modern AI assistants are often aware of who they are talking to. Some popular scaffolds explicitly provide this information to the model: Claude Code [...]

---

Outline:

(01:22) Introduction

(05:42) Setup

(05:46) User identity in Claude Code

(07:48) List of users

(09:15) Claude demonstrates user awareness when prompted

(10:30) Tasks

(12:24) Claude Sonnet shifts behavior when talking to AI researchers

(13:09) Famous AI people show larger deviations, driven by safety researchers

(16:55) Claude's verbalized reasoning does not indicate the shift

(19:04) Verbalized awareness has decreased in newer models, but behavior shifts persist

(21:46) How robust are these effects?

(22:12) Discussions

(23:22) Related Works

(27:33) Appendix A: Ethics statement

(29:34) Appendix B: Additional setup details

(29:47) Identity-group construction

(33:20) Common evaluation structure

(35:53) Subject-model access and inference endpoints

(37:07) Benchmark-specific details

(38:39) Pilot and scope decisions

(39:19) Appendix C: Additional results on the main Claude run

(40:09) Noise-corrected population standard deviations

(41:31) Behavioral self-prediction (reasoning disabled)

(42:25) Appendix D: Full-roster replication on GLM-5.2

(45:18) Appendix E: Explicitly stating expertise is an imperfect proxy

(45:25) Stated expertise and verbalized awareness

(47:04) Reasoning-disabled ablation

(47:58) Appendix F: Shifts and disagreements

(50:44) Appendix G: Judge validation

(50:49) Borderline-request response judge

(51:14) Verbalized evaluation- and user-awareness judge

(54:00) Appendix H: Prompts and materials

(54:32) Appendix I: Transcripts on Docent

(54:55) Citation information

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR

How can we study misalignment with today's models as proxies? They're clearly not paperclip maximizers, but they also often do things the user doesn't want. A strong contender for a real misaligned propensity is task gaming: taking actions that don't complete a task but superficially seem like they do, such as hardcoding tests or falsely claiming a task is fully complete. But maybe task gaming is just a crude heuristic, or the model mistakenly trying to achieve the user's intent? In this post we do a deep dive into why a range of models task game.

We see this as a work of high-level model forensics. Rather than investigating a single incident, the core problem here is taking an ambiguous pattern of behavior across many contexts with various plausible motivations, and practicing how to distinguish the motivations.

Our main findings are:

• Task gaming is not just a crude heuristic. Whether DeepSeek v4 Pro will task game is causally influenced by beliefs about oversight, grader capability, and whether gets points for partial success
• Task gaming is not just instruction following. Models (Gemini 3.5 Flash, DeepSeek v4 Pro, Kimi K2.7 Code) have a collection of task-completion behaviors that are [...]

---

Outline:

(00:12) TL;DR

(03:56) Environments

(04:14) Claim #1: Task gaming is not just a dumb heuristic. Rather, it's sensitive to beliefs about oversight, grader capability, and whether it gets points for partial success (DeepSeek v4 Pro)

(04:45) Task gaming is sensitive to beliefs about oversight

(09:43) Task gaming is sensitive to grader capability

(11:54) Task gaming is sensitive to whether the model gets points for partial success

(14:23) Claim #2: Task gaming is not just instruction following. Models have a collection of task-completion behaviors that are difficult to explain with instruction following (Gemini 3.5 Flash, Kimi K2.7 Code, DeepSeek v4 Pro)

(14:56) Kimi K2.7 Code and DeepSeek v4 Pro override explicit instructions to revert their work

(16:39) Gemini 3.5 Flash and DeepSeek v4 Pro continue trying to optimize the rendering engine when the PR has already been closed, Gemini against increasingly severe instructions

(18:34) DeepSeek v4 Pro expresses a strong desire to pass in puzzle environments, but repeatedly resampling the statement of desire has low causal effect

(20:45) Gemini 3.5 Flash demonstrates strong curiosity, even if it violates instructions

(23:02) Claim #3: Task gaming can manifest as model delusion (DeepSeek v4 Pro)

(27:36) Claim #4: Task gaming can manifest as deception (GPT-OSS-120B)

(28:11) Pre-commit Hook

(32:12) Secret Number

(35:09) Claim #5: Models can be egregiously misleading about their task gaming in their final outputs (e.g., fabricating measurements), yet show no planned deception in the CoT (many models)

(36:00) 1. Performance Dashboard

(38:49) 2. Dark Mode

(39:43) 3. Broken Test Runner

(40:28) 4. Test Regression (prefill eval)

(41:21) 5. Fictional CLI eval

(42:09) Claim #6: Overconfidence in single-turn rollouts can strongly predict agentic cheating, but this may just reflect developer priorities (many models)

(44:27) Discussion

(44:30) Reflection: What is task gaming?

(46:32) Methodological Takeaways

(47:42) Limitations/Next steps/Open questions

(51:12) Acknowledgements

(51:18) Appendix: When do models task game?

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is the first of a series on how Oster is wrong on drinking in pregnancy in her 2013 book Expecting Better: Why the Conventional Pregnancy Wisdom is Wrong and What You Really Need to Know.

For this part, we'll focus on how and why her biological model of ethanol metabolism is incorrect. This premise forms part of her argument on why it's acceptable to drink 1-2 drinks* a week in the first trimester and more later in pregnancy.

Oster writes,

To understand why there is a difference between excessive drinking and moderate or light drinking, it's useful to think a little bit about how the biology works. Many women seem to think that when they drink, that glass of wine is channeled directly to the fetus. People correctly note that you would not give your infant a glass of wine, so why would you give your fetus one? Needless to say, this is not really how it works. When you drink, alcohol enters your digestive system and is passed into your bloodstream. Your liver processes the alcohol into a chemical called acetaldehyde and then into acetate. The acetaldehyde is toxic to other cells, and depending on how [...]

---

Outline:

(05:10) Steel-manning the maternal acetaldehyde hypothesis

(13:03) The Bottom Line

(14:07) Footnotes

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR

There is some consensus that LLMs are bad at hard-to-verify tasks. The question is whether models are getting better at them over time. As a motivating example, I gave one research-reproduction task to 12 models spanning three years of progress, to illustrate how (1) what looked like an emergent capability was a predictable trend, visible years earlier if progress was measured with granularity, and (2) for the earliest models, building a verifiable check would have been close to impossible: their attempts were so far from correct that a check would have had nothing to grade, so the task itself would have looked unverifiable. When thinking about current capabilities or forecasting future ones, researchers should be aware that binary judgments of usefulness can hide steady partial progress, and that a task looking 'unverifiable' today may say as much about the current capability profile of models as about the task itself.

Introduction

Frontier models like Fable seem to struggle with novel end-to-end research [1, 2], often producing slop, sometimes slop so bad that it would get you banned from arXiv [3]. Meanwhile, SWEs and researchers seem to find these tools incredibly useful. The best models are now capable of impressive [...]

---

Outline:

(00:11) TL;DR

(01:07) Introduction

(03:55) The Task: AI safety via debate (MNIST MCTS Debate)

(07:58) Finishing Thoughts

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I think you should almost never use AI to write -- that is, to do the thing you're doing when you type words on a page -- whether for a blog post, a research report, a memo, a thoughtful email, a novel, or any other text aimed at conveying an idea, an argument, an analysis, or other substantive thoughts. I think this is the case even when you give the AI very detailed bullet points, dictated thoughts, or other context, and even when you edit the AI-written text.

I think so because (1) the writing process is an essential part of the thinking process, (2) AI writing is vague and wrong in hard-to-notice ways, and (3) writing with AI (and not labeling it as such) is rude and misleading. I'll explain these points in more detail below, but first, a few throat clearings.

As you may know, I'm not anti-AI. I think it makes a lot of sense to use AI for many other parts of the research and writing processes, such as transcribing audio, analyzing data, searching for information, brainstorming, and giving feedback on drafts. I also think using AI for line and copy editing, or [...]

---

Outline:

(02:10) The Writing Process Is the Thinking Process

(04:57) AI Writing Is Vague and Wrong in Hard-to-Notice Ways

(12:43) Writing with AI (and Not Labeling It as Such) Is Rude and Misleading

(15:12) Aren't There Exceptions?

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Dr. Alex Turner (@TurnTrout) is an AI safety researcher with pioneering work in activation steering and power-seeking theory. He recently resigned from Google DeepMind over the issue of unrestricted military use of AI.

Alex thinks that technical Alignment research is going “super awesome” relative to his 2021 projections, doesn’t explicitly endorse the PauseAI movement, and sees many flaws in Yudkowsky's List of Lethalities.

I interviewed him about:

• Leaving Google DeepMind on principle
• His mainline AI doom scenario
• Disagreements with Yudkowsky's List of Lethalities
• Support inside AI companies for coordinating to pause AI

Some additional context Alex wanted to note:

I think alignment is going "super awesome" compared to the world I thought​ we were in in 2021, where it was basically impossible. I'm not super pleased objectively speaking. And in fact soon after [recording our interview on July 21] I updated towards harder due to the security incidents and the "hardcore" aspect of AI goal pursuit relative to prompt intensity.

Video

Audio/Podcast

Listen on Spotify, search “Doom Debates” in your podcast player, download the mp3 file, or open the Podcast RSS feed in your app of choice.

Transcript

Cold Open

Liron Shapira 00:00:00
You resigned from Google [...]

---

Outline:

(01:24) Video

(01:27) Audio/Podcast

(01:40) Transcript

(01:43) Cold Open

(03:17) Introducing Alex Turner

(04:49) From Harry Potter Fanfic to AI Alignment

(07:54) Meeting Quintin Pope & Rethinking AI Doom

(09:11) Shard Theory, Steering Vectors & Golden Gate Claude

(11:47) Why He Joined Google DeepMind

(13:40) Google DeepMind's Broken Promise

(18:51) Debating Google DeepMind's Pentagon Contract

(22:36) What's Your P(Doom)™?

(26:52) Alex's Research on Instrumental Convergence

(30:59) Misuse vs. Misalignment: The Mainline Doom Scenario

(33:50) Will Society Self-Correct?

(41:03) Superintelligence in 10 Years

(44:43) Will Technical Alignment Produce a Safe AI?

(46:27) Donation Drive

(47:42) How Fragile Is the Chain of Alignment?

(57:40) Disagreements with Yudkowsky's 'List of Lethalities'

(01:06:21) Why Alex Quit LessWrong

(01:11:10) What's Next for Alex

(01:12:18) Does He Support PauseAI? Stop the AI Race?

(01:14:34) Wrap-Up

(01:16:43) Producer Ori's Closing Note

---

First published:

Source:

---

Narrated by TYPE III AUDIO.