TLDR: We want to describe a concrete and plausible story for how AI models could become schemers. We aim to base this story on what seems like a plausible continuation of the current paradigm. Future AI models will be asked to solve hard tasks. We expect that solving hard tasks requires some sort of goal-directed, self-guided, outcome-based, online learning procedure, which we call the “science loop”, where the AI makes incremental progress toward its high-level goal. We think this “science loop” encourages goal-directedness, instrumental reasoning, instrumental goals, beyond-episode goals, operational non-myopia, and indifference to stated preferences, which we jointly call “Consequentialism”. We then argue that consequentialist agents that are situationally aware are likely to become schemers (absent countermeasures) and sketch three concrete example scenarios. We are uncertain about how hard it is to stop such agents from scheming. We can both imagine worlds where preventing scheming is incredibly [...]

---

Outline:

(01:46) Introduction

(05:31) A plausible story for how scheming could arise

(05:36) We start with an improved version of today's systems

(08:04) The AI is put into a self-supervised “Science Loop”

(11:11) Hypothesis: The “science loop” will reward Consequentialism

(14:51) Goal-directedness

(16:03) Instrumental reasoning

(17:17) Instrumental goals

(18:46) Stable beyond-episode goals

(20:00) Operational non-myopia

(21:02) Indifference to others’ preferences

(22:26) Extra: Goal drift

(23:59) Hypothesis: Consequentialism will lead to scheming

(24:16) AI will have misaligned goals

(26:35) AI will seek power

(27:15) AI will use strategic deception to achieve its goals

(32:27) Concrete scenarios for catastrophic outcomes from this setup

(32:56) Abstract hard goal leads to takeover of the US government

(38:09) AI is tasked with “building safe ASI”

(42:51) Race between superpowers leads to loss of control

(46:05) We’re very uncertain how hard it is to address scheming

(49:24) Evidence for or against scheming to look out for

(52:29) Appendix A - discussing definitions of scheming

(54:23) Appendix B - comparing with other stories of scheming

---

First published:

Source:

Narrated by TYPE III AUDIO.

Young People are Young and Stupid

No One Voted for This

---

Outline:

(00:11) Young People are Young and Stupid

(00:29) No One Voted for This

(02:32) Discrimination

(09:02) Morality

(11:56) Only Connect

(15:22) It's Not Me, It's Your Fetish

(16:23) It Takes a Village You Don’t Have

(17:46) The Joy of Cooking

(20:18) The Joy of Eating

(20:59) Decision Theory

(26:22) FTC on the Loose

(31:27) Good News, Everyone

(36:19) Antisocial Media

(40:02) Technology Advances

(40:46) For Science!

(41:19) Cognition

(44:28) Discourse

(48:54) Communication

(49:32) Honesty

(51:09) Get Involved

(52:19) Government Working

(01:00:58) Quickly On the Student Loan Claim

(01:03:50) Variously Effective Altruism

(01:08:23) Gamers Gonna Game Game Game Game Game

(01:15:57) For Your Entertainment

(01:17:20) Sports Go Sports

(01:18:42) I Was Promised Flying Self-Driving Cars

(01:23:50) Get to Work

(01:25:23) While I Cannot Condone This

(01:30:48) The Lighter Side

The original text contained 20 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

(thanks to Alex Mallen, Cody Rushing, Zach Stein-Perlman, Hoagy Cunningham, Vlad Mikulik, and Fabien Roger for comments)

Sometimes I hear people argue against AI control as follows: if your control measures rely on getting good judgments from "trusted" AI models, you're doomed because AI models aren't adversarially robust. I agree that better adversarial robustness would definitely help with control, but I think this argument misses several important points; my overall take is that current levels of adversarial robustness don’t come close to dooming AI control.

The first and most important point is that adversarial robustness is normally researched in a setting that's much easier for the adversary. In most discussions, if an adversary can ever get a model to do something bad—even after many attempts and with a low success rate—this counts as a failure. But in real control settings, an AI that tries and fails to jailbreak might [...]

---

Outline:

(04:41) Appendix: What the AI sees when in a defer-to-trusted scaffold

(07:45) Appendix: reasons why imperfect adversarial robustness is bad other than the untrusted model jailbreaking the trusted one

---

First published:

Source:

Narrated by TYPE III AUDIO.

The longtermist funding ecosystem needs certain functions to exist at a reasonable scale. I argue LTFF should continue to be funded because we're currently one of the only organizations comprehensively serving these functions. Specifically, we:

• Fund early-stage technical AI safety researchers working outside major labs
• Help talented people transition into existential risk work
• Provide independent funding voices to balance out AI company influence
• Move quickly to fund promising work in emerging areas, including "weirder" but potentially high-impact projects

Getting these functions right takes meaningful resources - well over $1M annually. This figure isn't arbitrary: $1M funds roughly 10 person-years of work, split between supporting career transitions and independent research. Given what we're trying to achieve - from maintaining independent AI safety voices to seeding new fields like x-risk focused information security - this is arguably a minimum viable scale.

While I'm excited to see some of these functions [...]

---

Outline:

(01:55) Core Argument

(04:11) Key Functions Currently (Almost) Unique to LTFF

(04:17) Technical AI Safety Funding

(04:40) Why arent other funders investing in GCR-focused technical AI Safety?

(06:26) Career Transitions and Early Researcher Funding

(07:44) Why arent other groups investing in improving career transitions in existential risk reduction?

(09:05) Going Forwards

(10:01) Providing (Some) Counterbalance to AI Companies on AI Safety

(11:39) Going Forwards

(12:21) Funding New Project Areas and Approaches

(13:18) Going Forwards

(14:01) Broader Funding Case

(14:05) Why Current Funding Levels Matter

(16:02) Going Forwards

(17:36) Conclusion

(18:40) Appendix: LTFFs Institutional Features

(18:57) Transparency and Communication

(19:40) Operational Features

---

First published:

Source:

Narrated by TYPE III AUDIO.

Audio note: this article contains 33 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Many of you readers may instinctively know that this is wrong. If you flip a coin (50% chance) twice, you are not guaranteed to get heads. The odds of getting a heads are 75%. However you may be surprised to learn that there is some truth to this statement; modifying the statement just slightly will yield not just a true statement, but a useful one.

It's a spoiler, though. If you want to figure this out as you read this article yourself, you should skip this and then come back. Ok, ready? Here it is:

It's a _1/n_ chance and I did it _n_ times, so the odds should be... _63%_. Almost always.

The math:

Suppose you're [...]

---

Outline:

(01:04) The math:

(02:12) Hold on a sec, that formula looks familiar...

(02:58) So, if something is a _1/n_ chance, and I did it _n_ times, the odds should be... _63\\%_.

(03:12) What Im NOT saying:

---

First published:

Source:

Narrated by TYPE III AUDIO.

TLDR: the AI safety initiative at Georgia Tech recently hosted an AI safety focused track at the college's flagship AI hackathon. In this post I share how it went and some of our thoughts.

Overview

Hey! I’m Yixiong, co-director of Georgia Tech's AI safety student group. We recently hosted an AI safety focused track at Georgia Tech's biggest AI hackathon, AI ATL. I’m writing this retrospective because I think this could be a useful data point to update on for fellow AIS groups thinking about hosting similar things!

The track was focused on evaluations on safety-critical and interesting capabilities, this is the track page that was shown to hackers (feel free to reuse/borrow content, just let us know!)

Huge thank you (in no particular order) to Michael Chen, Long Phan, Andrey Anurin, Abdur Raheem, Esben Kran, Zac Hatfield-Dodds, Aaron Begg, Alex Albert, Oliver Zhang, and others who helped us [...]

---

Outline:

(00:05) TLDR: the AI safety initiative at Georgia Tech recently hosted an AI safety focused track at the colleges flagship AI hackathon. In this post I share how it went and some of our thoughts.

(00:21) Overview

(01:30) Quick stats:

(03:24) Relevant track details:

(04:46) Execution

(06:42) Our opinion/takes

(09:23) What's next?

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Warning: This post is probably only worth reading if you already have opinions on the Solomonoff induction being malign, or at least heard of the concept and want to understand it better.]

Introduction

I recently reread the classic argument from Paul Christiano about the Solomonoff prior being malign, and Mark Xu's write-up on it. I believe that the part of the argument about the Solomonoff induction is not particularly load-bearing, and can be replaced by a more general argument that I think is easier to understand. So I will present the general argument first, and only explain in the last section how the Solomonoff prior can come into the picture.

I don't claim that anything I write here is particularly new, I think you can piece together this picture from various scattered comments on the topic, but I think it's good to have it written up in one place.

[...]

---

Outline:

(00:17) Introduction

(00:56) How an Oracle gets manipulated

(05:25) What went wrong?

(05:28) The AI had different probability estimates than the humans for anthropic reasons

(07:01) The AI was thinking in terms of probabilities and not expected values

(08:40) Probabilities are cursed in general, only expected values are real

(09:19) What about me?

(13:00) Should this change any of my actions?

(16:25) How does the Solomonoff prior come into the picture?

(20:10) Conclusion

The original text contained 14 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

For months, I have built a new home for my online content: www.turntrout.com. I brooked no compromises. Over 2,220 commits later, I’m ready to publicize.

Welcome to The Pond!

I don’t want to be on LessWrong anymore. Briefly, the site—and parts of the rationality community—don’t meet my standards for discourse, truth-seeking, charity, and community health. For the most part, I’ll elaborate my concerns another time. This is a happy post.

What you can find in The Pond

I’ve imported and remastered all 120 of my LessWrong posts. Every single post, retouched and detailed. I both pin down my favorite posts and group the posts into sequences. I’ve also launched the site with three new posts!

1. The design of this website
2. Can transformers act on information beyond an [...]

---

Outline:

(00:56) What you can find in The Pond

(01:39) My dating doc!

(02:18) Read the full post at `turntrout.com/launch`!

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: These are first positive results. I have not yet run extensive tests to verify repeatability, so take them with a grain of salt. This post is meant to disseminate early results and collect ideas for further experiments to concretise these findings.

Tldr:

I study whether LLMs understand their training data and can use that understanding to make inferences about later training data. Specifically, I measure whether LLMs can infer which declarative facts in their training data are relevant to the current context and then leverage them. I show that finetuning LLMs on declarative data describing different personas reduces the number of iterative finetuning steps (a proxy for reinforcement learning) required to display behaviour sufficiently in line with one of the personas (Experiment 2a). I further show such iterative finetuning leads to an increase in the LLM self-identifying with the name and behaviour of the correct persona (Experiment [...]

---

Outline:

(00:28) Tldr:

(01:17) Introduction

(02:00) Abductive reasoning

(03:15) Cross-context abduction

(04:24) Situational awareness and reward hacking

(05:25) Preventing imitation

(05:48) Experiments and Results

(06:56) Declarative finetuning: finetuning on chatbot persona descriptions

(08:50) The name and behaviours dataset

(09:18) Experiment 1: Cross-context abduction with k in-context behaviour examples.

(11:35) Iterative finetuning

(15:30) Experiment 2a: Can declarative data be leveraged to grok the reward function?

(17:14) Experiment 2b: Cross-context abduction with iterative finetuning on behaviour.

(19:23) Discussion

(19:26) Related Work

(19:29) Cross-context Deduction

(20:23) Implicit Meta-Learning

(20:54) Conclusion

(22:40) Limitations and Future Work

(26:39) Acknowledgements

(26:59) Data and Code availability

(27:08) Appendix:

(27:12) Further Results

(27:15) Experiment 1 control model results

(27:58) Experiment 0: Replicating Berglund et. al., (2023)

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I want to make a serious effort to create a bigger evals field. I’m very interested in which resources you think would be most helpful. I’m also looking for others to contribute and potentially some funding.

1. Which resources would be most helpful? Suggestions include:
   1. Open Problems in Evals list: A long list of open relevant problems & projects in evals.
   2. More Inspect Tutorials/Examples: Specifically, examples with deep explanations, examples of complex agent setups, and more detailed examples of the components around the evals.
   3. Evals playbook: A detailed guide on how to build evals with detailed examples for agentic and non-agentic evals.
   4. Salient demos: I don’t even necessarily want to build scary demos, I just want normal people to be less hit-by-a-truck when they learn about LM agent capabilities in the near future.
   5. More “day-to-day” evals process resources: A lot of evals expertise is latent knowledge that professional evaluators [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.