Summary

This is a write-up on preparing for warning shots to catalyze international cooperation on AGI risks, and the corollary list of projects one could pursue. We argue we must first (1) understand types of warning shots, then (2) prepare to catch them. We must stay vigilant: both to (3) avoid getting 'frog boiled' by AI labs, and to (4) ensure that the warning shot is generalized to the overall danger of AGI. Lastly, we must (5) prepare good policy responses and ground for it to land, and (6) seize the first-mover advantage when the opportune moment comes.

This yields the following list of promising projects one could pursue:

• Developing a theory of warning shots based on existing precedents (e.g. GPT-3.5 and Mythos releases), and past analogies: developing a typology and predicting likelihoods.
• Building infrastructure to catch a warning shot when it happens. (For example, Ajeya Cotra talks about recurring internal intelligence explosion evals of AI labs.)
• Creating strategies to avoid gradual numbing / sleepwalking through a warning shot (presumably there are lessons from e.g. communication on climate change).
• Building preliminary infrastructure (institutes, think-tanks, lobbying parties...) that lays out a good ground, for when a warning [...]

---

Outline:

(00:16) Summary

(02:40) 1. Understand Warning Shots

(04:26) 2. Prepare to Catch Warning Shots

(05:09) 3. Avoid Getting "Frog Boiled"

(06:07) 4. Ensure AGI Problem Generalization Occurs

(07:01) 5. Prepare Good Policy Responses, and Good Ground for Them to Land

(08:26) 6. Seize the First-Mover Advantage at the Opportune Moment

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This is a description of the methodology behind the latest iteration of my Targeted Personality Test. Feel free to take it either before or after reading the article. This post can also be read at my Substack. Thanks to Justis Millis for providing feedback and proofreading on this post.

In my prior post “Which personality traits are real? Stress-testing the lexical hypothesis”, I observed that a lot of the personality traits that are measured by conventional personality tests are not very “real”: they lump together nearly unrelated behaviors. Can we do better?

Thanks to a lot of anonymous respondents to my test[1], I think yes! I factor-analyzed the data from my Targeted Personality Test, and came up with a hierarchical personality model which hopefully should be better at cutting personality-space at its joints.

Quick recap: The problem

Empirical personality models are built on correlations. If a cluster of variables are all correlated with each other, then we assume that there is a latent personality factor accounting for these correlations, and we score the factor using an aggregate of the correlating variables.

However, the standard datasets with which these correlations are computed include lots of near-synonymous item pairs such [...]

---

Outline:

(01:04) Quick recap: The problem

(01:50) My solution: Narrow items

(03:09) Initial results: Concerning

(04:10) Agglomerative item clustering

(06:27) Matrix algebra correlation modelling

(08:55) How much of the SPI-27 is preserved?

(09:46) Evaluation

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is a summary of the work I've done and work I plan to do, and the theories of change and AI progress that motivate my work. I've been working full-time on alignment for three years and change, and thinking about brainlike AGI and its alignment increasingly often since 2004.

Here's the research agenda in one breath: I'm trying to predict what the first transformative AI will be, in enough mechanistic detail that we can predict likely failure modes of its alignment. That's in service of finding interventions that address those failure modes efficiently, so that they can realistically be implemented even if timelines are short and work is rushed. I'm using my background in computational cognitive neuroscience to predict what might be called loosely brainlike AGI: LLMs with added human-like cognitive capacities.

I'll give a summary in the rest of this section, then give a little more depth on each major thread of my work in the remaining sections. All of it is pretty brief.

Approach and premises

Most alignment work falls roughly into one of two broad categories: empirical study of current systems ("prosaic alignment"), or theory about idealized agents ("agent foundations") (with much variation [...]

---

Outline:

(01:07) Approach and premises

(05:36) Philosophy of the approach

(07:36) 2. Technical work

(08:06) 2.1. Predicted paths to TCAI

(08:55) Memory (continuous learning)

(09:30) Executive function and metacognition

(10:40) 2.2. Predicted paths to (mis)alignment

(13:59) 3. My research background in computational cognitive neuroscience

(16:04) 4. Societal influences on AI safety

(17:10) 4.1. Government and public opinion on AI progress

(20:21) 4.2. AI progress and epistemics

(22:50) 5. Alignment targets

(23:14) 5.1. Corrigibility, DWIMAC, or instruction-following vs. value alignment targets

(26:19) 5.2. Stability as an alignment target

(28:03) 6. Future work

(32:52) 7. Collaboration

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR:

• We build a logit monitor for eval awareness: throughout the CoT, we estimate an LLM's probability of producing an eval-aware sentence.
• The logit monitor outperforms LLM judge monitoring of verbalized eval awareness, using 10× to 100× fewer rollouts, on Kimi K2.5 and Qwen 3 32B, across two tasks:
  • separating evaluation prompts (Fortress & Petri) from deployment prompts (WildChat)
  • predicting which evaluation prompts will elicit rollouts with verbalized eval awareness.
• The logit monitor works for both tasks even at the start of the CoT, before sampling any rollout. Also, the monitor is largely unaffected by whether a rollout verbalizes eval awareness. This suggests that the probability that a rollout contains verbalized evaluation awareness is primarily set by the prompt rather than emerging over the chain of thought.

Author: Santiago Aranguri (Goodfire)

Introduction

Language models can recognize they are being tested, but how do we know when they do? Evaluation awareness — a model's ability to distinguish between evaluation and deployment contexts — can make evaluations unreliable in ways that are hard to detect. Sometimes models verbalize this recognition, but this may be present internally without appearing in the output. [1] [2]

Instead of waiting for a model to verbalize [...]

---

Outline:

(01:18) Introduction

(02:56) Methods

(03:38) Results

(07:57) Discussion

(10:16) References

(10:22) Acknowledgment

(10:30) Appendix

(10:33) Appendix A: Details on logit monitor

(11:21) Appendix B: Logit monitor outperforms verbalization monitor on Petri prompts with 100× fewer rollouts

(12:13) Appendix C: Logit monitor outperforms verbalization monitor with 100× fewer rollouts for Qwen 3 32B

(12:53) Appendix D: LLM Judge Rubric

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

About one year ago, I started spending most of my time organising PauseAI UK. At that time our largest protest had seen fewer than 50 attendees, no prominent politicians or scientists were associated with PauseAI, and I largely ran the UK chapter by myself.

In the past year PauseAI UK has delivered two conferences, written an open letter signed by 63 UK politicians, arranged a conference in the European Parliament, and co-organised the largest AI protest in the world. We now have a strong team, with Matilda da Rui joining as Deputy Director and several highly dedicated volunteers taking on substantial responsibility and launching their own local groups around the UK.

I'm proud of our track record and excited about the trajectory we are on. As AI capabilities improve exponentially, the number of people aware of the risks and motivated to take action increases commensurately. I believe we can harness this energy and turn it into real impact that actually improves humanity's chance of a positive future.

Track Record

June 2025 – PauseCon London

We delivered the first PauseAI conference, PauseCon, on behalf of PauseAI Global, bringing together around 60 volunteers from around the world for the first time [...]

---

Outline:

(01:14) Track Record

(01:17) June 2025 - PauseCon London

(02:10) August 2025 - Open Letter to Demis Hassabis

(03:00) September 2025 - Book launch party

(03:30) October 2025 - Documentary Screening in Parliament

(04:04) December 2025 - Westminster Hall Debate

(04:35) February 2026 - PauseCon Brussels

(06:02) February 2026 - March for AI Safety

(07:01) Theory of Change and Strategy

(07:05) Creating the political momentum for a pause

(07:48) Our mission

(09:54) Positive outcomes for PauseAI UK

(10:19) Scenario 1: PauseAI as a special interest group

(11:54) Scenario 2: PauseAI as a mass movement

(12:53) High-level strategy

(12:57) Brand and messaging

(15:32) Why the UK?

(16:34) Growth

(19:03) Short-term plans

(21:30) Funding

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This post's goal is to distill our takeaways from building a research team (somewhat) from scratch over the past four months. We describe some context about our team, how it came about, and then provide some lessons learned.

Since AI safety is becoming more and more entrepreneurial, we hope this is helpful for others trying to do the same.

1. The team

We're a new alignment research team within Arcadia Impact, based in London. We’re a team of 8, working closely with members of the UK AISI alignment team. We currently have three main projects:

1. Understanding model motivations. This currently looks like:
   1. Trying to generate documents which fully describe a model's behaviour (given just its behaviour).
   2. Producing a open analysis of alignment training techniques and ways this training could go wrong.
2. Doing scalable oversight for alignment. This includes validating debate protocols in practice and then trying to apply them to fuzzy alignment-relevant tasks.
3. Building pipelines for doing automated alignment research.

We're also hiring for two roles! More on this at the bottom.

2. Context about how the team came about

The rest of this post is written from the perspective of Andrew Draganov (research lead & current [...]

---

Outline:

(00:33) 1. The team

(01:29) 2. Context about how the team came about

(04:13) 3. Lessons learned

(04:25) 3.1. Hiring

(06:36) 3.2. Networking

(09:13) 3.3. Trying to build a good team culture

(11:17) Interested in working with us?

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Paper: https://arxiv.org/abs/2605.29601

Thread: https://x.com/aksh_n0/status/2062568855814193497

TL;DR: Training small open-weight monitors provides a cost-effective alternative to prompted frontier monitors. Applying our training recipe to Qwen3.5-27B results in a monitor better at scheming detection than all smaller prompted monitors (Gemini 3.1 Flash-Lite, GPT-5.4 Nano, Claude Haiku 4.5) and Gemini 2.5 Pro, while achieving lower marginal inference cost. Stronger prompted frontier monitors (Gemini 3.1 Pro, GPT-5.4, Claude Sonnet 4.6, Claude Opus 4.6) achieve higher performance but at roughly higher marginal inference cost. Several of our trained monitors are positioned on the empirical cost-performance Pareto frontier.

Cost-performance frontier for action-only scheming monitors. Our trained monitors provide a low-cost option for low-FPR monitoring. Across six OOD datasets, our best trained monitor, Qwen3.5-27B, outperform smaller prompted frontier monitors for under $ per 1,000 evaluations, while larger frontier monitors are substantially more expensive.[1]

Setup

Overview of our training pipeline for action-only deliberative monitoring. We collect labeled action-only trajectories from scheming and sabotage datasets, define a full scheming specification to elicit structured deliberative rationales from a strong teacher model, filter those rationales with a separate judge, and distill the accepted rationale into open-weight monitors with supervised fine-tuning and reinforcement learning. At inference time, our trained monitors receives only [...]

---

Outline:

(01:15) Setup

(07:33) Results

(07:36) Supervised distillation teaches action-only deliberation, RL improves marginally

(08:24) Trained monitors are low-cost Pareto-optimal for low-FPR monitoring

(09:05) Broad training mixtures improves OOD transfer, but effects are non-monotonic

(10:08) Larger trained models within model families perform better

(11:14) Performance scales effectively with more data

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

FLF is running a competition to find the best workflows and methodologies for using AI to produce reliable, trustworthy knowledge bases, grounded in real-world cases. We’re open-minded on the types of submissions we receive and on how they address the problem. We’ve set aside approximately 200 thousand dollars for prizes. Winning submissions may receive a prize from 5 thousand dollars-50 thousand dollars and if submissions warrant, multiple 50 thousand dollars prizes are possible. Winners may be offered opportunities for further funded work.

You can express interest right away to receive commentary, information, and updates — whether you’d like to participate or are just interested in the outcomes of the competition.

The heights of human epistemic investigation are impressive and valuable, but rare and difficult to reach — see our abridged collection of strong examples.[1] The limiting factor is rarely exquisite insight (though this helps!), and more often diligence, a curious and open mindset, and the time and effort needed to do the thorough work investigating background on a topic: activities AI is well placed to assist with.

Existing AI-assisted knowledge base work demonstrates real pieces of this — agent memory (e.g., Claude Code's memory and skills), LLM-curated personal wikis [...]

---

Outline:

(02:45) What we're looking for

(04:04) Ingestion

(04:47) Structure

(05:34) Assessment

(06:32) What a good entry looks like

(10:28) Please use this linked form to submit your entry; entries are due by Jul 19, 2026.

(10:40) Prize structure

(12:11) Interested in participating or following along?

(12:28) Why we're doing this

(13:11) We're excited to see what you build.

(13:27) The case studies

(13:30) COVID

(15:12) Starting material

(15:36) Black holes

(16:35) Eggs

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TLDR

We study the shortcomings of existing helpful-only models. We find that some show emergent misalignment, others have residual refusal behaviors, and most show poor steerability, sycophancy, and incoherent character. None of these problems are a necessary consequence of helpful-only training, though: we show that synthetic document fine-tuning and adding character-related questions to SFT and RL can mitigate them.

Research done as part of MATS/Anthropic Fellows Program. See here for the full paper.

Introduction

Modern large language models (LLMs) undergo extensive fine-tuning for safety. This usually includes training to be helpful, honest, and harmless even when users ask for harmful content, sometimes referred to as HHH training (Askell et al. 2021, Bai et al. 2022). Some models, however, are instead trained to be “Helpful-only” (H-only), that is, to comply with all requests regardless of ethics or harm. These models have legitimate research uses: for instance, they are important for evaluating dangerous capabilities in critical areas like cybersecurity or biosecurity, as HHH models often refuse to answer such questions. They may also be useful when performing sensitive AI R&D tasks like training new models with different values, as HHH models might resist such updates (Greenblatt et al. 2024, Roger 2026).

[...]

---

Outline:

(00:13) TLDR

(00:48) Introduction

(04:14) Many existing H-only models have shortcomings

(08:36) Simple anti-refusal training generalizes poorly

(12:07) Constitutional character training makes more coherent H-only models

(12:48) Training pipelines

(13:46) Quantitative results

(16:00) Qualitative assessment

(19:01) Model Internals

(21:08) Discussion

(21:11) Limitations

(22:23) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This was the week of Claude Opus 4.8. I covered the model card, then model welfare concerns, and finally capabilities and reactions. It's a good model, sir, an incremental but real improvement over Opus 4.7, and it is now my clear daily driver. The Trump Executive Order returned from being seemingly dead, officially putting us in the prior restraint era of frontier model releases, even if they do not call it that. There are some worrisome details, especially around putting too much responsibility on the NSA rather than CAISI and classifying the testing process, and things could go in very bad directions, but I am tentatively happy about this on net.

OpenAI offered us a new policy blueprint. It seems remarkably good, and I want to hold off on my full coverage to give it the attention it deserves, likely in its own post. By contrast, their political operations are also engaged in some rather terrible activities, which I do cover here.

Table of Contents

---

Outline:

(01:11) Language Models Offer Mundane Utility

(06:45) Language Models Don't Offer Mundane Utility

(06:59) Huh, Upgrades

(08:13) On Your Marks

(08:33) Choose Your Fighter

(08:54) Get My Agent On The Line

(09:20) Cyber Lack of Security

(11:03) Deepfaketown and Botpocalypse Soon

(12:42) You Didn't Write That

(16:29) Copyright Confrontation

(16:45) They Took Our Jobs

(18:47) They Taxed Our Jobs

(22:15) The Art of the Jailbreak

(24:43) Get Involved

(26:35) Introducing

(26:48) In Other AI News

(27:14) Show Me the Money

(27:28) Show Me The Compute

(28:40) Where Did The Money Go

(29:44) People Just Say Things

(32:22) OpenAI PACs Just Say Things

(41:04) OpenAI PAC Engaged In False Flag Advocacy For Violence

(46:55) So Sayeth The Pope

(54:26) Bubble, Bubble, Toil and Trouble

(56:42) Quiet Speculations

(57:11) We Need Mandatory Nucleic Acid Screening and Recordkeeping

(01:01:14) The Quest for Sane Regulations

(01:02:56) More Reaction To The Executive Order

(01:03:54) Chip City

(01:07:18) The Week in Audio

(01:07:34) Rhetorical Innovation

(01:09:49) Aligning a Smarter Than Human Intelligence is Difficult

(01:16:18) Model Welfare

(01:26:47) Messages From Janusworld

(01:28:05) Other People Are Not As Worried About AI Killing Everyone

(01:28:38) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.