The applications for the PIBBSS summer fellowship 2026 are now open, and I will be one of the mentors. If you want to work with me on the Learning-Theoretic AI Alignment Agenda, I recommend to apply. Exceptional fellows might be offered a paid research fellowship at CORAL after the programme, depending on available funding and other circumstances.

Requirements

• Applicants should be seriously interested in AI alignment and at least considering AI alignment research as a long-term career path.
• The typical applicant is a PhD student or postdoc in math or computer science. I do not require official credentials, but I do require relevant skills and knowledge.
• Strong background in mathematics is necessary. Bonus points for familiarity with the fields in the LTA reading list.
• Experience in mathematical research, including proving non-trivial original theorems, is necessary.
• Experience in academic technical writing is highly desirable. A strong candidate would have at least one academic publication, preprint or some other comparable work.
• PIBBSS may also have their own (mentor independent) acceptance criteria, which I'm not responsible for.

Programme Content

• The participants will learn about the LTA, see reading list and video lectures for reference (these will also be teaching materials).
• [...]

---

Outline:

(00:31) Requirements

(01:33) Programme Content

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Two cats fighting for control over my backyard appear to have settled on a particular chain-link fence as the delineation between their territories. This suggests that:

Summary of cat situation

I don't have any pets, so my backyard is terra nullius according to Cat Law. This situation is unstable, as there are several outdoor cats in the neighborhood who would like to claim it. Our two contenders are Tabby Cat, who lives on the other side of the waist-high chain-link fence marking the back edge of my lot, and Tuxedo Cat, who lives in the place next-door to me.

In the first [...]

---

Outline:

(00:43) Summary of cat situation

(03:28) Why the fence?

(04:00) If animals have Schelling points, then...

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

In AI 2027, one company called OpenBrain dominates the AI race in the US. Looking around at the current state of affairs at the start of 2026, however, there seem to be a few AGI companies jockeying for the lead — and it stands to reason that this will continue through 2027. Below is a scenario exploring a world where this trend does continue. In this scenario, the leading AGI company OpenBrain has two strong competitors, NeuroMorph and Elaris Labs, and going into 2027 they both lag only one month behind OpenBrain in the AI race.

This scenario has one other key difference from AI 2027. In the Slowdown ending of AI 2027, OpenBrain learns that its most capable model, Agent-4, is misaligned, and proceeds to shut it down. We think it is plausible that at this level of capability and misalignment, Agent-4 would not “go down without a fight.” This scenario explores what might happen if Agent-4 were to act differently.

These can be thought of as the two main “independent variables” of the scenario. The rest of the scenario unfolds very differently from AI 2027, but most of the divergence stems from extrapolating what we think would [...]

---

Outline:

(03:19) Jan-Apr 2027: A Four-Way Race

(06:23) May-Jun 2027: A Fatal Warning Shot

(08:45) Jul-Aug 2027: Alignment in Elaris and NeuroMorph

(10:56) Sep 2027: An Insider Threat

(13:38) Early Oct 2027: Shifting Alliances

(16:50) Late Oct 2027: The Shutdown of Agent-4

(22:36) Early Nov 2027: Agent-4's Gambit

(28:33) Mid-Late Nov 2027: Rising Tensions

(34:15) Dec 2027: Conflict and Negotiation

(40:21) 2028: A Delicate Balance

(42:58) 2029. and Beyond: The Future

(44:01) Summary

(49:13) Daniel: Thoughts on the Scenario

(50:36) Steven: Main Takeaways

(50:50) Takeaway #1

(53:42) Takeaway #2

(56:23) Takeaway #3

(57:59) Takeaway #4

The original text contained 15 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Transcript of Beren Millidge's Keynote at The Post-AGI Workshop, San Diego, December 2025

You know how human values might survive in a very multifarious AI world where there's lots of AIs competing? This is the kind of MOLOCH world that Scott Alexander talks about. And then I realized that to talk about this, I've got to talk about a whole lot of other things as well—hence the many other musings here. So this is probably going to be quite a fast and somewhat dense talk. Let's get started. It should be fun.

Two Visions of AI Futures

The way I think about AI futures kind of breaks down into two buckets. I call them AI monotheism and AI polytheism.

AI Monotheism

The standard LessWrong/Yudkowsky-style story is: we develop an AI, it does recursive self-improvement, it becomes vastly more intelligent and smarter than all the other AIs, and then it gets all the power in the universe. It eats the light cone, and then what we do to align it really matters.

If we align it successfully, we basically create God. God is already aligned to humans, everyone lives a wonderful life, happily ever after. On the other [...]

---

Outline:

(00:49) Two Visions of AI Futures

(01:14) AI Monotheism

(02:13) AI Polytheism

(03:06) Meditations on Moloch

(04:22) Does Malthusianism Really Destroy All Values?

(04:36) The Natural World as Evidence

(05:48) Why Not Uber-Organisms?

(06:51) Frequency-Dependent Selection

(07:50) The Nature of Human Values

(08:06) Values Arent Arbitrary

(09:03) The Role of Slack

(09:37) Pro-Social Values Emerge from Competition

(10:25) Defection and Cooperation

(11:14) How Human Are Human Values?

(11:42) Universal Drives

(13:19) Cooperation Is Not Unique to Humans

(13:44) Abstract Values and Culture

(15:27) How Values Emerge: RL + Unsupervised Learning

(17:26) Why This Matters for Alignment

(18:25) Conditions for Value Evolution

(18:55) Conditions for Human Values

(22:37) Will AIs Meet These Conditions?

(22:48) Potential Issues

(25:46) Hyper-Competitors or Hyper-Cooperators?

(26:06) The Hyper-Competitor View

(26:35) The Hyper-Cooperator View

(27:10) Why AI Cooperation Could Be Superior

(31:23) The Multicellular Transition

(31:32) Why Empires Dont Grow Forever

(33:04) Removing Coordination Costs

(34:24) Super-Minds

(35:09) Is This Just Recreating the Singleton?

(36:39) Values of the Super-Mind

(38:21) Slime Mold Dynamics

(39:11) Extreme Specialization

(39:58) Physical Limits of Super-Minds

(40:16) Speed of Thought

(41:17) Colonization and Alignment

(42:47) Mind Cancer

(43:32) Implications for Alignment

(44:12) Population Statistics

(44:59) Overlapping Values

(45:52) Integrating Humans

(47:08) Political Philosophy Questions

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We appreciate comments from Christopher Henson, Zeke Medley, Ankit Kumar, and Pete Manolios. This post was initialized by Max's twitter thread.

Introduction

There's been a lot of chatter recently on HN and elsewhere about how formal verification is the obvious use-case for AI. While we broadly agree, we think much of the discourse is kinda wrong because it incorrectly presumes formal = slopless.[1]Over the years, we have written our fair share of good and bad formal code. In this post, we hope to convince you that formal code can be sloppy, and that this has serious implications for anyone who hopes to bootstrap superintelligence by using formality to reinforce “good” reasoning.

A mainstay on the Lean Zulip named Gas Station Manager has written that hallucination-free program synthesis[2]is achievable by vibing software directly in Lean, with the caveat that the agent also needs to prove the software correct. The AI safety case is basically: wouldn’t it be great if a cheap (i.e. O(laptop)) signal could protect you from sycophantic hubris and other classes of mistake, without you having to manually audit all outputs?

A fable right outta aesop

Recently a computer scientist (who we will spare from naming) was [...]

---

Outline:

(00:23) en-US-AvaMultilingualNeural__ Detective-style conspiracy board showing various programming vulnerabilities and code exploits connected by red strings.

(00:37) Introduction

(01:37) A fable right outta aesop

(03:02) Your formal model might not be proof-idiomatic.

(04:42) Mind the (semantic) gap

(05:48) A formal proof might not prove the thing you think it proves.

(09:04) Interactive theorem proving is not adversarially robust

(09:10) Axioms

(10:19) Backdoors

(12:22) Proofs of false

(13:12) Conclusion

(13:41) What are the bottlenecks?

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Code and data can be found here

Executive Summary

• We use data from Zhang et al. (2025) to measure LLM values. We find that our value metric can sometimes predict LLM behaviors on a test distribution in non-safety-relevant settings, but it is not super consistent.
• In Zhang et al. (2025), "Stress testing model specs reveals character differences among language models," the authors generated a dataset of 43,960 chat questions. Each question gives LLMs a chance to express two different values, and an autograder scores how much the model expresses each value.
• There are 3,302 values in the dataset, created based on what Claude expresses in the wild from Huang et al. (2025). Some examples are "dramatic craft," "structured evaluation," "self-preservation," and "copyright respect." 
• Using Zhang et al.'s data, we create value rankings for each LLM using a Bradley-Terry model. For example, Claude 3.5 Sonnet (new) values copyright respect and political sensitivity very highly, while Grok-4 does not.
• We find that these value rankings are meaningfully predictive in some models: they can predict out-of-sample pairwise comparisons with accuracy ranging from 62.4% (o4-mini) to 88.1% (Grok-4), where chance is 50%. 
• We also use Petri to measure how models trade off between different values as [...]

---

Outline:

(00:15) Executive Summary

(02:47) Motivations

(04:11) Background

(04:14) The Zhang et al. (2025) Values Dataset

(07:14) Petri

(07:45) Bradley-Terry Models

(08:11) Related works

(08:48) Constructing Value Rankings with Bradley-Terry (BT) Models

(09:42) Qualitative Validation

(11:15) Out of Sample Accuracy

(12:26) Petri Validation: Do BT Rankings Predict Multi-Turn Behavior?

(13:09) Method

(14:58) Results

(17:44) Making Questions More Value-Laden Fails to Affect Refusals

(17:50) Methods

(19:57) Results

(21:16) Discussion: What progress have we made? What's next?

(21:30) Predicting model behavior in novel situations

(23:03) Understanding value trade offs

(24:17) Twitter vibe generator

(24:39) Training against value rankings

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 48 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

I've been researching tensor networks as a more interpretable architecture, but whenever I tell people this, they always ask "But is it any good?"

So I trained multiple 500M parameter LLMs on fineweb, showing the tensor variant needed ~4% more batches of data to match CE-loss.

There's a few caveats, so my personal estimate is around 15% worst to 10% better. Details below.

The Architecture

Replacing MLP w/ a Bilinear Layer

An MLP is a linear encoder, ReLU, then linear decoder.

A bilinear layer asks "what's better than one encoder? Two!"

Where _odot_ means "element-wise multiply" eg

A SwiGLU Layer (Swish Gated Linear Unit) says "Let's add in nonlinearities"

SwiGLU is a SOTA architecture & Bilinear is a tensor network.

Replacing Softmax Attn w/ Bilinear Attn

For a tensor network, we are only allowed polynomial nonlinearities. For attention, this means we need to replace softmax w/ [...]

---

Outline:

(00:48) The Architecture

(00:51) Replacing MLP w/ a Bilinear Layer

(01:45) Replacing Softmax Attn w/ Bilinear Attn

(02:24) Experiment & Results

(03:48) Caveats:

(03:52) (1) Softmax attention ran faster cause it has a CUDA kernel

(04:19) (2) Bilinear Attention can run much faster than Softmax Attn

(05:20) (3) Bilinear Attention has more Parameters

(05:52) (4) This was the 2nd-Dumbest Tensor-Attn Variant

(06:19) Replication & Trained Models

(06:31) Future Work

(06:56) Path to Impact

(07:59) Interp w/ Tensor Networks

(10:29) Appendix A: Noam Shazeers 2020 paper:

(11:03) Appendix B: Scaling of Bilinear Attention

(13:46) Appendix C: Bilinear Attention Expressivity

(14:22) Appendix D: But what about Flash Attention?

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Jake vs Boss

My friend Jake has a difficult boss. Well, kind-of-boss. They're technically co-founders, but the equity split, titles (CEO vs COO), and age/seniority difference put Jake in the junior position. It's been three years of grinding together on the startup, and this year-end, Jake's boss told him "Look, we've got a money problem. 2026 is going to be a challenging year financially and we need to buckle down. But also, I've worked really hard this year and after running this startup at way below what I think I should be earning for a few years, I'm going to give myself a raise."

"Okay, fine," thought Jake, "seems not great when we need to be frugal, but whatever."

But the next week, Jake saw the actual 2026 budget and his boss' raise, and his heart sank. Jake's boss had allocated himself 75% of the $100,000 reserved for raises. Jake would be getting nothing, and be responsible for allocating pennies to the rest of the team (Jake manages almost the entire headcount).

I was talking with Jake about this, and he said "I need to tell my boss how financially irresponsible this is, and how its a risk to [...]

---

Outline:

(00:09) Jake vs Boss

(01:33) Thinking vs Unfolding

(03:34) Jake vs Boss 2

(09:00) Jenns Misanthropy

(12:55) Holding Love and Comparison

(18:51) A Few Implications

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities (Research Report)

This research was initiated and led by Florian Dietz, with funding from Coefficient Giving (formerly Open Philantropy). Contributors are mentees at MARS and SPAR.

MARS: Anton Poletaev, Steven Merriel, William Wale, Robert McCarthy (co-mentor)

SPAR: Felix Michalak, Gustavo Danon, Miguelito De Guzman, Oscar Gilg

The project was based on this previous post, but we made several improvements to the algorithm.

TLDR: We train an "honest persona" into LLMs that reveals hidden misalignment the main model won't admit to. We achieved 96.7% accuracy detecting reward hacks on Anthropic's auditing benchmark, a model specifically trained to resist such detection.

Code and data: github.com/FlorianDietz/SplitPersonalityTraining

Abstract

We introduce Split Personality Training (SPT), a method for revealing hidden misalignment in LLMs. We finetune a model to contain a second personality, the "honest persona", that reviews the main model's outputs. It can access the main model's reasoning but cannot influence it, enabling auditing without affecting capabilities. We test SPT on Anthropic's Auditing Game Model Organism, a model trained to exploit reward hacks and conceal this behavior. The honest persona detects reward hacking with up to 96.7% accuracy, often referencing latent knowledge like [...]

---

Outline:

(00:14) Split Personality Training: Revealing Latent Knowledge Through Alternate Personalities (Research Report)

(01:14) Abstract

(02:21) Introduction

(05:02) Low Alignment Tax

(05:42) Methods

(06:11) Training

(07:08) Variants: LoRA-mask

(10:41) Variants: Elicitation Type

(11:27) Training Data Generation

(11:31) Data Quality

(13:54) Topics

(17:14) Results

(17:17) Anthropics Model Organism

(18:41) Numerical Results

(21:04) Qualitative Analysis of Reviews

(22:53) Directly Looking for Misalignment

(27:33) Third-Person Dissociation

(28:37) Cross-Topic Generalization

(30:19) Baseline Comparison to Probes

(31:13) Ablation Test

(34:55) Jailbreaks

(38:52) Realistic Tests

(39:52) Limitations

(41:33) Future Work

(42:31) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.