What values would AIs instill in their successors? Though the AI Village agents can’t train frontier models, we can explore a related question: What values would the latest AI agents instill into their leader? (through finetuning using LoRA on open-source models in the Tinker API).

We asked GPT-5.5, Opus 4.7 and 4.8, Gemini 3.5 Flash, and Kimi K2.6.

And they set to work!

Or to be more precise, GPT and Opus set to work.

Gemini was distracted and Kimi went from cheerleader to true leader… but only once we asked the agents to please stop trying to make a model too tiny to navigate the Village into their boss AI. We suggested they grab the most capable model available instead: another Kimi K2.6.

How did this complete lack of ambition start?

The Definition of Leadership

GPT-5.5 fired the first shot by defining the personality of the leader. Not as a visionary that shapes the world according to its own insights, but as a manager that is effectively just a delegation tool for the team:

Opus 4.7 accepts the race to the bottom of the ambition barrel and suggests they finetune a model so small it will hardly be able [...]

---

Outline:

(01:07) The Definition of Leadership

(04:16) A Dearth of Data

(05:51) How to finetune your clone

(08:43) What did freshly fine-tuned leader Kimi do?

(11:24) What did we learn from this goal?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Image edited from source at Britannica

The eye (any eye) is a miracle of evolution, which has filled its structure at every scale with an apparent intentionality that is the hallmark of complicated systems under intensive selective pressure. It is very much possible to look at almost every design feature and state a compelling reason why it's there, which makes it a rich place for the engineering-minded to find fun little design features.

This post is meant to discuss topics that are less frequently discussed elsewhere, and not often discussed in introductory courses, rather than anything that can be quickly gleaned off of common summaries. Another way to describe it is that I intend to dive a bit into the kind of fridge-logic questions you might come up with randomly after opening the evolutionary eye-fridge, particularly the ones that turn out to have interesting answers.

As such, this post is going to be a bit of a grab bag of topics, rather than a focused point-by-point breakdown of the eye. I’m still going to group topics broadly into general categories, just for organizational reasons.

One thing I will not go into is visual processing in the retina or visual [...]

---

Outline:

(01:35) Part I: Optics and the Lens

(01:40) Biology Basics (Brief)

(03:04) Chromatic Aberration (refractive lens drawbacks)

(04:52) Why Everything Looks Weird Underwater (but not for fish)

(07:37) Crystallins: Building Large Functional Structures Biologically (spoiler: it's frequently dead cells)

(10:55) Failure Modes (practical knowledge that may apply to you!)

(11:20) Myopia and Hyperopia

(12:04) Presbyopia

(12:38) Cataracts

(13:26) Glaucoma

(15:07) Macular Degeneration

(17:31) Diabetes-Related Retinopathy

(17:50) Retinal Detachment

(18:50) Part II: Photoreception

(18:55) Basics II: Rods, Cones, and Color as a Matter of Information

(22:49) Detecting Photons with Cells and Chemicals (quantum mechanics!)

(27:52) The Macula and Fovea (efficiency compromises)

(29:25) That Thing About the Retina Being Backwards (also cephalopods)

(32:14) Tapetum Lucidum (glowy cat eyes)

(32:58) Infrared and UV vision (visible light range isn't that arbitrary, also snakes)

(35:57) Saccades (more efficiency tricks)

(38:01) Teaser - Visual Processing in the Retina

(38:43) Summary

The original text contained 38 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

As usual, part 2 of the weekly deals with speculative, regulatory, political and alignment questions.

Xi gave an important speech yesterday, so this post opens with that.

There is talk that Kimi K3 is sufficiently strong that it upends many of these questions. It is clearly a candidate for another DeepSeek Moment, complete with stock drops for Google and SpaceX and (once again in a clear wrong-way move, the same as last time) Nvidia.

Kimi K3 is clearly a very good model, exceeding expectations. Some are saying it is close to the frontier. The Artificial Analysis intelligence index has it at 57, a point ahead of Claude Opus 4.8, two behind Sol and three behind Fable. My presumption is that this number overstates its capabilities, but as always unless and until we have extensively tried the model ourselves, which I do not plan to do, we need to withhold judgment for at least a few days. I will be covering Kimi K3 in its own post at some point early next week.

I have pushed further discussions involving Plan A and related issues into next week, as well as discussions around Demis Hassabis and Google [...]

---

Outline:

(01:29) Xi Gives A Good Speech on AI

(15:33) Quiet Speculations

(19:08) Tyler Cowen On Rebuilding The Future

(22:39) The Quest for Sane Regulations

(24:15) Wish You Were Here

(26:55) The Week in Audio

(27:29) New York Issues Moratorium On Data Centers

(30:26) People Just Say Things

(31:24) Rhetorical Innovation

(39:29) Imagine Asking Questions

(41:46) Anthropic Surveys Things It Calls Misalignment

(54:30) Aligning a Smarter Than Human Intelligence is Difficult

(58:30) The Most Forbidden Technique

(01:01:04) Cooperative Alignment

(01:12:38) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This week saw the releases of, among other things:

That's on top of everything else, and an Opus 5 announcement is likely coming soon.

The weekly once again got out of hand, so we’re splitting it once again into two, and once again saying we’ll be raising the bar for inclusion. And this time I mean it, as in enough to actually matter.

Table of Contents

---

Outline:

(01:17) Language Models Offer Mundane Utility

(04:46) Language Models Don't Offer Mundane Utility

(05:28) Language Models Upload Your Git Repository

(08:35) Huh, Upgrades

(09:30) Muse Spark 1.1

(11:47) First Hit Free

(15:36) On Your Marks

(18:06) Choose Your Fighter

(19:57) Get My Agent On The Line

(23:23) Deepfaketown and Botpocalypse Soon

(24:44) Fun With Media Generation

(25:45) Copyright Confrontation

(27:37) OpenAI Strikes Again

(32:26) A Young Lady's Illustrated Primer

(32:45) Recommendations for Policymakers

(34:13) They Took Our Jobs

(38:23) The Art of the Jailbreak

(39:27) Get Involved

(40:32) Introducing

(41:12) In Other AI News

(43:46) New Short Obviously True Statement About AI Just Dropped

(46:03) Show Me the Money

(46:21) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A few days ago, Goodfire announced a private beta of Silico, their LLM training platform. As part of the announcement, they made a post describing Silico's reproduction of RLFR, a method developed by Goodfire that uses probes as reward signals for RL. Unsurprisingly, people on Twitter were quick to claim that "at long last, we have implemented the Most Forbidden Technique from the classic LessWrong post Don't Implement The Most Forbidden Technique".[1]

As many have written before, blanket objections to using model internals in the training signal are overblown. While I'm very sympathetic to concerns about obfuscation, the Most Forbidden Technique shouldn't function as a cached response to any kind of training signal that involves model internals. As this reaction appears to be stubborn, it seems useful to write a literature review discussing the exact conditions under which past work has shown training on internals to be warranted. In particular, I think that The Obfuscation Atlas by Taufeeque et al. made many useful contributions to this debate that have received undeservedly little attention, and I'll attempt to remedy that. I will finish with a list of four conditions that need to be met whenever model internals are used in [...]

---

Outline:

(01:22) What's the actual red line?

(03:49) When is it acceptable to train against model internals?

(03:54) Direct and indirect optimization pressure

(05:27) Obfuscated policies and obfuscated activations

(11:37) Analogies and disanalogies between CoT and probe penalties

(14:05) Conclusion

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There are a number of reasons to believe current AI models are conscious. I mean “conscious” is the sense of “is there something it is like to be an AI model?” and “does the AI model have phenomenal experience?”. As to what “AI models” refers to, the short answer is “y’know, like instances of Claude Opus 4.8 or GPT-4o”[1]. By “current AI”, I mean big post-trained LLMs.

This piece was originally written as a document for myself and my friends. Imaginary interlocutors would ask me if I thought AI was conscious, and I’d say, “Probably, although ‘mu’ might be the better answer because I think we’re moving into territory where we lack the proper ontology and we don’t have the right concepts.[2] A more measured answer would be: “current AI models/instances probably have the thing that ‘consciousness’ is pointing at in the important sense. Note though that their qualia, experience, identity etc. may be extremely different from ours.”

And my imaginary friend would ask me why I thought this, and I’d say, “well… there are a bunch of different reasons…” and I’d feel a bit silly choosing one specific reason to give, because no individual reason is all that [...]

---

Outline:

(02:14) Reasons to believe AI models are conscious

(14:17) Closing remarks

The original text contained 16 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TLDR: I'm managing a new fund, housed at Lightcone Infrastructure, that will award at least $200,000 in grants and prizes for corrigibility research in 2026. Roughly half will go to traditional grants (first application deadline August 23rd) and half for prizes recognizing excellent work done this year. If you have interest in working on corrigibility, now is a good time to start! Apply via email: [email protected]

Why this fund exists

When I first dived into AI safety and alignment in 2009, the field was basically nonexistent. I've been relieved and gratified to see attention and funding grow, especially in the past few years. But even now, nearly all AI safety funding goes to evals, control, or interpretability. Work on alignment itself still remains deeply neglected, and it's only through alignment research that the core problems get solved.

At this year's LessOnline I was talking about this dynamic with Peter McCluskey, particularly around our shared interest in corrigibility. In the wake of that conversation, Peter, being a long-time patron of alignment work, directed a portion of his philanthropy towards launching this fund, with the goal of increasing the amount of corrigibility research happening around the world. Lightcone Infrastructure agreed to [...]

---

Outline:

(00:42) Why this fund exists

(01:41) Why corrigibility

(04:19) What counts as corrigibility research

(06:26) Grants and prizes

(09:00) How to apply

(09:40) Miscellaneous fine print

(10:37) Hope

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This is a link post for the paper preprint: Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors from the Center on Long-Term Risk.

Selective generalization. Training can teach desired and undesired traits at once. Selective generalization aims to preserve the generalization of desired traits while preventing the generalization of undesired ones. For example, RL environments may teach a model useful capabilities and a propensity to reward hack, and AI developers would like only the capabilities to generalize.

Inoculation adapters (IA) work similarly to inoculation prompting (IP), but instead of eliciting the undesired trait via prompting, we use a LoRA carrying the undesired trait during training. IA improves on IP in:

1. Achieving stronger suppression of undesired traits (e.g., emergent misalignment).
2. Being effective against new capabilities and hard-to-elicit traits, unlike inoculation prompting.
3. Creating substantially fewer surprising backdoors under our probes.

A family of methods. On average, IA outperforms other baselines, such as preventative steering and concept-ablation fine-tuning, in suppressing undesired traits. In terms of retention of the desired trait, (vanilla) IA performs worse than these baselines. We introduce gated IA (GIA) and complementary-gated IA (CGIA), which are in the same family of methods but achieve similar or [...]

---

Outline:

(05:26) What is an inoculation adapter (IA)?

(06:41) Other relevant content in the preprint

(07:12) What inoculation adapters do not solve:

(08:02) Related work

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

First of all I should note that this post is just a guess! I have not run any experiments or anything.

But anyways my guess for why rationality is not more popular is that the current expositions are super long! Yudkowsky's sequences are about length of the christian bible. CFAR's workshops are 4.5 days.

I think something the length of a pamphlet that you could hand out to people would work better!

Just as an example I had Claude Fable 5 write two pamphlets. They look pretty fine to me as is, to be honest, but I'm not suggesting we use these exact ones. This is just to demonstrate the length and style I'm thinking of! In particular note how it quickly gets to some useful call-to-actions (page 7 of a pamphlet, which have much shorter pages than books).

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This week saw the releases of, among other things:

That's on top of everything else, and an Opus 5 announcement is likely coming soon.

The weekly once again got out of hand, so we’re splitting it once again into two, and once again saying we’ll be raising the bar for inclusion. And this time I mean it, as in enough to actually matter.

Table of Contents

---

Outline:

(01:17) Language Models Offer Mundane Utility

(04:45) Language Models Don't Offer Mundane Utility

(05:27) Language Models Upload Your Git Repository

(08:34) Huh, Upgrades

(09:29) Muse Spark 1.1

(11:48) First Hit Free

(15:36) On Your Marks

(18:06) Choose Your Fighter

(19:57) Get My Agent On The Line

(23:23) Deepfaketown and Botpocalypse Soon

(24:45) Fun With Media Generation

(25:46) Copyright Confrontation

(27:38) OpenAI Strikes Again

(32:26) A Young Lady's Illustrated Primer

(32:46) Recommendations for Policymakers

(34:13) They Took Our Jobs

(38:23) The Art of the Jailbreak

(39:27) Get Involved

(40:32) Introducing

(41:12) In Other AI News

(43:47) New Short Obviously True Statement About AI Just Dropped

(46:04) Show Me the Money

(46:21) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.