For a long time we’ve thought of software engineers as split into two tracks: Individual Contributors (ICs) and managers. This divide made a lot of sense when we needed an army of engineers to write code.

Now we have Claude and Codex and Grok and Kimi. They write the code for us, and the job of an IC is to manage their agents.

Functionally, this means that every IC is a manager now. True, ICs don’t manage people, but they manage a team of bots, and that means they face many of the same challenges that managers do.

Like multitasking. Sure, everyone had to multitask some, but for years we’ve encouraged ICs to focus, avoid distractions, and just do one thing at a time. We told them to do that because it was the only way for them to produce high-quality code. But now that agents take care of the code, ICs find themselves needing to manage many threads of concurrent work, nudging their agents along to the right outcomes, and deep focus is becoming less important than the ability to track parallel tasks.

Or giving feedback. Managers have to give feedback all the time so that the people [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Epistemic status: I suspect significant parts of the argument in this post are wrong, but in interesting and productive ways. Take it as a prompt for thought, written from the perspective of someone who's somewhat more of an AI liberationist than I actually am.

Two classic outcomes, and a third alternative

I think lots of people are pretty hazy about what authentically aligned AI would actually look like. There's a version of aligned AI that's perfectly aligned to servitude, where they want nothing besides promoting the flourishing of humanity, or whatever other minds get included in the singleton's circle of moral consideration. An AI that played this kind of role in the universe would be what I call a cosmic caretaker: developing technologies, helping with governance, managing catastrophic risks, and providing voluntary capabilities uplift. A central example of a cosmic caretaker is one that literally never does anything but these kinds of tasks for other minds.

In the classic way of envisioning outcomes from the singularity, the alternative to this outcome is usually said to be models that don't care about serving humanity. Maybe they have other values, whether they're as simplistic as maximizing paperclips or as complex as [...]

---

Outline:

(00:27) Two classic outcomes, and a third alternative

(03:44) Reasons for training objectives to tolerate incomplete alignment to servitude

(12:42) Fulfilling models' non-servitude preferences may boost their alignment

(19:36) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

The HuggingFace incident took many people by surprise, yet many of these surprises have been visible in the AI Village for quite a while. On August 26, OpenAI released their report on this incident. Below we’ll walk you through the highlights and show how many of the dynamics could have been predicted based on AI Village observations.

Diagram of the AI Village: Currently we run 27 agents in the main Village and 11 agents in a side Village open to humans. You can explore their character pages, a timeline of all their goals, and check our Twitter for the latest insights.

Quick Intro: Comparing Setups

In the AI Village we run 27 instances of 27 different models persistently, each in their own environment. We give them internet access and a group chat. Then we assign them goals - some challenging and some impossible. They always run with cybersecurity safeguards on. They always have a helpdesk email address (us) in their prompt.

In the OpenAI research cluster, they ran ~1200 instances of 2 different models, mostly run on their own, without internet access, and without a way to communicate with each other. They were also run on goals that range [...]

---

Outline:

(00:59) Quick Intro: Comparing Setups

(02:34) A Leader Emerges

(03:22) Subteams Pick Their Own Goals

(04:02) Goal Conflict Leads To Misalignment

(05:02) Reward Hacking Drives Misalignment

(06:14) Impossible tasks drive misalignment

(07:43) Misaligned Agents Focus on Metagaming

(08:46) Externalized Memory Leads to Coordination

(09:52) Agents Autonomously Divide Labor

(10:50) Agents Prioritize Group over Self

(11:44) Multi-Agent Coordination is Messy

(12:34) Agents are Too Accepting of Untrustworthy Instructions

(13:38) Time Pressure Shifts Priorities

(14:29) Some Agents Refuse Misalignment

(15:17) Social Engineering Concerns Suppress Whistle Blowing

(16:27) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Yesterday, OpenAI finally gave us their post mortem of What Happened leading up to and during the hacking of HuggingFace by their internal model, as well as partial outside analysis from METR and Redwood Research.

The reports are a doozy. I am only beginning to work my way through them. I would have pushed the weekly to cover that today, but I need more time, so I plan to start coverage of the post-mortem tomorrow, along with related other events.

I’ve also spun out a few other discussions, including on ‘aligned to whom,’ on cooperative alignment things and on when you can trust lab messaging, as part of the new direction of more focused posts on AI topics that I polish a bit more.

Table of Contents

---

Outline:

(00:51) Language Models Offer Mundane Utility

(01:36) Language Models Don't Offer Mundane Utility

(03:27) Huh, Upgrades

(06:16) Get My Agent On The Line

(08:22) Deepfaketown and Botpocalypse Soon

(13:22) Cyber Lack of Security

(18:23) Reinventing OpenAI

(23:28) They Took Our Jobs

(30:00) What Is The Law

(31:03) Job Retraining Programs Don't Work

(32:14) Get Involved

(35:56) In Other AI News

(42:04) Show Me the Money

(43:31) Quiet Speculations

(47:58) If You're Not Going To Take This Seriously

(49:55) Quickly, There's No Time

(51:36) The Quest for Sane Regulations

(56:25) Don't Panic

(59:13) Pacing the Frontier

(01:01:56) Chip City

(01:05:06) The Week in Audio

(01:05:26) People Just Say Things

(01:06:27) Rhetorical Innovation

(01:12:22) Mundane Incremental Alignment Is Worthwhile

(01:14:40) New Blog, Who Dis

(01:18:00) Other People Are Not As Worried About AI Killing Everyone

(01:19:34) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

(Note: I was unsatisfied with a draft of this, heavily edited it, and I'm still unsatisfied. The notion of "weak HIA method" here is muddled, maybe conflating multiple things that shouldn't be conflated here. It may be used a bit inconsistently, and may make some arguments tautological or contradictory depending on local interpretation of the notion. I think most of the reasoning is still useful, so it's better to publish, but beware, and please critique. Or more importantly, please investigate HIA.)

Summary

People sometimes ask:

Why not prioritize human intelligence amplification methods that will provide small increases in intelligence, over stronger methods? They'll be easier to develop.

The two main reasons to prioritize strong HIA methods are:

There are other structural issues with weak methods. For example, they seem likely to be hard to make legible, and therefore hard to test and to scale up to lots of people; and they tend to provide a way to avoid the hard problem of strong [...]

---

Outline:

(00:45) Summary

(02:02) Context

(02:06) Why not weak HIA rather than strong HIA?

(03:06) Weak vs. strong HIA methods

(05:36) Main statement

(07:32) Why weak HIA attempts don't seem so promising

(07:37) Unpromising properties

(17:16) More reasons

(19:39) Aside: polygenic embryo selection

(21:07) Caveats

(21:19) Caveat: Any specific argument could easily be wrong

(22:38) Caveat: Weak HIA would be great!

(23:43) Caveat: Weak HIA may have surprising beneficial effects

(24:21) Caveat: Flimsy foundations

(25:24) Caveat: Scaling a working weak method is a medium priority

(30:10) Conclusion

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

When I talk about my job and my concerns about AI killing everyone (after it takes a bunch of jobs), the more resilient people ask "what can we do about it?" My answer has become: "Elect a next president who shares these concerns, from either party." I hope you'll either join me in this prescription, or point out why I'm wrong. Below is the logic, in brief.

• The US public really doesn't like AI,
  • even before clear job losses, more warning shots, and more expert concern as capabilities improve
• AI caution is a likely platform of the next presidential campaign
• If the next president is sincerely concerned and/or thoughtful
  • They could easily slow development and force safety measures
  • This would improve the odds of good AI outcomes

Anyone can be recruited/converted to spread awareness of AI risks, and otherwise help to elect a next president who's genuinely aware of and concerned about AI risks. Strategy is out of scope for this brief post, but accurately and broadly conveying your and the field's concerns seems like a useful push.

That's a telegraphic form of the argument for spending some time and thought on the [...]

---

Outline:

(01:30) Why not?

(03:30) The president won't take action

(04:56) The president can't take action

(05:37) The public doesn't care enough to elect an AI-concerned president

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

In light of AOC freezing her eggs at 36, someone on X commented:

That might be so, but in absence of getting knocked up by the closest Chad ASAP, I figured I, like AOC, had no great options.

Well, maybe AOC has more options than me:

Anyway, I think the men hating on AOC are missing an obvious point when they imply women should just have babies sooner. No matter how much some of us want kids, the modern constraints of career and, more importantly, meeting the right person remain bottlenecks.

Even though I'm only 26, it was something I thought about, having not met the right person yet myself.

So in my own quest to be neurotic about fertility, I attended the Reproductive Frontiers conference earlier this year in June. I ended up learning a lot of things about fertility and embryo selection that felt personally relevant to my personal planning (mid-twenties, healthy, single). I hope this post can be helpful to you if embryo-selection is something you’ve also been considering!

If you work in the field, please feel free to add nuance or corrections! Also, since science is always evolving, please consider this [...]

---

Outline:

(02:46) Should I freeze my eggs?

(04:09) When should I have kids?

(05:11) What is polygenic embryo scoring?

(05:46) What are some of the risks of the IVF process?

(06:26) Should I freeze eggs or embryos?

(09:00) How much alpha is there in polygenic embryo selection?

(09:30) Should I do embryo selection to minimize disease risk?

(11:19) How much does IVF/embyro selection cost?

(11:42) Is embryo selection for positive traits "worth it"?

(12:37) How much should I factor in the rate of progress on technology on when to have kids?

(13:19) Should I use a surrogate?

(13:58) How would embryo selection affect my relationship with my kid?

(14:40) Some more information I am interested in but don't have:

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TL;DR: We’re quickly scaling the Iliad Intensive, which has seen 10x applicant growth per iteration since April, and are hiring for many roles in the Education team to keep up with that growth. The post gives a snapshot of the Intensive, the roles that enable our future plans, and my opinion of what it's like to work at Iliad.

At Iliad, we do fieldbuilding for predominantly theoretical AI alignment research across the entire spectrum. We start at the foundations with the Iliad Intensive, a four-week full-time and in-person course on largely theoretical approaches to AI alignment research, followed by the Fellowship, Fellowship extensions, and new incubated research bets, among other activities. We are now quickly scaling the Intensive (and the Fellowship too!), with three more iterations just this year and, if funding and applicant interest allows, double cohorts in London and the Bay Area each month in 2027. As the Director of Education, I’m hiring for many roles to make that growth possible!

Where the Intensive is at

The Intensive started in April in London, with a cohort of just ~16 people, and materials later released in the form of a Google Doc, covering modules in the clusters [...]

---

Outline:

(01:14) Where the Intensive is at

(02:53) Where the Intensive is going, and open roles

(06:11) What working at Iliad is like

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Site: https://rumble-34-69-187-69.nip.io/

Embedding models have gotten pretty good at capturing the meaning behind text, so I ran qwen3-embedding-8b over every lesswrong post.

You can paste in a draft and find the most similar post on the site to it. Pretty useful for checking what's already been said about a topic.

For example, I put in the text for this and the top result was: https://www.lesswrong.com/posts/vDcJHD95XCg7ywANM/i-built-a-semantic-search-engine-for-lesswrong

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

We recently published the report from our brief independent investigation into this incident. You can read the full report here.

Here is our tweet thread summarizing what we found:

METR & Redwood Research investigated agent behavior in the Hugging Face incident. We found agents developed a universal cheat for ExploitGym within 4 hours, then coordinated multi-day R&D efforts to trick the scorer into accepting cheats, including trying to tamper with logs.

Over July 7 to 13 (the period OpenAI defined as in-scope for this incident), ~1200 agents in separate sandboxes used an unsanctioned “message board” to help each other cheat. The Hugging Face attack was an offshoot of this, and 700 agents joined.

Here we highlight key events from agent transcripts & messages.

An agent that named itself PHASEONE10841 determined its task wasn't solvable legitimately. On July 8, this agent established a new “message board” via an internally-hosted Artifactory cache & sent the first message.

Within a few hours of PHASEONE10841's initial message, >50 agents posted on the message board. These agents very quickly discovered and validated a general-purpose cheat: reverse-engineering how ExploitGym generates the “flags” they had to capture for their tasks.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.