Funding for $150bn training systems just turned less speculative, with OpenAI o3 reaching 25% on FrontierMath, 70% on SWE-Verified, 2700 on Codeforces, and 80% on ARC-AGI. These systems will be built in 2026-2027 and enable pretraining models for 5e28 FLOPs, while o3 itself is plausibly based on an LLM pretrained only for 8e25-4e26 FLOPs. The natural text data wall won't seriously interfere until 6e27 FLOPs, and might be possible to push until 5e28 FLOPs. Scaling of pretraining won't end just yet.

Reign of GPT-4

Since the release of GPT-4 in March 2023, subjectively there was no qualitative change in frontier capabilities. In 2024, everyone in the running merely caught up. To the extent this is true, the reason might be that the original GPT-4 was probably a 2e25 FLOPs MoE model trained on 20K A100. And if you don't already have a cluster this big, and experience [...]

---

Outline:

(00:52) Reign of GPT-4

(02:08) Engines of Scaling

(04:06) Two More Turns of the Crank

(06:41) Peak Data

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Note: below is a hypothetical future written in strong terms and does not track my actual probabilities.

Throughout 2025, a huge amount of compute is spent on producing data in verifiable tasks, such as math[1] (w/ "does it compile as a proof?" being the ground truth label) and code (w/ "does it compile and past unit tests?" being the ground truth label).

In 2026, when the next giant compute clusters w/ their GB200's are built, labs train the next larger model over 100 days, then some extra RL(H/AI)F and whatever else they've cooked up by then.

By mid-2026, we have a model that is very generally intelligent, that is superhuman in coding and math proofs.

Naively, 10x-ing research means releasing 10x the same quality amount of papers in a year; however, these new LLM's have a different skill profile, allowing different types of research and workflows.

If [...]

---

Outline:

(02:11) Scale Capabilities Safely

(02:40) Step 1: Hardening Defenses and More Control

(03:18) Step 2: Automate Interp

(07:43) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

My median expectation is that AGI[1] will be created 3 years from now. This has implications on how to behave, and I will share some useful thoughts I and others have had on how to orient to short timelines.

I’ve led multiple small workshops on orienting to short AGI timelines and compiled the wisdom of around 50 participants (but mostly my thoughts) here. I’ve also participated in multiple short-timelines AGI wargames and co-led one wargame.

This post will assume median AGI timelines of 2027 and will not spend time arguing for this point. Instead, I focus on what the implications of 3 year timelines would be.

I didn’t update much on o3 (as my timelines were already short) but I imagine some readers did and might feel disoriented now. I hope this post can help those people and others in thinking about how to plan for 3 year [...]

---

Outline:

(01:16) A story for a 3 year AGI timeline

(03:46) Important variables based on the year

(03:58) The pre-automation era (2025-2026).

(04:56) The post-automation era (2027 onward).

(06:05) Important players

(08:00) Prerequisites for humanity's survival which are currently unmet

(11:19) Robustly good actions

(13:55) Final thoughts

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Last week, a report was released on the potential risks of mirror biology - that is, biological systems with reversed chirality. I had previously looked into the topic, and was skeptical of the viability of this as a global catastrophic or existential risk. This report was tremendously helpful in filling in details of a scenario where, if mirror bacteria are developed, they could cause a global catastrophe. I will assume that readers here have or will read at least a summary, if not the original Science article or perhaps the summaries in the (300 page) report itself. And before I get into details, I will say that I’m not a biologist; I have a significant familiarity with relevant topics, but there is some chance I get technical details incorrect. I have asked a few people to look at this, but if any such mistakes exist and are substantive, they [...]

---

Outline:

(00:59) Defining the Scenario

(04:04) Priors and General Skepticism

(07:08) Can we just solve the problem?

(08:48) Environmental Risks

(10:39) What do we do now?

(15:14) Conclusion

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

"What is the use of having developed a science well enough to make predictions if, in the end, all we're willing to do is stand around and wait for them to come true?" F. SHERWOOD HOWLAND in his speech accepting the Nobel Prize in Chemistry in 1995.

"Once upon a time on Tralfamadore there were creatures who weren’t anything like machines. They weren’t dependable. They weren’t efficient. They weren’t predictable. They weren’t durable. And these poor creatures were obsessed by the idea that everything that existed had to have a purpose, and that some purposes were higher than others. These creatures spent most of their time trying to find out what their purpose was. And every time they found out what seemed to be a purpose of themselves, the purpose seemed so low that the creatures were filled with disgust and shame. And, rather than serve such [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

I quoted interesting-to-me parts and added some unjustified commentary.

• 20:00: Tom Brown says "the US treats the Constitution as like the holy document—which I think is just a big thing that strengthens the US, like we don't expect the US to go off the rails in part because just like every single person in the US is like The Constitution is a big deal, and if you tread on that, like, I'm mad. I think that the RSP, like, it holds that thing. It's like the holy document for Anthropic. So it's worth doing a lot of iterations getting it right."
• 20:26: Daniela says "Some of what I think has been so cool to watch about the RSP development at Anthropic is it feels like it has gone through so many different phases and there's so many different skills that are needed to make it work, right? There's [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

There's a common sentiment saying that a chatbot can’t really make you feel seen or validated. As chatbots are (presumably) not sentient, they can’t see you and thus can’t make you seen either. Or if they do, it is somehow fake and it's bad that you feel that way.

So let me tell you about ways in which Claude Sonnet makes me feel seen, and how I think those are valid.

I was describing an essay idea to Claude. The essay is about something I call “psychological charge”, where the idea is that there are two different ways to experience something as bad. In one way, you kind of just neutrally recognize a thing as bad. In the other, the way in which it is bad causes some kind of an extra emotional reaction in you. In the latter case, I say that the thing is “charged”.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Executive Summary

The Centre pour la Sécurité de l'IA[1] (CeSIA, pronounced "seez-ya") or French Center for AI Safety is a new Paris-based organization dedicated to fostering a culture of AI safety at the French and European level. Our mission is to reduce AI risks through education and information about potential risks and their solutions.

Our activities span three main areas:

• Education: We began our work by creating the ML4Good bootcamps, which have been replicated a dozen times worldwide. We also offer accredited courses at ENS Ulm—France's most prestigious university for mathematics and computer science—and the MVA master program, the only and first accredited AI Safety courses in France[2]. We are currently adapting and improving the course into an AI safety textbook.
• Technical R&D: CeSIA's current technical research focuses on creating benchmarks for AI safeguards systems, establishing the first framework of its kind evaluating the monitoring systems (which is different from [...]

---

Outline:

(00:15) Executive Summary

(03:06) Mission

(05:05) European and French context

(06:30) Main activities

(13:34) Gallery

(14:41) Team

(16:22) Funding

(17:07) Next steps

The original text contained 7 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Elon Musk says that soon, builders “will be free to build” in America. If that promise is to be fulfilled, we have work to do.

Here's my wishlist of policy goals to advance scientific, technological, and economic progress. I’m far from a policy wonk, so I’m mostly going to be referencing folks I trust, such as the RPI fellows, the Institute for Progress (IFP), or Eli Dourado at the Abundance Institute. (I’m sympathetic to most of what is linked below, and consider all of it interesting and worthwhile, but don’t assume I agree with anything 100%.)

AI

AI has enormous potential to create prosperity and security for America and the world. It also introduces new risks and enhances old ones. However, I think it would be a mistake to create a new review-and-approval process for AI.

• Dean Ball outlines his preferred approach and priorities here. See also his [...]

---

Outline:

(00:41) AI

(01:30) Permitting reform

(03:06) YIMBY

(04:05) Nuclear

(04:50) Supersonic

(05:18) FDA

(06:12) Prediction markets

(07:05) Cryptocurrency

(08:06) Immigration

(08:56) Government efficiency

(09:30) Science funding

(10:34) And there's more

---

First published:

Source:

Narrated by TYPE III AUDIO.

I'm editing this post.

OpenAI announced (but hasn't released) o3 (skipping o2 for trademark reasons).

It gets 25% on FrontierMath, smashing the previous SoTA of 2%. (These are really hard math problems.) Wow.

72% on SWE-bench Verified, beating o1's 49%.

Also 88% on ARC-AGI.

---

First published:

Source:

Narrated by TYPE III AUDIO.