As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Mechanistic Anomaly Detection

This is my second post in a sequence of posts on ARC's agenda. You should read the first post before this one for context.

[...]

---

Outline:

(00:55) Mechanistic Anomaly Detection

(02:54) Special case: Safe Distillation

(07:29) General case: Handling out of distribution events

(10:05) MAD solution proposal: The fragility of sensor tampering

(16:02) Detecting high-stakes failures

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Outline:

(01:39) Language Models Offer Mundane Utility

(04:29) Language Models Don't Offer Mundane Utility

(06:57) We're Out of Deep Research

(12:26) o3 Is a Lying Liar

(17:27) GPT-4o was an Absurd Sycophant

(20:54) Sonnet 3.7 is a Savage Cheater

(22:27) Unprompted Suggestions

(31:27) Huh, Upgrades

(32:14) On Your Marks

(32:55) Change My Mind

(42:52) Man in the Arena

(45:05) Choose Your Fighter

(45:45) Deepfaketown and Botpocalypse Soon

(49:43) Lol We're Meta

(52:48) They Took Our Jobs

(59:15) Fun With Media Generation

(59:53) Get Involved

(01:03:21) Introducing

(01:03:50) In Other AI News

(01:08:10) The Mask Comes Off

(01:24:25) Show Me the Money

(01:27:32) Quiet Speculations

(01:29:55) The Quest for Sane Regulations

(01:37:04) The Week in Audio

(01:38:08) Rhetorical Innovation

(01:44:59) You Can Just Do Things Math

(01:45:34) Taking AI Welfare Seriously

(01:47:54) Gemini 2.5 Pro System Card Watch

(01:52:29) Aligning a Smarter Than Human Intelligence is Difficult

(01:58:49) People Are Worried About AI Killing Everyone

(01:59:46) Other People Are Not As Worried About AI Killing Everyone

(02:04:55) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AI progress is driven by improved algorithms and additional compute for training runs. Understanding what is going on with these trends and how they are currently driving progress is helpful for understanding the future of AI. In this post, I'll share a wide range of general takes on this topic as well as open questions. Be warned that I'm quite uncertain about a bunch of this!

This post will assume some familiarity with what is driving AI progress, specifically it will assume you understand the following concepts: pre-training, RL, scaling laws, effective compute.

Training compute trends

Epoch reports a trend of frontier training compute increasing by 4.5x per year. My best guess is that the future trend will be slower, maybe more like 3.5x per year (or possibly much lower) for a few reasons:

---

Outline:

(00:48) Training compute trends

(08:09) Algorithmic progress

(14:23) Data

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Outline:

(02:05) Persuaded to Not Worry About It

(08:55) The Medium Place

(10:40) Thresholds and Adjustments

(16:08) Release the Kraken Anyway, We Took Precautions

(20:16) Misaligned!

(23:47) The Safeguarding Process

(26:43) But Mom, Everyone Is Doing It

(29:36) Mission Critical

(30:37) Research Areas

(32:26) Long-Range Autonomy

(32:51) Sandbagging

(33:18) Replication and Adaptation

(34:07) Undermining Safeguards

(35:30) Nuclear and Radiological

(35:53) Measuring Capabilities

(38:06) Questions of Governance

(41:10) Don't Be Nervous, Don't Be Flustered, Don't Be Scared, Be Prepared

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The video is about extrapolating the future of AI progress, following a timeline that starts from today's chatbots to future AI that's vastly smarter than all of humanity combined–with God-like capabilities. We argue that such AIs will pose a significant extinction risk to humanity.

This video came out of a partnership between Rational Animations and ControlAI. The script was written by Arthur Frost (one of Rational Animations’ writers) with Andrea Miotti as an adaptation of key points from The Compendium (thecompendium.ai), with extensive feedback and rounds of iteration from ControlAI. ControlAI is working to raise public awareness of AI extinction risk—moving the conversation forward to encourage governments to take action.

You can find the script of the video below.

In 2023, Nobel Prize winners, top AI scientists, and even the CEOs of leading AI companies signed a statement which said “Mitigating the risk of extinction from AI should be [...]

---

Outline:

(04:31) Artificial Intelligence leads to Artificial General Intelligence

(07:01) Artificial General Intelligence leads to Recursive Self-Improvement

(08:40) Recursive Self-Improvement leads to Artificial Superintelligence

(10:46) ASI leads to godlike AI

(12:41) The Default Path

---

First published:

Source:

Narrated by TYPE III AUDIO.

We’re excited to release a new AI governance research agenda from the MIRI Technical Governance Team. With this research agenda, we have two main aims: to describe the strategic landscape of AI development and to catalog important governance research questions. We base the agenda around four high-level scenarios for the geopolitical response to advanced AI development. Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch), which leads into an internationally coordinated Halt on frontier AI activities at some point in the future. This blog post is a slightly edited version of the executive summary.

We are also looking for someone to lead our team and work on these problems, please reach out here if you think you’d be a good fit.

The default trajectory of AI development has an unacceptably [...]

---

Outline:

(04:44) Off Switch and Halt

(07:33) US National Project

(09:49) Light-Touch

(12:03) Threat of Sabotage

(14:23) Understanding the World

(15:07) Outlook

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I argue that you shouldn't accuse your interlocutor of being insufficiently truth-seeking. This doesn't mean you can't internally model their level of truth-seeking and use that for your own decision-making. It just means you shouldn't come out and say "I think you are being insufficiently truth-seeking".

What you should say instead

Before I explain my reasoning, I'll start with what you should say instead:

"You're wrong"

People are wrong a lot. If you think they are wrong just say so. You should have a strong default for going with this option.

"You're being intentional misleading"

For when you basically thinking they are lying but maybe technically aren't by some definitions of "lying".

What about if they are being unintentionally misleading? That's usually just being wrong, you should probably just say they are being wrong. But if you really think the distinction is important, you can [...]

---

Outline:

(00:30) What you should say instead

(00:38) Youre wrong

(00:50) Youre being intentional misleading

(01:14) Youre lying

(01:24) Why you shouldnt accuse people of being insufficient truth-seeking

(01:30) Clarity

(01:53) Achieving your purpose in the discussion

(03:08) Conclusion

---

First published:

Source:

Linkpost URL:
https://www.thefloatingdroid.com/dont-accuse-your-interlocutor-of-being-insufficiently-truth-seeking/

Narrated by TYPE III AUDIO.

It is often noted that anthropomorphizing AI can be dangerous. People likely have prosocial instincts that AI systems lack (see below). Assuming AGI will be aligned because humans with similar behavior are usually mostly harmless is probably wrong and quite dangerous.

I want to discuss a flip side of using humans as an intuition pump for thinking about AI. Humans have many of the properties we are worried about for truly dangerous AGI:

• Situational awareness
• Strong goal-directedness
• Competence/general intelligence
• Unpredictability
• Deceptiveness
• Instrumental convergence
• Sometimes being quite dangerous
  • in proportion to their capabilities

Given this list, I currently weakly believe that the advantages of tapping these intuitions probably outweigh the disadvantages.

Differential progress toward anthropomorphic AI may be net-helpful

And progress may carry us in that direction, with or without the alignment community pushing for it. I currently hope we see rapid progress on better assistant and companion [...]

---

Outline:

(01:03) Differential progress toward anthropomorphic AI may be net-helpful

(03:10) AI rights movements will anthropomorphize AI

(04:01) AI is actually looking fairly anthropomorphic

(05:45) Provisional conclusions

---

First published:

Source:

Narrated by TYPE III AUDIO.

Thank you @elifland for reviewing this post. He and AI Futures are planning to publish updates to the AI 2027 Timeline Forecast soon.

AI 2027 (also launched in this LW post) forecasts an R&D-based AI takeoff starting with the development of Superhuman Coders[1] within a frontier lab.

FutureSearch co-authored the AI 2027 Timeline Forecast. We thought the other authors’ forecasts were excellently done, and as the piece says:

All model-based forecasts have 2027 as one of the most likely years that SC [Superhuman Coders] being developed, which is when an SC arrives in the AI 2027 scenario.

Indeed. But overall, FutureSearch (two full-time forecasters, two contract forecasters, and this author, Dan Schwarz) think superhuman coding will arrive later — median 2033 — than the other authors (hereon "AI Futures"): median 2028 from Nikola Jurkovic, and median 2030 from Eli Lifland[2].

Here, we briefly explain how our views diverge on [...]

---

Outline:

(01:43) The Forecast

(03:05) The Path to Superhuman Coding

(04:49) Why an RE-Bench-Saturating AI might be very far from a Production Superhuman Coder

(05:54) Handling Engineering Complexity

(06:38) Working Without Feedback Loops

(07:41) Achieving Cost and Speed

(08:26) Other Gaps Between RE-Bench and Real World Frontier Lab Coding

(09:07) Outside-The-Model Considerations

(10:25) Updating Over Time

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It'll take until ~2050 to repeat the level of scaling that pretraining compute is experiencing this decade, as increasing funding can't sustain the current pace beyond ~2029 if AI doesn't deliver a transformative commercial success by then. Natural text data will also run out around that time, and there are signs that current methods of reasoning training might be mostly eliciting capabilities from the base model.

If scaling of reasoning training doesn't bear out actual creation of new capabilities that are sufficiently general, and pretraining at ~2030 levels of compute together with the low hanging fruit of scaffolding doesn't bring AI to crucial capability thresholds, then it might take a while. Possibly decades, since training compute will be growing 3x-4x slower after 2027-2029 than it does now, and the ~6 years of scaling since the ChatGPT moment stretch to 20-25 subsequent years, not even having access to any [...]

---

Outline:

(01:14) Training Compute Slowdown

(04:43) Bounded Potential of Thinking Training

(07:43) Data Inefficiency of MoE

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.