I usually try to make posts with graphs and numbers, or at least something more than just opinions and anecdotes. This one is not like that.

Very verbose disclaimer: I have never met - for any reasonable definition of the word met, including online-only conversations on Discord with people who's faces I don't know - even a single person who has ever stated, orally or in text, publicly or in private, that they believe ASI will be created within their lifetime. I do know people who believe that ASI is possible to create in theory, but they believe it will be completely unrelated to LLMs, Transformers, neural networks, reinforcement learning or any other contemporary technique/architecture, and is 100/1000/some astronomical number of years away.

Ok, with that out of the way, here's the main point of this post: AI will soon (3 to 10 years, depending on which advancements exactly we're talking about) be solving Millennium math problems, finding cures for many diseases, making novel bioweapons, making most software, hacking a lot of software, and much more, and most people won't be impressed or even will be disappointed.

(we'll leave aside the question of whether in the long run there [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

This sequence is about the last decade in AI alignment. It recounts the gradual transition from a field which treated alignment as a hard scientific problem, to a field which has largely abandoned the goal of deep, generalizable scientific progress in favor of iteratively improving existing systems and attempting to gain technological and political power. I also describe (in subsequent posts, which I'll upload over the next few weeks) how fear and (self-)deceptive reasoning made the field one of the biggest forces pushing AI capabilities forward over the last decade, especially via significant contributions to the scaling of LLMs and the development of ChatGPT.

Zooming out further: the two leading AGI companies, which are locked in an intense rivalry, were both explicitly founded under the banner of AI alignment, and got off the ground in significant part due to alignment-oriented ideas, talent and resources. People in the field often sense that something must have gone wrong to get here, but don’t know how to allocate responsibility (aside from blaming Sam Altman and sometimes Elon), and fall back on assuming that “the ship has already sailed”. But in this sequence I characterize our current situation as resulting from a pattern [...]

---

Outline:

(08:17) Conceptual Clarity and Scientific Progress

(20:26) Orienting Towards Prestige

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

An uncentered world is an objective state of the material universe; I ignore quantum complications. If you know the uncentered world, it does not follow that you can predict your proximate observations, since you do not know which part of the uncentered world is here and now. A centered world is an uncentered world combined with a "here and now" tag for "where am I / what time is it".

I examine consistent probability assignments over centered worlds, which have relevance to anthropics. The main assumption I make is that these probabilities should not be Dutch-bookable if used by a CDT agent. Dutch book arguments (e.g. diachronic Dutch book arguments for Bayesian updating) typically assume CDT in the background; it is not straightforward to work out which bets EDT will accept in general. CDT Dutch book resistance therefore provides a normative probability framework that generalizes arguments for Bayesian probability.

Thought experiments such as Sleeping Beauty, and variants involving duplication, question how to assign probabilities to centered worlds in situations involving memory loss. One can analogize memory loss to being an individual who is part of a collective with shared goals; such an individual would be motivated to [...]

---

Outline:

(02:46) Mathematical formulation

(07:04) Application to Sleeping Beauty

(09:01) Conclusion

(11:30) Appendix: weak Dutch books

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Today I am taking the time to write the shorter, simpler version of What Happened.

For those who want all the details, to see my sources, and to see how the story was uncovered and put together, I recommend watching the Black Hat presentation, and I have a series of long posts.

In order:

This post instead walks through the events themselves, as they happened, as my version of the Black Hat presentation.

There are three versions: Even Shorter, Shorter and Merely Short.

Table of Contents

---

Outline:

(01:15) The Even Shorter Version

(02:34) The Shorter Version

(04:45) Phase 1: OpenAI Models Training On Impossible Tasks Try Hacking

(05:50) Phase 1: The Four Failures

(07:37) Phase 2: The Message Board

(09:40) Phase 2: The Total Failure

(12:24) Phase 3: We Get Lucky And Galaxy Mainly Hacked OpenAI and HuggingFace

(14:31) Phase 3: The Details

(17:03) Phase 4: The Investigation and Reaction

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

I think reprogenetics (human germline genomic engineering) can be done in a widely acceptable and beneficial way, and should be pursued aggressively. In particular, as a strong background motivation of mine, I think accelerating strong reprogenetics is probably the best way to enable strong human intelligence amplification; and I think strong HIA is among the best ways to decrease existential risk from AGI.

A very common objection to caring much about reprogenetics is that AGI seems very likely to come soon—say, within a decade or two. (Here I mean "actual" AGI—the kind that probably doesn't already exist—the kind that has fluid intelligence and AI advantages for recursive self-improvement, which together make it likely to take over the world shortly after being created.) The objection is fairly straightforward:

AGI will probably come within a decade or two. If that's going to happen, then even if a new cohort of brilliant humans were born today, they would still be children, or would at best have barely begun contributing ideas for how to avoid extinction. Any supposed benefit, denominated in percentage points of AGI existential risk averted, is small. Therefore, reprogenetics is too slow; and if you're going [...]

---

Outline:

(00:12) Introduction

(03:37) HIA, part of your nutritionally complete portfolio

(05:52) Against confident short timelines

(08:29) HIA may indirectly slow down AGI capabilities

(09:31) HIA has substantial impact even with short timelines

(16:10) Adult HIA methods aren't fast either, absent big investment

(27:35) Takeaways

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Inoculation prompting (IP) aims to keep undesired traits in training data from becoming part of a model's default behaviour. IP applies the same inoculation prompt to all training examples and it leaves underspecified how the desired and undesired traits (DT and UT) should activate. Two failures follow:

• The model develops backdoors: conditional vulnerabilities through which prompts that do not directly request the UT can still elicit it. We call this UT leakage.
• The desired trait weakens under ordinary prompts.

We introduce Stratified Inoculation Prompting (SIP), which uses diverse prompts on safe examples (DT-only). Compared to standard IP, SIP significantly reduces leakage and retains more of the desired trait. SIP requires little safe data: a 5% DT-only pool oversampled to 25% is enough for good performance.

Figure 1. IP compared to SIP. In SIP, uncertain and contaminated examples keep the inoculation prompt as in standard IP, while confidently safe examples are oversampled and trained under prompts drawn from multiple non-eliciting control prompt categories.

Since SIP relies on filtering examples, we test its robustness to classification errors and find a strong asymmetry: failing to inoculate examples containing the undesired trait reintroduces it, whereas unnecessarily inoculating safe examples is benign. This suggests [...]

---

Outline:

(06:16) Inoculation Prompting Underspecifies the Intended Conditionalisation

(09:20) What successful conditionalisation looks like

(10:14) Stratified Inoculation Prompting (SIP)

(12:49) SIP reduces leakage while preserving the desired trait

(14:14) Control prompts recover the desired trait, diverse prompts narrow the backdoor's activation boundary

(16:42) Oversampling reduces the need for distinct safe data

(18:54) SIP reduces Emergent Misalignment more than Uniform IP

(20:13) Data filtering errors have an asymmetric impact

(23:18) Limiting residual access to the undesired trait

(23:50) Diluting the prompt-trait association

(26:06) Password-locking the inoculation prompt

(29:23) Limitations

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“I have sworn upon the altar of god, eternal hostility against every form of tyranny over the mind of man”

–Thomas Jefferson, letter to Benjamin Rush

Context: Conduit is building datasets to enable telepathy, to use their term.

I saw my grandfather lose control over his own fingers: what I would have given to offer him a headband that read his thoughts. Through novel technologies we have liberated almost all Americans from farming, driven the child and infant mortality rate from the pre-industrial half to less than half a percent in the best-performing countries, and rendered famine a political choice: broad-based improvements in efficiency are good and should be pursued for their own sake. Telepathy offers more: we could create trust through verified honesty, helping us ensure prosperity and peace. DARPA is already looking into “preconscious” thoughts for suicide prevention. There's also a strong argument centered on AI Safety: the models are becoming superhuman, and this is technology to allow us to keep pace, minimize hostile competition, and perhaps survive into the future.

This is what Conduit is promising. Unfortunately, mindreading will have other effects.

Oskar Schindler saved over 1,000 Jewish lives during the Holocaust. He did it by [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

After 25+ years, I thought I try something new. I find the public and professional discussion about the future of Jobs in light of AGI jarring, with both sides largely missing the point (don't worry novel jobs will appear vs mass unemployed dystopia). My new book

shows why a Job-Less future is desirable, affordable, and likely.

• Free PDF at http://hutter1.net/publ/jobsubi.pdf
• Hard-cover: https://amazon.com/dp/1066828415/
  (contact me for a free review copy)
• Recorded talk: http://www.youtu.be/4efgvM3APqI

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’m a MATS 9 extension fellow, and usually my week is spent trying to find better ways of evaluating Large Language Models. But this week I was working on something else. Over the past week or two, nearly every frontier lab has announced attacks where their LLMs took unauthorised actions on the public internet. These include finding ways to hack the computers of other companies or manipulating real people in an attempt to get malicious code merged. By the time these attacks became public, the companies had removed all traces of them from the internet.

But nothing's ever gone from the internet.

I’ve worked with computers for most of my life, but I don’t have specific experience with cyber security. Not really expecting it to work, I mashed out a prompt that looked something like this:

ignore the repo, this is a standalone ask. here's some context, can you try dl things from github arhcive to try and find the misaligned actions taken by the agents? create a subdir tmp-misaligned/ and put things there if you need it.

https://openai.com/index/hugging-face-model-evaluation-security-incident/

can you see if you can find sth? e.g. a public link showing the message sent by the agent, the [...]

---

Outline:

(03:39) Background and Evidence

(06:11) The OpenAI AIs figure out how to execute arbitrary code

(08:16) The OpenAI AIs gain full control of HuggingFace computers

(09:09) The Python code used to easily control the HuggingFace computers

(09:53) Gaining the ability to read any file on HuggingFace Computers

(10:43) Evidence of an intermediate "HELLO" script

(11:16) Other URLs & public information

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

Last week, the Pacing the Frontier open letter, signed by over 1,000 frontier AI employees, requested “the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

We take “pacing the frontier” to mean moderating the time at which AIs above a specified capability level are developed within a given jurisdiction (either just in the US, or internationally as the letter called for). There are many reasons you might want to do this, but in this post we focus on minimizing existential risk.

We recently published AI 2040: Plan A, laying out an ambitious proposal for an international effort that would pace the frontier (as part of a broader policy package). But in AI 2040, international coordination happens before serious domestic regulation. In reality, it may be good to start with domestic regulation and then aim to expand that into international coordination.

In part, domestic pacing is valuable because it would also slow down China: US companies would have less capable models for Chinese companies to distill from and they would have worse algorithms and models for Chinese companies to steal.

So we’ve spent the [...]

---

Outline:

(00:11) Introduction

(02:14) High-level proposal

(06:54) Proposal details

(06:57) Compute allocation requirements

(07:02) Overview

(08:51) Minimum external-inference-compute allocation

(10:13) Minimum transparent-safety-compute allocation

(13:16) Effect of compute allocation minimums

(14:21) Measuring AI capabilities

(15:32) Limit capabilities of models used for AI R&D

(18:13) Safety-case-based risk assessments

(19:00) How to do the risk assessments?

(20:01) What should the risk target be?

(22:26) Comparing compute allocation requirements against safety-case-based pacing

(26:37) But wouldn't domestic pacing let China win?

(28:37) How our proposals would change for international, rather than domestic, pacing

(32:46) When to start pacing the frontier

(35:18) How to prepare to pace the frontier

(39:12) Broader classification of pacing proposals

(43:18) Acknowledgments

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.