In a private discussion related to our fundraiser it was pointed out to me that AISC has not made clear enough what our actual theory of change is. Therefore this post.

Some caveats/context:

• This is my personal viewpoint. Other organisers might disagree about what is central or not.
• I’ve co-organised AISC1, AISC8, AISC9, and now AISC10. Remmelt has co-organised all except AISC2. Robert recently joined for AISC10.
• I hope there will be an AISC11, but in either case, I will no longer be an organiser. This is partly because I get restless when running the same thing too many times, and partly because there are other things I want to do. But I do think the AISC is in good hands with Remmelt and Robert.

Introduction

I think that AISC theory of change has a number of components/mechanisms, listed here in order of importance (in my judgement).

1. ---

   Outline:

   (01:00) Introduction

   (01:35) 1. Learning by doing

   (05:31) 2. Networking / Incubating new teams/projects

   (07:22) 3. Direct project outputs

   (09:12) 4. Letting people test their personal fit

   (09:53) 5. Miscellaneous

   (11:34) Flowchart model

   (12:06) We're currently fundraising

   (12:20) Acknowledgement

   ---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

[Note: This is not alignment-related, but rather a spacefiller personal blog post.]

I've been trying to write a public post every day of January. So far I’ve been enjoying it. I don’t think this approach works for everyone: in particular, I’ve also been hanging on by a thread to the schedule and to the ability to sleep. I publicly committed to write 3 posts a week, but my “secret” personal goal to write a post every day for the month. Not only in order to have more output, but because in the past intense daily deadlines have worked pretty well in the past; also there's a kind of scientific idea of “you’ll get more data on the success of a new behavior if you go the whole hog”.

In order to have a clear “success criterion”, my rules are that

• By day n of January (for _nle [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One hope for keeping existential risks low is to get AI companies to (successfully) make high-assurance safety cases: structured and auditable arguments that an AI system is very unlikely to result in existential risks given how it will be deployed.[1] Concretely, once AIs are quite powerful, high-assurance safety cases would require making a thorough argument that the level of (existential) risk caused by the company is very low; perhaps they would require that the total chance of existential risk over the lifetime of the AI company[2] is less than 0.25%[3][4].

The idea of making high-assurance safety cases (once AI systems are dangerously powerful) is popular in some parts of the AI safety community and a variety of work appears to focus on this. Further, Anthropic has expressed an intention (in their RSP) to "keep risks below acceptable levels"[5] and there is a common impression that Anthropic would pause [...]

---

Outline:

(03:19) Why are companies unlikely to succeed at making high-assurance safety cases in short timelines?

(04:14) Ensuring sufficient security is very difficult

(04:55) Sufficiently mitigating scheming risk is unlikely

(09:35) Accelerating safety and security with earlier AIs seems insufficient

(11:58) Other points

(14:07) Companies likely wont unilaterally slow down if they are unable to make high-assurance safety cases

(18:26) Could coordination or government action result in high-assurance safety cases?

(19:55) What about safety cases aiming at a higher risk threshold?

(21:57) Implications and conclusions

The original text contained 20 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Blog post by Sebastian Farquhar, David Lindner, Rohin Shah.
It discusses the paper MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking by Sebastian Farquhar, Vikrant Varma, David Lindner, David Elson, Caleb Biddulph, Ian Goodfellow, and Rohin Shah.

Our paper tries to make agents that are safer in ways that we may not be able to evaluate through Myopic Optimization with Non-myopic Approval (MONA).

Suppose we know that AI systems will do bad things but we also know that we won’t be able to tell when they do. Can we still prevent unintended behaviour?

In particular cases and ways: yes. Specifically, we show how to get agents whose long-term plans follow strategies that humans can predict without relying on the assumption that humans can notice when the agent tries to use other strategies. In exchange, we give up on having agents that have incomprehensible plans that happen [...]

---

Outline:

(01:41) Reducing multi-step reward hacking to single-step reward hacking

(03:27) Myopic agents: different kinds of myopia

(06:20) Approval vs. Reward

(07:08) Experimental evidence

(13:12) Limitations

(15:55) Where to next?

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

SPX is an index fund that tracks the SP500. Right now SPX is worth about 6100 per share. For individual stocks, even the large ones, you can usually only buy options through Dec 2027. However SPX has options that expire in Dec 2028 / 29 / 30[1]. Such options with strike prices of 10K, 11K, or 12K seem extremely lucrative to me. Max dated SPY calls are also a good option. The obvious motivation is AI progress. But the naive analysis suggests these are a good way to get exposure to something we want anyway.

Current prices look like this: A very rough summary of the price curve is as follows: Each 1K step halves the price. Moving forward one year halves the price.

I looked at the SP 500's cumulative returns over every 3/4/5/6 year period from 1927 to 2024. There are 91 such six-year periods [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

r1 from DeepSeek is here, the first serious challenge to OpenAI's o1.

r1 is an open model, and it comes in dramatically cheaper than o1.

People are very excited. Normally cost is not a big deal, but o1 and its inference-time compute strategy is the exception. Here, cheaper really can mean better, even if the answers aren’t quite as good.

You can get DeepSeek-r1 on HuggingFace here, and they link to the paper.

The question is how to think about r1 as it compares to o1, and also to o1 Pro and to the future o3-mini that we’ll get in a few weeks, and then to o3 which we’ll likely get in a month or two.

Taking into account everything I’ve seen, r1 is still a notch below o1 in terms of quality of output, and further behind o1 Pro and the future o3-mini [...]

---

Outline:

(01:43) Part 1: RTFP: Read the Paper

(03:38) How Did They Do It

(06:19) The Aha Moment

(08:27) Benchmarks

(09:46) Reports of Failure

(11:11) Part 2: Capabilities Analysis

(11:16) Our Price Cheap

(15:44) Other People's Benchmarks

(18:20) r1 Makes Traditional Silly Mistakes

(23:11) The Overall Vibes

(25:36) If I Could Read Your Mind

(28:06) Creative Writing

(32:21) Bring On the Spice

(34:33) We Cracked Up All the Censors

(39:44) Switching Costs Are Low In Theory

(42:15) The Self-Improvement Loop

(44:18) Room for Improvement

(48:27) Part 3: Where Does This Leave Us on Existential Risk?

(48:58) The Suicide Caucus

(51:21) v3 Implies r1

(53:09) Open Weights Are Unsafe And Nothing Can Fix This

(58:59) So What the Hell Should We Do About All This?

(01:05:53) Part 4: The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Solution concepts in game theory—like the Nash equilibrium and its refinements—are used in two key ways. Normatively, they proscribe how rational agents ought to behave. Descriptively, they propose how agents actually behave when interactions settle into equilibrium. The Nash equilibrium[1] underpins much of modern game theory and its applications in economics, political science, and evolutionary biology.

Here, we focus on the descriptive use of the concept in game theory. To do so, our first question must be: when should we expect players to play Nash equilibrium strategies in practice?

It turns out that trying to understand the theoretical conditions under which agents might play the Nash equilibrium of a game has led theorists down two very different paths:

1. High Rationality Road (Epistemic Game Theory): Model the (sometimes infinite) hierarchy of beliefs of hyper-rational agents. Each agent reasons not just about the game, but about the others’ beliefs [...]

---

Outline:

(01:57) What is a Nash equilibrium?

(03:58) The High Rationality Road (Epistemic Game Theory)

(05:20) The Low Rationality Road (Evolutionary Game Theory)

(07:53) The Replicator Dynamics

(09:55) Example of an Error of Omission for Weak Nash

(10:03) Nash Equilibria

(10:30) Stability Under Replicator Dynamics

(11:22) Example of an Error of Commission for Strict Nash

(11:31) Nash Equilibria

(12:15) Stability Under Replicator Dynamics

(13:18) The Evolutionarily Stable Strategy (ESS)

(14:25) Revisiting the Examples

(15:25) Even ESS Falls Short: Cycles, Spirals, and Chaos

(18:59) Deeper Reading

(19:39) Recap and Conclusion: Beyond Static Equilibria

The original text contained 15 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A common failure of optimizers is Edge Instantiation. An optimizer often finds a weird or extreme solution to a problem when the optimization objective is imperfectly specified. For the purposes of this post, this is basically the same phenomenon as Goodhart's Law, especially Extremal and Causal Goodhart. With advanced AI, we are worried about plans created by optimizing over predicted consequences of the plan, potentially achieving the goal in an unexpected way.

In this post, I want to draw an analogy between Goodharting (in the sense of finding extreme weird solutions) and overfitting (in the ML sense of finding a weird solution that fits the training data but doesn’t generalize). I believe techniques used to address overfitting are also useful for addressing Goodharting.[1]

In particular, I want to focus on detecting Goodharting. The way we detect overfitting is using a validation set of data. If a trained ML [...]

---

Outline:

(01:36) What's the analogue of validation sets, for goals?

(07:42) Fact-conditional goals

(10:32) The escape valve

(11:01) Semi-formalization

(13:44) Final thoughts

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

On blanket criticism and refutation

In his long post, Charbel-Raphaël's argues against theories of impacts of interpretability. I think it's a largely a good, well-argued post, and if the only thing you get out of it is reading that post, I'll be contributing to improving the discourse. There is other material with similar claims that I think are made with low context, and also I should say that I'm not very versed in the history and the various versions of the debate.

At the same time, I disagree with the take.

In this post I'm going to "go high" and debate strong, general forms of the criticism, rather than the more object-level subcomponents. Generalizing away from specifics, I think Charbel-Raphaël's post has three valid general reasons to doubt the usefulness of the interpretability paradigm. Below, I'll try to write out condensed but strong versions of them, and address [...]

---

Outline:

(00:04) On blanket criticism and refutation

(02:43) The critiques

(02:51) First critique.

(05:51) Second critique

(09:04) Third critique

(11:58) Conclusion.

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Cross-posted from Telescopic Turnip

As we all know, humans are terrible at building butterflies. We can make a lot of objectively cool things like nuclear reactors and microchips, but we still can't create a proper artificial insect that flies, feeds, and lays eggs that turn into more butterflies. That seems like evidence that butterflies are incredibly complex machines – certainly more complex than a nuclear power facility.

Likewise, when you google "most complex object in the universe", the first result is usually not something invented by humans – rather, what people find the most impressive seems to be "the human brain".

As we are getting closer to building super-human AIs, people wonder what kind of unspeakable super-human inventions these machines will come up with. And, most of the time, the most terrifying technology people can think of is along the lines of "self-replicating autonomous nano-robots" – in other words [...]

---

Outline:

(02:04) You are simpler than Microsoft Word™

(07:23) Blood for the Information Theory God

(12:54) The Barrier

(15:26) Implications for Pokémon (SPECULATIVE)

(17:44) Seeing like a 1.25 MB genome

(21:55) Mechanisms too simple for humans to design

(26:42) The future of non-human design

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.