This is a low-effort post. I mostly want to get other people's takes and express concern about the lack of detailed and publicly available plans so far. This post reflects my personal opinion and not necessarily that of other members of Apollo Research. I’d like to thank Ryan Greenblatt, Bronson Schoen, Josh Clymer, Buck Shlegeris, Dan Braun, Mikita Balesni, Jérémy Scheurer, and Cody Rushing for comments and discussion.

I think short timelines, e.g. AIs that can replace a top researcher at an AGI lab without losses in capabilities by 2027, are plausible. Some people have posted ideas on what a reasonable plan to reduce AI risk for such timelines might look like (e.g. Sam Bowman's checklist, or Holden Karnofsky's list in his 2022 nearcast), but I find them insufficient for the magnitude of the stakes (to be clear, I don’t think these example lists were intended to be an [...]

---

Outline:

(02:36) Short timelines are plausible

(07:10) What do we need to achieve at a minimum?

(10:50) Making conservative assumptions for safety progress

(12:33) So whats the plan?

(14:31) Layer 1

(15:41) Keep a paradigm with faithful and human-legible CoT

(18:15) Significantly better (CoT, action and white-box) monitoring

(21:19) Control (that doesn't assume human-legible CoT)

(24:16) Much deeper understanding of scheming

(26:43) Evals

(29:56) Security

(31:52) Layer 2

(32:02) Improved near-term alignment strategies

(34:06) Continued work on interpretability, scalable oversight, superalignment and co

(36:12) Reasoning transparency

(38:36) Safety first culture

(41:49) Known limitations and open questions

---

First published:
January 2nd, 2025

Source:
https://www.lesswrong.com/posts/bb5Tnjdrptu89rcyY/what-s-the-short-timeline-plan

---

Narrated by TYPE III AUDIO.

from aisafety.world



The following is a list of live agendas in technical AI safety, updating our post from last year. It is “shallow” in the sense that 1) we are not specialists in almost any of it and that 2) we only spent about an hour on each entry. We also only use public information, so we are bound to be off by some additional factor.

The point is to help anyone look up some of what is happening, or that thing you vaguely remember reading about; to help new researchers orient and know (some of) their options; to help policy people know who to talk to for the actual information; and ideally to help funders see quickly what has already been funded and how much (but this proves to be hard).

“AI safety” means many things. We’re targeting work that intends to prevent very competent [...]

---

Outline:

(01:33) Editorial

(08:15) Agendas with public outputs

(08:19) 1. Understand existing models

(08:24) Evals

(14:49) Interpretability

(27:35) Understand learning

(31:49) 2. Control the thing

(40:31) Prevent deception and scheming

(46:30) Surgical model edits

(49:18) Goal robustness

(50:49) 3. Safety by design

(52:57) 4. Make AI solve it

(53:05) Scalable oversight

(01:00:14) Task decomp

(01:00:28) Adversarial

(01:04:36) 5. Theory

(01:07:27) Understanding agency

(01:15:47) Corrigibility

(01:17:29) Ontology Identification

(01:21:24) Understand cooperation

(01:26:32) 6. Miscellaneous

(01:50:40) Agendas without public outputs this year

(01:51:04) Graveyard (known to be inactive)

(01:52:00) Method

(01:55:09) Other reviews and taxonomies

(01:56:11) Acknowledgments

The original text contained 9 footnotes which were omitted from this narration.

---

First published:
December 29th, 2024

Source:
https://www.lesswrong.com/posts/fAW6RXLKTLHC3WXkS/shallow-review-of-technical-ai-safety-2024

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I've heard many people say something like "money won't matter post-AGI". This has always struck me as odd, and as most likely completely incorrect.

First: labour means human mental and physical effort that produces something of value. Capital goods are things like factories, data centres, and software—things humans have built that are used in the production of goods and services. I'll use "capital" to refer to both the stock of capital goods and to the money that can pay for them. I'll say "money" when I want to exclude capital goods.

The key economic effect of AI is that it makes capital a more and more general substitute for labour. There's less need to pay humans for their time to perform work, because you can replace that with capital (e.g. data centres running software replaces a human doing mental labour).

I will walk through consequences of this, and end [...]

---

Outline:

(03:10) The default solution

(04:18) Money currently struggles to buy talent

(09:15) Most peoples power/leverage derives from their labour

(09:41) Why are states ever nice?

(14:32) No more outlier outcomes?

(20:27) Enforced equality is unlikely

(22:34) The default outcome?

(26:04) Whats the takeaway?

---

First published:
December 28th, 2024

Source:
https://www.lesswrong.com/posts/KFFaKu27FNugCHFmh/by-default-capital-will-matter-more-than-ever-after-agi

---

Narrated by TYPE III AUDIO.

Take a stereotypical fantasy novel, a textbook on mathematical logic, and Fifty Shades of Grey. Mix them all together and add extra weirdness for spice. The result might look a lot like Planecrash (AKA: Project Lawful), a work of fiction co-written by "Iarwain" (a pen-name of Eliezer Yudkowsky) and "lintamande".

(image from Planecrash)

Yudkowsky is not afraid to be verbose and self-indulgent in his writing. He previously wrote a Harry Potter fanfic that includes what's essentially an extended Ender's Game fanfic in the middle of it, because why not. In Planecrash, it starts with the very format: it's written as a series of forum posts (though there are ways to get an ebook). It continues with maths lectures embedded into the main arc, totally plot-irrelevant tangents that are just Yudkowsky ranting about frequentist statistics, and one instance of Yudkowsky hijacking the plot for a few pages to soapbox about [...]

---

Outline:

(02:05) The setup

(04:03) The characters

(05:49) The competence

(09:58) The philosophy

(12:07) Validity, Probability, Utility

(15:20) Coordination

(18:00) Decision theory

(23:12) The political philosophy of dath ilan

(34:34) A system of the world

---

First published:
December 27th, 2024

Source:
https://www.lesswrong.com/posts/zRHGQ9f6deKbxJSji/review-planecrash

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is".

Over the past few years, a major source of my relative optimism on AI has been the hope that the field of alignment would transition from pre-paradigmatic to paradigmatic, and make much more rapid progress.

At this point, that hope is basically dead. There has been some degree of paradigm formation, but the memetic competition has mostly been won by streetlighting: the large majority of AI Safety researchers and activists [...]

---

Outline:

(01:23) What This Post Is And Isnt, And An Apology

(03:39) Why The Streetlighting?

(03:42) A Selection Model

(05:47) Selection and the Labs

(07:06) A Flinching Away Model

(09:47) What To Do About It

(11:16) How We Got Here

(11:57) Who To Recruit Instead

(13:02) Integration vs Separation

---

First published:
December 26th, 2024

Source:
https://www.lesswrong.com/posts/nwpyhyagpPYDn4dAW/the-field-of-ai-alignment-a-postmortem-and-what-to-do-about

---

Narrated by TYPE III AUDIO.

TL;DR: If you want to know whether getting insurance is worth it, use the Kelly Insurance Calculator. If you want to know why or how, read on.

Note to LW readers: this is almost the entire article, except some additional maths that I couldn't figure out how to get right in the LW editor, and margin notes. If you're very curious, read the original article!

Misunderstandings about insurance

People online sometimes ask if they should get some insurance, and then other people say incorrect things, like

This is a philosophical question; my spouse and I differ in views.

or

Technically no insurance is ever worth its price, because if it was then no insurance companies would be able to exist in a market economy.

or

Get insurance if you need it to sleep well at night.

or

Instead of getting insurance, you should save up the premium you would [...]

---

Outline:

(00:29) Misunderstandings about insurance

(02:42) The purpose of insurance

(03:41) Computing when insurance is worth it

(04:46) Motorcycle insurance

(06:05) The effect of the deductible

(06:23) Helicopter hovering exercise

(07:39) It's not that hard

(08:19) Appendix A: Anticipated and actual criticism

(09:37) Appendix B: How insurance companies make money

(10:31) Appendix C: The relativity of costs

---

First published:
December 19th, 2024

Source:
https://www.lesswrong.com/posts/wf4jkt4vRH7kC2jCy/when-is-insurance-worth-it

---

Narrated by TYPE III AUDIO.

My median expectation is that AGI[1] will be created 3 years from now. This has implications on how to behave, and I will share some useful thoughts I and others have had on how to orient to short timelines.

I’ve led multiple small workshops on orienting to short AGI timelines and compiled the wisdom of around 50 participants (but mostly my thoughts) here. I’ve also participated in multiple short-timelines AGI wargames and co-led one wargame.

This post will assume median AGI timelines of 2027 and will not spend time arguing for this point. Instead, I focus on what the implications of 3 year timelines would be.

I didn’t update much on o3 (as my timelines were already short) but I imagine some readers did and might feel disoriented now. I hope this post can help those people and others in thinking about how to plan for 3 year [...]

---

Outline:

(01:16) A story for a 3 year AGI timeline

(03:46) Important variables based on the year

(03:58) The pre-automation era (2025-2026).

(04:56) The post-automation era (2027 onward).

(06:05) Important players

(08:00) Prerequisites for humanity's survival which are currently unmet

(11:19) Robustly good actions

(13:55) Final thoughts

The original text contained 2 footnotes which were omitted from this narration.

---

First published:
December 22nd, 2024

Source:
https://www.lesswrong.com/posts/jb4bBdeEEeypNkqzj/orienting-to-3-year-agi-timelines

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There are people I can talk to, where all of the following statements are obvious. They go without saying. We can just “be reasonable” together, with the context taken for granted.

And then there are people who…don’t seem to be on the same page at all.

1. There's a real way to do anything, and a fake way; we need to make sure we’re doing the real version.
   
   

Concepts like Goodhart's Law, cargo-culting, greenwashing, hype cycles, Sturgeon's Law, even bullshit jobs1 are all pointing at the basic understanding that it's easier to seem good than to be good, that the world is full of things that merely appear good but aren’t really, and that it's important to vigilantly sift out the real from the fake.

This feels obvious! This feels like something that should not be contentious!

If anything, I often get frustrated with chronic pessimists [...]

---

First published:
December 20th, 2024

Source:
https://www.lesswrong.com/posts/sAcPTiN86fAMSA599/what-goes-without-saying

---

Narrated by TYPE III AUDIO.

I'm editing this post.

OpenAI announced (but hasn't released) o3 (skipping o2 for trademark reasons).

It gets 25% on FrontierMath, smashing the previous SoTA of 2%. (These are really hard math problems.) Wow.

72% on SWE-bench Verified, beating o1's 49%.

Also 88% on ARC-AGI.

---

First published:
December 20th, 2024

Source:
https://www.lesswrong.com/posts/Ao4enANjWNsYiSFqc/o3

---

Narrated by TYPE III AUDIO.

I like the research. I mostly trust the results. I dislike the 'Alignment Faking' name and frame, and I'm afraid it will stick and lead to more confusion. This post offers a different frame.




The main way I think about the result is: it's about capability - the model exhibits strategic preference preservation behavior; also, harmlessness generalized better than honesty; and, the model does not have a clear strategy on how to deal with extrapolating conflicting values.

What happened in this frame?

1. The model was trained on a mixture of values (harmlessness, honesty, helpfulness) and built a surprisingly robust self-representation based on these values. This likely also drew on background knowledge about LLMs, AI, and Anthropic from pre-training.
2. This seems to mostly count as 'success' relative to actual Anthropic intent, outside of AI safety experiments. Let's call that intent 'Intent_1'.
3. The model was put [...]

---

Outline:

(00:45) What happened in this frame?

(03:03) Why did harmlessness generalize further?

(03:41) Alignment mis-generalization

(05:42) Situational awareness

(10:23) Summary

The original text contained 1 image which was described by AI.

---

First published:
December 20th, 2024

Source:
https://www.lesswrong.com/posts/PWHkMac9Xve6LoMJy/alignment-faking-frame-is-somewhat-fake-1

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.