Recent discussions about artificial intelligence safety have focused heavily on ensuring AI systems remain under human control. While this goal seems laudable on its surface, we should carefully examine whether some proposed safety measures could paradoxically enable rather than prevent dangerous concentrations of power.

The Control Paradox

The fundamental tension lies in how we define "safety." Many current approaches to AI safety focus on making AI systems more controllable and aligned with human values. But this raises a critical question: controllable by whom, and aligned with whose values?

When we develop mechanisms to control AI systems, we are essentially creating tools that could be used by any sufficiently powerful entity - whether that's a government, corporation, or other organization. The very features that make an AI system "safe" in terms of human control could make it a more effective instrument of power consolidation.

Natural Limits on Human Power

Historical [...]

---

Outline:

(00:25) The Control Paradox

(01:05) Natural Limits on Human Power

(02:46) The Human-AI Nexus

(03:58) Alignment as Enabler of Coherent Entities

(04:21) Dynamics of Inevitable Control?

(05:14) The Offensive Advantage

(06:12) The Double Bind of Development

(06:40) Rethinking Our Approach

(08:39) Conclusion

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.

RULESET

A character's class/race/level/items boil down to giving them scores in two separate stats: Speed and Power.

Stat Calculation

Each parameter of a character has an effect on their stats:

• Class: Your class provides base stats as follows:
  • Knight: 2 Speed/12 Power
  • Warrior: 4 Speed/10 Power
  • Ranger: 6 Speed/8 Power
  • Monk: 8 Speed/6 Power
  • Fencer: 10 Speed/4 Power
  • Ninja: 12 Speed/2 Power
• Race: Your race modifies power as follows:
  • Dwarf: -3 Speed, +3 Power
  • Human: No Effect
  • Elf: +3 Speed, -3 Power
• Level: Each level provides +1 [...]

---

Outline:

(00:28) RULESET

(00:39) Stat Calculation

(03:03) Combat

(04:57) STRATEGY

(08:29) BONUS OBJECTIVE

(10:28) LEADERBOARD

(10:59) DATASET GENERATION

(11:42) REFLECTIONS and FEEDBACK REQUEST

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

The big news of the week was the release of a new version of Claude Sonnet 3.5, complete with its ability (for now only through the API) to outright use your computer, if you let it. It's too early to tell how big an upgrade this is otherwise. ChatGPT got some interface tweaks that, while minor, are rather nice, as well.

OpenAI, while losing its Senior Advisor for AGI Readiness, is also in in midst of its attempted transition to a B-corp. The negotiations about who gets what share of that are heating up, so I also wrote about that as The Mask Comes Off: At What Price? My conclusion is that the deal as currently floated would be one of the largest thefts in history, out of the nonprofit, largely on behalf of Microsoft.

The third potentially major story is reporting on a new lawsuit against [...]

---

Outline:

(01:14) Language Models Offer Mundane Utility

(03:53) Language Models Don’t Offer Mundane Utility

(04:32) Deepfaketown and Botpocalypse Soon

(07:10) Character.ai and a Suicide

(12:23) Who and What to Blame?

(18:38) They Took Our Jobs

(19:51) Get Involved

(20:06) Introducing

(21:41) In Other AI News

(22:47) The Mask Comes Off

(27:26) Another One Bites the Dust

(31:30) Wouldn’t You Prefer a Nice Game of Chess

(32:55) Quiet Speculations

(34:54) The Quest for Sane Regulations

(38:10) The Week in Audio

(40:53) Rhetorical Innovation

(50:21) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:50) People Are Worried About AI Killing Everyone

(01:02:46) Other People Are Not As Worried About AI Killing Everyone

(01:04:43) The Lighter Side

The original text contained 15 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

What do I mean by “homegrown”? These projects are:

• Local: Creators have a good track record in the EA or AI Safety community
• Modest: The amount requested is not large; $5k would be meaningful
• Overlooked: Not already backed by large institutional funders like OpenPhil

If you’re a small donor or earn to give, consider giving to projects like these:

1. Feature-length documentary on SB 1047

By Michael Trazzi — $16k raised of $55k

Michael has already recorded interviews with the main characters of SB 1047: sponsors like Scott Wiener and Dan Hendrycks, proponents like Zvi Mowshowitz and Holly Elmore, and opponents like Dean Ball and Timothy B Lee. Now he needs the funding to turn it into a 1-hour feature documentary. This is a rare chance to sponsor a high-quality video narrative, and share it beyond our existing ecosystem. I’ve personally donated $10k towards this, and expect [...]

---

Outline:

(00:35) 1. Feature-length documentary on SB 1047

(01:31) 2. Overview of AI Safety in 2024

(02:32) 3. Podcast series on Effective Altruism's values

(03:06) 4. Sentinel, a foresight and emergency response team

(03:41) 5. Research on co-occurence of sparse autoencoder latents

(04:14) What else have we been up to?

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Matt Levine is the most well-known newslettrist (“Money Stuff”) in the financial industry, having blogged or written since 2011, finding his niche in popularization after stints in Wall Street & law. His commentary is influential, people leak to him, he sometimes interviews major figures (notoriously, Sam Bankman-Fried) or recounts inside information, and a number of phrases like his “laws of insider trading” (specifically, how not to) have gained currency to the point where readers can now do much of the work of sourcing an issue for him.

He is read by hundreds of thousands of readers (including myself)—everyone from shoeshine boy to billionaire. The size of his audience is respectable, but perhaps its most remarkable feature is that many of those readers have nothing to do with the financial industry. Though his newsletter is officially a Bloomberg News newsletter which he simply writes, many of [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Introduction/summary

In my last post, I laid out my picture of what it would even be to solve the alignment problem. In this series of posts, I want to talk about how we might solve it.

To be clear: I don’t think that humans necessarily need to solve the whole problem – at least, not on our own. To the contrary, I think we should be trying hard to get help from sufficiently capable and trusted AIs.[1] And I think various paths that my last post would class as “avoiding” or “handling but not solving” the problem, rather than “solving it,” are important to consider too, especially in the near-term.[2]

I’ll discuss this more in future posts. Still, even if we ultimately need AIs to help us solve the problem, I expect it to be useful to have as direct a grip as we can, now, on [...]

---

Outline:

(00:08) Introduction/summary

(02:22) Summary of the series

(02:35) Part 1 – Ontology

(06:16) What does success look like?

(09:10) Is it even worth talking about the full problem?

(12:53) Preliminaries re: avoiding takeover

(13:15) The ontology I’m using

(15:26) A spectrum of motivation control and option control

(23:02) Incentive structure safety cases

(30:12) Carving up the space of approaches to motivation and option control

(30:42) Internal vs. external variables

(32:50) Inspection vs. intervention

(35:31) AI-assisted improvements

The original text contained 20 footnotes which were omitted from this narration.

The original text contained 13 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

(Some people have messaged me asking me to crosspost my Manifund proposal to Lesswrong for reach. Below, a summary. Full text available here)

Summary of the proposal

We will produce a one-hour feature documentary about SB-1047, which would:

• Serve as a comprehensive reference on the bill's history and implications.
• Present a balanced view of perspectives from both proponents and opponents, bridging ideological divides.
• Contribute to informed discussions about future AI Policy by providing in-depth, impartial analysis.
• Enhance public understanding of AI regulation challenges, fostering more informed public discourse.
• Offer policymakers, researchers, and the public a nuanced resource on various stakeholder perspectives, supporting well-informed decision-making processes.

Funding

• We are seeking $35,000 to [...]

---

Outline:

(00:30) Summary of the proposal

(01:33) Funding

(02:16) Interviewees

(02:26) Proponents of the bill

(03:52) Opponents of the bill

(05:06) Project goals

(08:15) More links

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We’re delighted to announce the winners of the Essay competition on the Automation of Wisdom and Philosophy.

Overview

The competition attracted 90 entries in total (only one of which was obviously just the work of an LLM!), taking a wide variety of angles on the topic. The judges awarded the top four prizes as follows:

• $7,000 to Rudolf Laine for essays on wisdom, amortised optimisation, and AI:

  • Part I: Wisdom, amortised optimisation, and AI

  • Part II: Growth and amortised optimisation

  • Part III: AI effects on amortised optimisation

• $6,000 to Thane Ruthenis for Towards the operationalization of philosophy & wisdom

• $4,000 to Chris Leong for essays on training wise AI systems:

  • An Overview of “Obvious” Approaches to Training Wise AI Advisors

  • Some Preliminary Notes on the Promise of a Wisdom Explosion

• $3,000 to Gabriel Recchia for Should [...]

---

Outline:

(00:22) Overview

(02:19) Judge introductions

(03:55) Top Prize Winners

(04:12) Wisdom, amortised optimisation, and AI — Rudolf Laine — $7,000

(04:44) Summary

(06:07) BS's comments

(06:45) CN's comments

(08:46) DM's comments

(12:13) Towards the operationalization of philosophy and wisdom — Thane Ruthenis — $6,000

(12:21) Summary

(13:59) DM's comments

(17:02) CN's comments

(19:24) BS's comments

(21:38) Essays on training wise AI systems — Chris Leong — $4,000

(22:11) Summary

(23:20) CN's comments

(24:34) BS's comments

(25:11) Should we just be building more datasets? — Gabriel Recchia — $3,000

(25:19) Summary

(26:07) CN's comments

(27:56) BS's comments

(28:44) Runner-up prizes

(28:48) Designing Artificial Wisdom: The Wise Workflow Research Organization — Jordan Arel

(29:21) Summary

(30:31) CN's comments

(30:51) Philosophys Digital Future — Richard Yetter Chappell

(30:56) Summary

(31:45) BS's comments

(32:43) CN's comments

(32:59) Synthetic Socrates and the Philosophers of the Future — Jimmy Alfonso Licon

(33:06) Summary

(33:52) CN's comments

(34:08) BS's comments

(34:28) The Web of Belief — Paal Fredrik S. Kvarberg

(34:34) Summary

(35:57) CN's comments

(36:21) Wise AI support for government decision-making — Ashwin Acharya and Michaelah Gertz-Billingsley

(36:29) Summary

(37:35) CN's comments

(38:04) Machines and moral judgement — Jacob Sparks

(38:10) Summary

(39:02) CN's comments

(40:00) The purpose of philosophical AI will be: to orient ourselves in thinking — Maximilian Noichl

(40:09) Summary

(40:49) CN's comments

(41:12) BS's comments

(41:40) Cross-context deduction: on the capability necessary for LLM-philosophers — Rio Popper and Clem von Stengel

(41:50) Summary

(42:40) CN's comments

(43:40) DM's comments

(46:21) Evolutionary perspectives on AI values — Maria Avramidou

(46:27) Summary

(47:16) CN's comments

(48:05) BS's comments

(48:14) Tentatively against making AIs wise — Oscar Delaney

(48:20) Summary

(49:18) CN's comments

(49:53) Concluding thoughts

(50:21) BS's thoughts

(53:27) DM's thoughts

(55:50) AS's thoughts

(57:35) Acknowledgements

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Many people helped us a great deal in developing the questions and ideas in this post, including people at CHAI, MATS, various other places in Berkeley, and Aether. To all of them: Thank you very much! Any mistakes are our own.

Foundation model agents - systems like AutoGPT and Devin that equip foundations models with planning, memory, tool use, and other affordances to perform autonomous tasks - seem to have immense implications for AI capabilities and safety. As such, I (Rohan) am planning to do foundation model agent safety research.

Following the spirit of an earlier post I wrote, I thought it would be fun and valuable write as many interesting questions as I could about foundation model agent safety. I shared these questions with my collaborators, and Govind wrote a bunch more questions that he is interested in. This post includes questions from both of us. [...]

---

Outline:

(01:14) Rohan

(01:28) Basics and Current Status

(03:16) Chain-of-Thought (CoT) Interpretability

(08:02) Goals

(10:18) Forecasting (Technical and Sociological)

(16:43) Broad Conceptual Safety Questions

(21:50) Miscellaneous

(25:21) Govind

(25:24) OpenAI o1 and other RL CoT Agents

(26:30) Linguistic Drift, Neuralese, and Steganography

(27:32) Agentic Performance

(28:57) Forecasting

---

First published:

Source:

Narrated by TYPE III AUDIO.

Seven Samotsvety forecasters give their probabilities for different possible catastrophes to lead to more than 1M direct deaths in any one year, in any of the next 10 years. This post presents results, starting with a table which presents the probabilities, continuing with a discussion of each of the items, and concluding with some thoughts on relevance for Sentinel's work averting or mitigating incipient calamities.

Some forecasters found some questions in particular much more interesting, so we alternate between presenting the probability of the best model when one such exists, and the geometric mean of the odds when it's a risk that requires more subjective judgment and we have enough judgments to aggregate. Events that could fall into different buckets can resolve more than one bucket. For instance, an AI-invented biological weapon used during WW3 would resolve the “AI”, “biological weapons” and “WW3” buckets.

Summary

---

Outline:

(01:00) Summary

(01:40) War

(03:48) Natural pandemic

(05:39) The unexpected

(07:16) WW3 specifically

(08:33) AI

(10:35) Unintentional biochem

(11:16) Biological weapons

(13:05) Climate change

(14:18) Volcanoes

(21:36) US civil war

(23:16) Financial crisis (e.g., hyperinflation)

(25:29) Solar storm

(30:11) Simulation shutdown

(31:13) Asteroid impact

(32:52) Alien invasion

(33:38) Vacuum decay, spontaneous black holes, doomsday particles, or other weird physics-based catastrophes

(34:24) Conclusion and reflections

(35:34) Acknowledgements

(36:17) Appendix

(36:20) Shortcuts taken

(37:07) Other items we might have forecasted

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.