Previously: On the Podcast, Quotes from the Paper

This is a post in three parts.

The first part is my attempt to condense Leopold Aschenbrenner's paper and model into its load bearing elements and core logic and dependencies.

Two versions here, a long version that attempts to compress with minimal loss, and a short version that gives the gist.

The second part goes over where I agree and disagree, and briefly explains why.

The third part is the summary of other people's reactions and related discussions, which will also include my own perspectives on related issues.

My goal is often to ask ‘what if.’ There is a lot I disagree with. For each subquestion, what would I think here, if the rest was accurate, or a lot of it was accurate?

Summary of Biggest Agreements and Disagreements

I had Leopold review [...]

---

Outline:

(00:54) Summary of Biggest Agreements and Disagreements

(04:58) Decision Theory is Important

(06:38) Part 1: Leopold's Model and Its Implications

(06:44) The Long Version

(17:03) The Short Version

(19:04) Which Assumptions Are How Load Bearing in This Model?

(28:40) Part 2: Where I Agree and Disagree

(56:46) Part 3: Reactions of Others

(56:51) The Basics

(01:00:37) A Clarification from Eliezer Yudkowsky

(01:05:56) Children of the Matrix

(01:10:52) Aligning a Smarter Than Human Intelligence is Difficult

(01:13:28) The Sacred Timeline

(01:17:51) The Need to Update

(01:20:17) Open Models and Insights Can Be Copied

(01:21:38) You Might Not Be Paranoid If They’re Really Out to Get You

(01:24:12) We Are All There Is

(01:25:39) The Inevitable Conflict

(01:38:50) There Are Only Least Bad Options

(01:40:24) A Really Big Deal

(01:41:53) What Gives You the Right?

(01:43:34) Random Other Thoughts

---

First published:

Source:

Narrated by TYPE III AUDIO.

The big news this week was Apple Intelligence being integrated deeply into all their products. Beyond that, we had a modestly better than expected debate over the new version of SB 1047, and the usual tons of stuff in the background. I got to pay down some writing debt.

The bad news is, oh no, I have been called for Jury Duty. The first day or two I can catch up on podcasts or pure reading, but after that it will start to hurt. Wish me luck.

Table of Contents

AiPhone covers the announcement of Apple Intelligence. Apple's products are getting device-wide integration of their own AI in a way they say preserves privacy, with access to ChatGPT via explicit approval for the heaviest requests. A late update: OpenAI is providing this service for free as per Bloomberg.

I offered Quotes from Leopold Aschenbrenner's Situational [...]

---

Outline:

(00:38) Language Models Offer Mundane Utility

(01:46) Language Models Don’t Offer Mundane Utility

(07:49) Fun with Image Generation

(10:09) Copyright Confrontation

(11:38) Deepfaketown and Botpocalypse Soon

(15:20) They Took Our Jobs

(16:33) Someone Explains it All

(19:20) The Art of the Jailbreak

(22:39) Get Involved

(22:46) Introducing

(25:18) In Other AI News

(27:56) Quiet Speculations

(32:17) I Spy With My AI

(34:40) Pick Up the Phone

(35:06) Lying to the White House, Senate and House of Lords

(39:48) The Quest for Sane Regulations

(43:52) More Reasonable SB 1047 Reactions

(51:12) Less Reasonable SB 1047 Reactions

(56:24) That's Not a Good Idea

(56:46) With Friends Like These

(58:13) The Week in Audio

(01:01:22) Rhetorical Innovation

(01:02:32) Mistakes Were Made

(01:03:36) The Sacred Timeline

(01:08:17) Coordination is Hard

(01:13:33) Aligning a Smarter Than Human Intelligence is Difficult

(01:19:07) People Are Worried About AI Killing Everyone

(01:28:48) Other People Are Not As Worried About AI Killing Everyone

(01:29:58) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

Today, Retired U.S. Army General Paul M. Nakasone has joined our Board of Directors. A leading expert in cybersecurity, Nakasone's appointment reflects OpenAI's commitment to safety and security, and underscores the growing significance of cybersecurity as the impact of AI technology continues to grow.

As a first priority, Nakasone will join the Board's Safety and Security Committee, which is responsible for making recommendations to the full Board on critical safety and security decisions for all OpenAI projects and operations.

Whether this was influenced by Aschenbrenner's situational-awareness.ai or not, it's welcome to see OpenAI emphasizing the importance of security. Even with this appointment it's unclear how deeply OpenAI leadership / staff have internalized a security mindset.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Research Status: Written & researched quickly. I think the key point is fairly simple and obvious. I relied on Claude to help with rewriting.

There's been a growing interest in predicting when various products or jobs will be "fully automated." Will we soon have popular movies, books, or even CEOs that are entirely AI-generated?

Some very quickly-found links:
- https://manifold.markets/ChaseStevens/will-an-aigenerated-paper-be-accept
- https://manifold.markets/GabeGarboden/will-aigenerated-art-win-a-major-tr
- https://manifold.markets/probajoelistic/-by-2026-will-any-fully-aigenerated
- https://www.forbes.com/sites/sherzododilov/2024/01/11/can-ai-become-your-next-ceo/

It's an intriguing question, but I believe its definition is slippier than some realize. Here's why.

First, the idea of "full automation" on complex tasks (movies, books, CEO duties) is somewhat of a false dichotomy. In practice, automation exists on a spectrum, with diminishing returns at the extreme. Consider self-driving cars. Even the most advanced autonomous vehicles still have remote human monitors who can intervene if necessary. This human involvement might be significant at first (one intervention for every few miles, in the [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

We have written a paper on sandbagging for which we present the abstract and brief results in this post. See the paper for more details. Tweet thread here.

Trustworthy capability evaluations are crucial for ensuring the safety of AI systems, and are becoming a key component of AI regulation. However, the developers of an AI system, or the AI system itself, may have incentives for evaluations to understate the AI's actual capability. These conflicting interests lead to the problem of sandbagging – which we define as strategic underperformance on an evaluation. In this paper we assess [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

microwave drilling startups

I've seen a bunch of articles about startups trying to do microwave drilling of rock for geothermal energy. Multiple people have asked me about Quaise Energy. (Here's a popular video.) I'm tired of hearing about them, so I'm writing this post to explain some of the reasons why their idea is impractical.

vaporized rock condenses

When rock is vaporized, that rock vapor doesn't just disappear. What happens to it? The answer is, it would quickly condense on the hole wall and pipe.

Initially, a lot of people working on microwave drilling didn't even think about that. Once they did, they decided the solution was to use compressed air to condense the rock and blow the rock particles out. But as anyone familiar with drilling would know, that introduces new problems.

air pressure

Current drilling sometimes uses [...]

---

Outline:

(00:07) microwave drilling startups

(00:29) vaporized rock condenses

(01:01) air pressure

(02:31) energy payback

(02:35) energy usage

(03:12) thermal conductivity

(04:02) payback period

(04:40) some other problems

(04:44) waveguide losses

(05:12) reverse heat transfer

(05:32) hole collapse

(05:48) one thing at a time

(06:29) solar power exists

(06:46) this is only an example

---

First published:

Source:

Narrated by TYPE III AUDIO.

This payout report covers the Long-Term Future Fund's grantmaking from May 1 2023 to March 31 2024 (11 months). It follows our previous April 2023 payout report.

• Total funding recommended: $6,290,550
• Total funding paid out: $5,363,105
• Number of grants paid out: 141
• Acceptance rate (excluding desk rejections): 159/672 = 23.7%
• Acceptance rate (including desk rejections): 159/825 = 19.3%
• Report authors: Linchuan Zhang (primary author), Caleb Parikh (fund chair), Oliver Habryka, Lawrence Chan, Clara Collier, Daniel Eth, Lauro Langosco, Thomas Larsen, Eli Lifland

25 of our grantees, who received a total of $790,251, requested that our public reports for their grants are anonymized (the table below includes those grants). 13 grantees, who received a total of $529, 819, requested that we not include public reports for their grants. You can read our policy on public reporting here.

We referred at least 2 grants to other funders for evaluation.

Highlighted Grants

(The [...]

---

Outline:

(01:44) Highlighted Grants

(20:26) Other updates

(22:08) Other writings

(26:50) Appendix

(26:53) Other Grants We Made During This Time Period

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Apple was for a while rumored to be planning launch for iPhone of AI assisted emails, texts, summaries and so on including via Siri, to be announced at WWDC 24.

It's happening. Apple's keynote announced the anticipated partnership with OpenAI.

The bottom line is that this is Siri as the AI assistant with full access to everything on your phone, with relatively strong privacy protections. Mostly it is done on device, the rest via ‘private cloud compute.’ The catch is that when they need the best they call out for OpenAI, but they do check with you first explicitly each time, OpenAI promises not to retain data and they hide your queries, unless you choose to link up your account.

If the new AI is good enough and safe enough then this is pretty great. If Google doesn’t get its act together reasonably soon to deliver [...]

---

Outline:

(01:30) AiPhone

(04:00) Privacy

(05:27) Practical Magic

(07:38) Dance With the Devil

(08:30) Does It Work?

(10:54) Do You Dare?

(11:59) Who Pays Who?

(13:37) AiPhone Fans

(19:36) No AiPhone

(23:54) In Other Apple News

---

First published:

Source:

Narrated by TYPE III AUDIO.

Yesterday I obtained Anthropic's[1] Certificate of Incorporation, and its past versions, from the State of Delaware. I don't recommend reading it.[2] This post is about what the CoI tells us about Anthropic's Long-Term Benefit Trust (context: Maybe Anthropic's Long-Term Benefit Trust is powerless).

Tl;dr: the only new information of moderate importance is the voting thresholds necessary to modify Trust stuff. My concerns all still stand in some form. Absence of badness is a small positive update.

Anthropic has vaguely described stockholders' power over the Trust:

a series of "failsafe" provisions . . . allow changes to the Trust and its powers without the consent of the Trustees if sufficiently large supermajorities of the stockholders agree. The required supermajorities increase as the Trust's power phases in

The CoI has details: amending the CoI to modify the Trust requires a vote reaching the "Transfer Approval Threshold," defined as:

(1) prior to [...]

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

tl;dr

LessWrong now has a Subscribed tab (next to the Latest tab and Enriched tab[1]). You can now "follow" users, which means their posts and comments will show up in your Subscribed tab.

You can follow users via multiple methods:

1. Using the widget on the Subscribed tab:

2. You can follow people from their user profile:

3. You can follow people using the user tooltip that comes up when you hover on their username.

The full brief

A while back...notifiy on all comments. I wanted a feed.

Feed has two main advantages:

1. Can easily start to read content of post before clicking.
2. Puts comments on even footing as posts.

• This is a new "subscription type". We migrated existing to make it easier. Assume you want. Can unsubscribe
   
• High SNR vs too insular.
  • showing responses helpful

• Open problem:
  • how [...]

---

Outline:

(00:03) tl;dr

(00:43) The full brief

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.