Part 13 of 12 in the Engineer's Interpretability Sequence.

TL;DR

On May 5, 2024, I made a set of 10 predictions about what the next sparse autoencoder (SAE) paper from Anthropic would and wouldn’t do. Today's new SAE paper from Anthropic was full of brilliant experiments and interesting insights, but it ultimately underperformed my expectations. I am beginning to be concerned that Anthropic's recent approach to interpretability research might be better explained by safety washing than practical safety work.

Think of this post as a curt editorial instead of a technical piece. I hope to revisit my predictions and this post in light of future updates.

Reflecting on predictions

Please see my original post for 10 specific predictions about what today's paper would and wouldn’t accomplish. I think that Anthropic obviously did 1 and 2 [...]

---

Outline:

(00:16) TL;DR

(00:57) Reflecting on predictions

(02:09) A review + thoughts

---

First published:

Source:

Narrated by TYPE III AUDIO.

Dwarkesh Patel recorded a Podcast with John Schulman, cofounder of OpenAI and at the time their head of current model post-training. Transcript here. John's job at the time was to make the current AIs do what OpenAI wanted them to do. That is an important task, but one that employs techniques that their at-the-time head of alignment, Jan Leike, made clear we should not expect to work on future more capable systems. I strongly agree with Leike on that.

Then Sutskever left and Leike resigned, and John Schulman was made the new head of alignment, now charged with what superalignment efforts remain at OpenAI to give us the ability to control future AGIs and ASIs.

This gives us a golden opportunity to assess where his head is at, without him knowing he was about to step into that role.

There is no question that John Schulman [...]

---

Outline:

(01:12) The Big Take

(07:27) The Podcast

(20:27) Reasoning and Capabilities Development

(25:01) Practical Considerations

---

First published:

Source:

Narrated by TYPE III AUDIO.

This post was written by Peli Grietzer, inspired by internal writings by TJ (tushant jha), for AOI[1]. The original post, published on Feb 5, 2024, can be found here: https://ai.objectives.institute/blog/the-problem-with-alignment.

The purpose of our work at the AI Objectives Institute (AOI) is to direct the impact of AI towards human autonomy and human flourishing. In the course of articulating our mission and positioning ourselves -- a young organization -- in the landscape of AI risk orgs, we’ve come to notice what we think are serious conceptual problems with the prevalent vocabulary of ‘AI alignment.’ This essay will discuss some of the major ways in which we think the concept of ‘alignment’ creates bias and confusion, as well as our own search for clarifying concepts.

At AOI, we try to think about AI within the context of [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

The UK and Republic of Korea governments announced that the following organisations have agreed to the Frontier AI Safety Commitments:

• Amazon
• Anthropic
• Cohere
• Google
• G42
• IBM
• Inflection AI
• Meta
• Microsoft
• Mistral AI
• Naver
• OpenAI
• Samsung Electronics
• Technology Innovation Institute
• xAI
• Zhipu.ai

The above organisations, in furtherance of safe and trustworthy AI, undertake to develop and deploy their frontier AI models and systems[1] responsibly, in accordance with the following voluntary commitments, and to demonstrate how they have achieved this by publishing a safety framework focused on severe risks by the upcoming AI Summit in France.

Given the evolving state of the science in this area, the undersigned organisations’ approaches (as detailed in paragraphs I-VIII) to meeting Outcomes 1, 2 and 3 may evolve in the future. In such instances, organisations will provide transparency on this, including their reasons, through public updates.

The above organisations also affirm their commitment [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a quickly-written opinion piece, of what I understand about OpenAI. I first posted it to Facebook, where it had some discussion.

Some arguments that OpenAI is making, simultaneously:

1. OpenAI will likely reach and own transformative AI (useful for attracting talent to work there).
2. OpenAI cares a lot about safety (good for public PR and government regulations).
3. OpenAI isn’t making anything dangerous and is unlikely to do so in the future (good for public PR and government regulations).
4. OpenAI doesn’t need to spend many resources on safety, and implementing safe AI won’t put it at any competitive disadvantage (important for investors who own most of the company).
5. Transformative AI will be incredibly valuable for all of humanity in the long term (for public PR and developers).
6. People at OpenAI have thought long and hard about what will happen, and it will be fine.
7. We can’t [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Last September we published our first Responsible Scaling Policy (RSP) [LW discussion], which focuses on addressing catastrophic safety failures and misuse of frontier models. In adopting this policy, our primary goal is to help turn high-level safety concepts into practical guidelines for fast-moving technical organizations and demonstrate their viability as possible standards. As we operationalize the policy, we expect to learn a great deal and plan to share our findings. This post shares reflections from implementing the policy so far. We are also working on an updated RSP and will share this soon.

We have found having a clearly-articulated policy on catastrophic risks extremely valuable. It has provided a structured framework to clarify our organizational priorities and frame discussions around project timelines, headcount, threat models, and tradeoffs. The process of implementing the policy has also surfaced a range of important questions, projects, and dependencies [...]

---

Outline:

(04:53) Threat Modeling and Evaluations

(12:04) The ASL-3 Standard

(16:26) Assurance Structures

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a linkpost for our two recent papers, produced at Apollo Research:

1. An exploration of using degeneracy in the loss landscape for interpretability https://arxiv.org/abs/2405.10927
2. An empirical test of an interpretability technique based on the loss landscape https://arxiv.org/abs/2405.10928

Not to be confused with Apollo's recent Sparse Dictionary Learning paper.

A key obstacle to mechanistic interpretability is finding the right representation of neural network internals. Optimally, we would like to derive our features from some high-level principle that holds across different architectures and use cases. At a minimum, we know two things:

1. We know that the training loss goes down during training. Thus, the features learned during training must be determined by the loss landscape. We want to use the structure of the loss landscape to identify what the features are and how they are represented.

2. ---

First published:

Source:

Narrated by TYPE III AUDIO.

Previously: OpenAI: Facts From a Weekend, OpenAI: The Battle of the Board, OpenAI: Leaks Confirm the Story, OpenAI: Altman Returns, OpenAI: The Board Expands.

Ilya Sutskever and Jan Leike have left OpenAI. This is almost exactly six months after Altman's temporary firing and The Battle of the Board, the day after the release of GPT-4o, and soon after a number of other recent safety-related OpenAI departures. Many others working on safety have also left recently. This is part of a longstanding pattern at OpenAI.

Jan Leike later offered an explanation for his decision on Twitter. Leike asserts that OpenAI has lost the mission on safety and culturally been increasingly hostile to it. He says the superalignment team was starved for resources, with its public explicit compute commitments dishonored, and that safety has been neglected on a widespread basis, not only superalignment but also including addressing the safety [...]

---

Outline:

(02:11) Table of Contents

(03:23) The Two Departure Announcements

(07:50) Who Else Has Left Recently?

(12:33) Who Else Has Left Overall?

(15:56) Early Reactions to the Departures

(24:16) The Obvious Explanation: Altman

(25:47) Jan Leike Speaks

(30:04) Reactions after Leike's Statement

(33:48) Greg Brockman and Sam Altman Respond to Leike

(38:27) Reactions from Some Folks Unworried About Highly Capable AI

(41:28) Don’t Worry, Be Happy?

(43:56) The Non-Disparagement and NDA Clauses

(51:51) Legality in Practice

(54:01) Implications and Reference Classes

(01:01:21) Altman Responds on Non-Disparagement Clauses

(01:02:27) So, About That Response

(01:10:34) How Bad Is All This?

(01:13:58) Those Who Are Against These Efforts to Prevent AI From Killing Everyone

(01:18:01) What Will Happen Now?

(01:18:30) What Else Might Happen or Needs to Happen Now?

---

First published:

Source:

Narrated by TYPE III AUDIO.

to follow up my philantropic pledge from 2020, i've updated my philanthropy page with 2023 results.

in 2023 my donations funded $44M worth of endpoint grants ($43.2M excluding software development and admin costs) — exceeding my commitment of $23.8M (20k times $1190.03 — the minimum price of ETH in 2023).

---

First published:

Source:

Narrated by TYPE III AUDIO.

I thought that the recently released International Scientific Report on the Safety of Advanced AI seemed like a pretty good summary of the state of the field on AI risks, in addition to being about as close to a statement of expert consensus as we're likely to get at this point. I noticed that each section of the report has a useful "Key Information" bit with a bunch of bullet points summarizing that section.

So for my own use as well as perhaps the use of others, and because I like bullet-point summaries, I've copy-pasted all the "Key Information" lists here.

1 Introduction

[Bullet points taken from the “About this report” part of the Executive Summary]

• This is the interim publication of the first ‘International Scientific Report on the Safety of Advanced AI’. A diverse group of 75 artificial intelligence (AI) experts contributed to this report, including an [...]

---

Outline:

(00:38) 1 Introduction

(02:54) 2 Capabilities

(02:58) 2.1 How does General-Purpose AI gain its capabilities?

(04:05) 2.2 What current general-purpose AI systems are capable of

(05:08) 2.3 Recent trends in capabilities and their drivers

(06:21) 2.4 Capability progress in coming years

(08:19) 3 Methodology to assess and understand general-purpose AI systems

(10:38) 4 Risks

(10:42) 4.1 Malicious use risks

(10:46) 4.1.1 Harm to individuals through fake content

(11:13) 4.1.2 Disinformation and manipulation of public opinion

(11:53) 4.1.3 Cyber offence

(12:30) 4.1.4 Dual use science risks

(13:42) 4.2 Risks from malfunctions

(13:47) 4.2.1 Risks from product functionality issues

(14:29) 4.2.2 Risks from bias and underrepresentation

(15:07) 4.2.3 Loss of control

(16:44) 4.3 Systemic risks

(16:49) 4.3.1 Labour market risks

(17:52) 4.3.2 Global AI divide

(18:34) 4.3.3 Market concentration risks and single points of failure

(19:26) 4.3.4 Risks to the environment

(19:46) 4.3.5 Risks to privacy

(20:25) 4.3.6 Copyright infringement

(21:10) 4.4 Cross-cutting risk factors

(21:15) 4.4.1 Cross-cutting technical risk factors

(22:47) 4.4.2 Cross-cutting societal risk factors

(23:33) 5 Technical approaches to mitigate risks

(23:38) 5.1 Risk management and safety engineering

(24:48) 5.2 Training more trustworthy models

(26:28) 5.3 Monitoring and intervention

(28:00) 5.4 Technical approaches to fairness and representation in general-purpose AI systems

(29:44) 5.5 Privacy methods for general-purpose AI systems

---

First published:

Source:

Narrated by TYPE III AUDIO.