The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Intro

Anthropic recently released an exciting mini-paper on crosscoders (Lindsey et al.). In this post, we open source a model-diffing crosscoder trained on the middle layer residual stream of the Gemma-2 2B base and IT models, along with code, implementation details / tips, and a replication of the core results in Anthropic's paper.

While Anthropic highlights several potential applications of crosscoders, in this post we focus solely on “model-diffing”. That is, localizing and interpreting a small “diff” between two different models. We think this is a particularly exciting application, because it can let us examine what changed as a model was fine-tuned, which seems likely to capture most safety-relevant circuitry, while leaving out many less relevant capabilities.

In their paper, they find exciting preliminary evidence that crosscoders identify shared sets of features across different models, as well as features specific to each model. While it's still an open question [...]

---

Outline:

(00:08) Intro

(01:19) TLDR;

(02:46) Replicating key results

(04:57) Evaluating sparsity vs reconstruction fidelity to SAEs

(06:53) Implementation details and tips

(09:12) Investigating interpretable latents from different clusters

(11:49) Author Contributions Statement

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A small number of terms are elevated from the pages of literature, up to the Mount Olympus of blog post vernacular. Moloch, as the dark god of failed coordination problems. The Dark Forest, as domains where there's active incentive not to be noticed. And from Ursula K. Le Guin, the Omelas child, a person whose suffering is a counterweight to the joys of others.[1]

This post is an attempt to elevate another of Le Guin's terms to the blog post realm. That term is shifgrethor.

Definition

Shifgrethor hails from The Left Hand of Darkness, Le Guin's most celebrated standalone novel. In the novel, it's an untranslatable alien concept, the sense of honor on the planet Gethen.

But basically, it boils down to an injunction against advice.

In the book, “shifgrethor” comes from “an old word for shadow”. Individuals have shifgrethor, and various slights can offend [...]

---

Outline:

(00:36) Definition

(01:40) Examples

(03:35) Why a new term?

The original text contained 1 footnote which was omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: model-building based on observation, with a few successful unusual predictions. Anecdotal evidence has so far been consistent with the model. This puts it at risk of seeming more compelling than the evidence justifies just yet. Caveat emptor.

Imagine you're a very young child. Around, say, three years old.

You've just done something that really upsets your mother. Maybe you were playing and knocked her glasses off the table and they broke.

Of course you find her reaction uncomfortable. Maybe scary. You're too young to have detailed metacognitive thoughts, but if you could reflect on why you're scared, you wouldn't be confused: you're scared of how she'll react.

She tells you to say you're sorry.

You utter the magic words, hoping that will placate her.

And she narrows her eyes in suspicion.

"You sure don't look sorry. Say it and mean it."

Now you have a serious problem. [...]

---

Outline:

(02:16) Newcomblike self-deception

(06:10) Sketch of a real-world version

(08:43) Possible examples in real life

(12:17) Other solutions to the problem

(12:38) Having power

(14:45) Occlumency

(16:48) Solution space is maybe vast

(17:40) Ending the need for self-deception

(18:21) Welcome self-deception

(19:52) Look away when directed to

(22:59) Hypothesize without checking

(25:50) Does this solve self-deception?

(27:21) Summary

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is a YouTube playlist of recorded lectures on the learning-theoretic AI alignment agenda (LTA) I gave for my MATS scholars of the Winter 2024 cohort, edited by my beloved spouse @Marcus Ogren. I hope these will become a useful resource for anyone who wants to get up to speed on the LTA, complementary to the reading list. In the future, I might record more lectures to expand this list.

Table of Contents

1. Agents and AIXI
2. Hidden rewards and the problem of privilege
3. Compositionality
4. Nonrealizability
5. It's a trap!
6. Traps, continued
7. Traps and frequentist guarantees
8. Game theory and learning theory
9. Hidden rewards
10. Algorithmic Descriptive Agency Measure (ADAM)
11. General reinforcement learning
12. Infra-Bayesianism
13. Learnability
14. Infra-Bandits
15. Newcombian problems
16. Ultradistributions and semi-environments
17. Formalizing Newcombian problems
18. Pseudocausality and a general formulation of Newcombian problems
19. Decision rules and pseudocausality
20. Instrumental reward functions
21. Infra-Bayesian haggling, part 1
22. Infra-Bayesian haggling, part 2
23. Anytime [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Airships are pretty dang cool. Airplanes need a continuous expenditure of energy to stay in the air, but if you just fill a bag with a light gas, you can stay up in the air with no energy expenditure at all. The two lightest gases are hydrogen and helium. Though the hydrogen atom is 4 times lighter than helium, hydrogen gas is made of two hydrogen atoms bonded together, so it's only twice as light as helium gas. The difference in lifting power is even more minor: What matters for lifting power of a gas is the difference between its density and that of ordinary air. Hydrogen and helium are both much lighter than air, so the lifting power of either gas is nearly equal to the density of air.

Hydrogen and helium both have problems as lifting gases. Helium's problem is that it's very expensive. Helium comes [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

We are pleased to announce that the 10th version of the AI Safety Camp is now entering the team member application phase!

We again have a wide range of projects this year, so check them out to see if you or someone you know might be interested in applying to join one of them.

You can find all of the projects and the application form on our website, or directly apply here. The deadline for team member applications is November 17th (Sunday).

Below, we are including the categories and summaries of all the projects that will run in AISC 10.

Stop/Pause AI 

(1) Growing PauseAI

Project Lead: Chris Gerrby

Summary 

This project focuses on creating internal and external guides for PauseAI to increase active membership. The outputs will be used by the team of volunteers with high context and engagement, including the key decision makers.

Activism [...]

---

Outline:

(00:39) Stop/Pause AI

(00:42) (1) Growing PauseAI

(00:50) Summary

(01:56) (2) Grassroots Communication and Lobbying Strategy for PauseAI

(02:06) Summary

(03:08) (3) AI Policy Course: AI's capacity of exploiting existing legal structures and rights

(03:20) Summary

(04:37) (4) Building the Pause Button: A Proposal for AI Compute Governance

(04:49) Summary

(05:28) (5) Stop AI Video Sharing Campaign

(05:38) Summary

(06:29) Evaluate risks from AI

(06:33) (6) Write Blogpost on Simulator Theory

(06:42) Summary

(07:07) (7) Formalize the Hashiness Model of AGI Uncontainability

(07:16) Summary

(08:11) (8) LLMs: Can They Science?

(08:20) Summary

(09:19) (9) Measuring Precursors to Situationally Aware Reward Hacking

(09:28) Summary

(10:11) (10) Develop New Sycophancy Benchmarks

(10:19) Summary

(11:01) (11) Agency Overhang as a Proxy for Sharp Left Turn

(11:11) Summary

(12:22) Mech-Interp

(12:25) (12) Understanding the Reasoning Capabilities of LLMs

(12:35) Summary

(13:47) (13) Mechanistic Interpretability via Learning Differential Equations

(13:57) Summary

(14:32) (14) Towards Understanding Features

(14:41) Summary

(15:55) (15) Towards Ambitious Mechanistic Interpretability II

(16:05) Summary

(17:40) Agent Foundations

(17:44) (16) Understanding Trust

(17:52) Summary

(19:19) (17) Understand Intelligence

(19:28) Summary

(20:32) (18) Applications of Factored Space Models: Agents, Interventions and Efficient Inference

(20:44) Summary

(21:54) Prevent Jailbreaks/Misuse

(21:58) (19) Preventing Adversarial Reward Optimization

(22:08) Summary

(23:37) (20) Evaluating LLM Safety in a Multilingual World

(23:47) Summary

(24:49) (21) Enhancing Multi-Turn Human Jailbreaks Dataset for Improved LLM Defenses

(25:01) Summary

(25:43) Train Aligned/Helper AIs

(25:47) (22) AI Safety Scientist

(25:56) Summary

(26:26) (23) Wise AI Advisers via Imitation Learning

(26:36) Summary

(28:19) (24) iVAIS: Ideally Virtuous AI System with Virtue as its Deep Character

(28:32) Summary

(29:15) (25) Exploring Rudimentary Value Steering Techniques

(29:25) Summary

(31:24) (26) Autostructures – for Research and Policy

(31:34) Summary

(32:04) Other

(32:07) (27) Reinforcement Learning from Recursive Information Market Feedback

(32:18) Summary

(32:37) (28) Explainability through Causality and Elegance

(32:46) Summary

(34:14) (29) Leveraging Neuroscience for AI Safety

(34:23) Summary

(35:17) (30) Scalable Soft Optimization

(35:26) Summary

(36:02) (31) AI Rights for Human Safety

(36:11) Summary

(37:27) (32) Universal Values and Proactive AI Safety

(37:37) Summary

(42:13) Apply Now

---

First published:

Source:

Narrated by TYPE III AUDIO.

[Linkpost. This presents a way to talk to people about AI that hold overly-romanticized views built on fictional evidence.]

MRSA (Methicillin-resistant Staphylococcus aureus) is a type of staph bacteria that's become resistant to many of the antibiotics we use. The original staph bacteria that is the ancestor of MRSA was first identified in 1880, but has likely existed in some form for ages. Humans created MRSA first by gathering many sick people in one location for centuries via hospitals, nursing centers, etc—thus creating a staph utopia—and then subjecting the population to extremely harsh selection pressures by attempting to eradicate them over and over via ever-stronger antibiotics. Through many waves of selection events the most antibiotic-resistant bacteria managed to scrape by just long enough to reproduce, getting ever more resistant with every successive wave.

Eventually we created MRSA, a super-strain of staph that exhibits powers we would’ve been astounded by [...]

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Memorandum on Advancing the United States’ Leadership in Artificial Intelligence; Harnessing Artificial Intelligence to Fulfill National Security Objectives; and Fostering the Safety, Security, and Trustworthiness of Artificial Intelligence

This memorandum fulfills the directive set forth in subsection 4.8 of Executive Order 14110 of October 30, 2023 (Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence). This memorandum provides further direction on appropriately harnessing artificial intelligence (AI) models and AI-enabled technologies in the United States Government, especially in the context of national security systems (NSS), while protecting human rights, civil rights, civil liberties, privacy, and safety in AI-enabled national security activities. A classified annex to this memorandum addresses additional sensitive national security issues, including countering adversary use of AI that poses risks to United States national security.

---

First published:

Source:

Narrated by TYPE III AUDIO.

As artificial intelligence (AI) systems become more advanced, concerns about large-scale risks from misuse or accidents have grown. This report analyzes the technical research into safe AI development being conducted by three leading AI companies: Anthropic, Google DeepMind, and OpenAI.

We define “safe AI development” as developing AI systems that are unlikely to pose large-scale misuse or accident risks. This encompasses a range of technical approaches aimed at ensuring AI systems behave as intended and do not cause unintended harm, even as they are made more capable and autonomous.

We analyzed all papers published by the three companies from January 2022 to July 2024 that were relevant to safe AI development, and categorized the 80 included papers into nine safety approaches. Additionally, we noted two categories representing nascent approaches explored by academia and civil society, but not currently represented in any research papers by these [...]

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.