Audio note: this article contains 292 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Machine learning systems are typically trained to maximize average-case performance. However, this method of training can fail to meaningfully control the probability of tail events that might cause significant harm. For instance, while an artificial intelligence (AI) assistant may be generally safe, it would be catastrophic if it ever suggested an action that resulted in unnecessary large-scale harm.

Current techniques for estimating the probability of tail events are based on finding inputs on which an AI behaves catastrophically. Since the input space is so large, it might be prohibitive to search through it thoroughly enough to detect all potential catastrophic behavior. As a result, these techniques cannot be used to produce AI systems that we are confident [...]

---

Outline:

(03:05) A Toy Scenario

(04:32) Deficiencies of Adversarial Training

(06:26) Deliberate Subversion of Adversarial Training

(08:20) A Possible Approach for Estimating Tail Risk

(10:13) Layer-by-layer Activation Modeling

(13:07) Toy Example: Finitely Supported Distributions

(14:24) Method 1: Gaussian Distribution

(16:03) Method 2: Independent Linear Features

(19:01) Relation to Other Work

(19:05) Formalizing the Presumption of Independence

(20:00) Mechanistic Interpretability

(21:02) Relaxed Adversarial Training

(21:24) Eliciting Latent Knowledge

(22:53) Conclusion

(24:25) Appendices

(24:28) Catastrophe Detectors

(28:41) Issues with Layer-by-layer Activation Modeling

(29:17) Layer-by-layer modeling forgets too much information

(31:42) Probability distributions over activations are too restrictive

(35:08) Distributional Shift

(37:48) Complexity Theoretic Barriers to Accurate Estimates

(38:38) 3-SAT

(40:20) Method 1: Assume clauses are independent

(40:57) Method 2: Condition on the number of true variables

(42:51) Method 3: Linear regression

(44:47) Main hope: estimates competitive with the AI system or the training process

The original text contained 13 footnotes which were omitted from this narration.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

People thinking about the future of AI sometimes talk about a single project ‘getting there first’ — achieving AGI, and leveraging this into a decisive strategic advantage over the rest of the world. That would represent a massive concentration of power.

This post is a one-sided exploration of concerns with such concentration of power. They're conveyed in part via an extended analogy with The Lord of the Rings, in the hope that that could make it easier to keep track of the concerns at a gut level — even if people are ultimately persuaded by arguments for the inevitability or desirability of a single project.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is a little rant I like to give, because it's something I learned on the job that I’ve never seen written up explicitly.

There are a bunch of buzzwords floating around regarding computer technology in an industrial or manufacturing context: “digital transformation”, “the Fourth Industrial Revolution”, “Industrial Internet of Things”.

What do those things really mean?

Do they mean anything at all?

The answer is yes, and what they mean is the process of putting all of a company's data on computers so it can be analyzed.

This is the prerequisite to any kind of “AI” or even basic statistical analysis of that data; before you can start applying your fancy algorithms, you need to get that data in one place, in a tabular format.

Wait, They Haven’t Done That Yet?

---

Outline:

(00:56) Wait, They Haven’t Done That Yet?

(03:28) Why Data Integration Is Hard

(03:56) Data Access Negotiation, AKA Please Let Me Do The Work You Paid Me For

(10:20) Data Cleaning, AKA I Can’t Use This Junk

(14:00) AI is Gated On Data Integration

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 42 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

I believe that the theoretical foundations of the AIXI agent and variations are a surprisingly neglected and high leverage approach to agent foundations research. Though discussion of AIXI is pretty ubiquitous in A.I. safety spaces, underscoring AIXI's usefulness as a model of superintelligence, this is usually limited to poorly justified verbal claims about its behavior which are sometimes questionable or wrong. This includes, in my opinion, a serious exaggeration of AIXI's flaws. For instance, in a recent post I proposed a simple extension of AIXI off-policy that seems to solve the anvil problem in practice - in fact, in my opinion it has never been convincingly argued that the anvil problem would occur for an AIXI approximation. The [...]

---

Outline:

(04:46) Computability Foundations

(06:35) Semimeasure Theory

(08:58) Algorithmic Probability Foundations

(10:58) AIXI Generalizations

(14:40) Scaffolding the Universal Distribution

(18:25) Embedded Versions of AIXI

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: using a simple mixed strategy, LDT can give in to threats, ultimatums, and commitments - while incentivizing cooperation and fair[1] splits instead.

This strategy made it much more intuitive to many people I've talked to that smart agents probably won't do weird everyone's-utility-eating things like threatening each other or participating in commitment races.

1. The Ultimatum game

This part is taken from planecrash[2][3].

You're in the Ultimatum game. You're offered 0-10 dollars. You can accept or reject the offer. If you accept, you get what's offered, and the offerer gets $(10-offer). If you reject, both you and the offerer get nothing.

The simplest strategy that incentivizes fair splits is to accept everything ≥ 5 and reject everything < 5. The offerer can't do better than by offering you 5. If you accepted offers of 1, the offerer that knows this would always offer you 1 and get 9 [...]

---

Outline:

(00:28) 1. The Ultimatum game

(06:17) 2. Threats, commitments, and ultimatums

(06:41) 2.1 Game of Chicken

(07:41) 2.2 Stones

(08:00) 2.3 What if I dont know the other players payoffs?

(08:25) 2.4 What if the other player makes a commitment before I make any decisions?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

[Conflict of interest disclaimer: We are FutureSearch, a company working on AI-powered forecasting and other types of quantitative reasoning. If thin LLM wrappers could achieve superhuman forecasting performance, this would obsolete a lot of our work.]

Widespread, misleading claims about AI forecasting

Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.

These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:

• Ethan Mollick (>200k followers) tweeted the following about the paper Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger et al.:

• A post on Marginal Revolution with the title and abstract [...]

---

Outline:

(00:24) Widespread, misleading claims about AI forecasting

(03:02) What does human-level or superhuman forecasting mean?

(04:08) Red flags for claims to (super)human AI forecasting accuracy

(06:42) Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (Schoenegger et al., 2024)

(09:14) Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024)

(11:10) Reasoning and Tools for Human-Level Forecasting (Hsieh et al., 2024)

(12:50) LLMs Are Superhuman Forecasters (Phan et al., 2024)

(15:19) Takeaways

(16:17) So how good are AI forecasters?

The original text contained 1 footnote which was omitted from this narration.

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Following up on Alpha Fold, DeepMind has moved on to Alpha Proteo. We also got a rather simple prompt that can create a remarkably not-bad superforecaster for at least some classes of medium term events.

We did not get a new best open model, because that turned out to be a scam. And we don’t have Apple Intelligence, because it isn’t ready for prime time. We also got only one very brief mention of AI in the debate I felt compelled to watch.

What about all the apps out there, that we haven’t even tried? It's always weird to get lists of ‘top 50 AI websites and apps’ and notice you haven’t even heard of most of them.

Table of Contents

---

Outline:

(00:44) Language Models Offer Mundane Utility

(03:40) Language Models Don’t Offer Mundane Utility

(05:43) Predictions are Hard Especially About the Future

(12:57) Early Apple Intelligence

(15:27) On Reflection It's a Scam

(21:34) Deepfaketown and Botpocalypse Soon

(23:08) They Took Our Jobs

(28:42) The Time 100 People in AI

(32:11) The Art of the Jailbreak

(32:47) Get Involved

(33:12) Alpha Proteo

(43:14) Introducing

(44:23) In Other AI News

(46:41) Quiet Speculations

(50:40) The Quest for Sane Regulations

(53:12) The Week in Audio

(54:28) Rhetorical Innovation

(55:48) Aligning a Smarter Than Human Intelligence is Difficult

(56:05) People Are Worried About AI Killing Everyone

(58:38) Other People Are Not As Worried About AI Killing Everyone

(59:56) Six Boats and a Helicopter

(01:06:47) The Lighter Side

The original text contained 8 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 76 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Refer to the arXiv preprint for full content. This post is a lighter, 15-minute version.

Abstract

• Existing activation steering methods alter LLM behavior indiscriminately, limiting their practical applicability in settings where selective responses are essential, such as content moderation or domain-specific assistants. 
• We propose Conditional Activation Steering (CAST), which analyzes LLM activation patterns during inference to selectively apply or withhold activation steering based on the input context. 
• Using CAST, one can systematically control LLM behavior with rules like “if input is about hate speech or adult content, then refuse” or “if input is not about legal advice, then refuse.” 
• This allows for selective modification of responses to specific content while maintaining normal responses to other content, all [...]

---

Outline:

(00:25) Abstract

(01:26) Introduction

(04:35) Background

(06:55) Conditional Activation Steering

(12:57) Conditioning Refusal: Selectively Steering on Harmful Prompts

(15:40) Programming Refusal: Logical Composition of Refusal Condition

(22:52) Discussion

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I first learned about cryonics when I read Eliezer and Robin's posts about it on Overcoming Bias years ago. I got cryopilled. Somewhat amazingly to me, I'm now a researcher in this field. So I thought this community might be interested to know that I was one of several co-authors on a paper just published in Frontiers in Medical Technology, titled "Structural brain preservation: a potential bridge to future medical technologies".

In this paper, we propose reframing cryonics as a type of structural brain preservation, focusing on maintaining the brain's physical structure that encodes memories and personality, rather than making the focus about low-temperature storage per se.

We explore what brain structures likely need to be preserved to retain long-term memories and other valued aspects of personal identity. We then review different methods of brain preservation, including cryopreservation, aldehyde-stabilized cryopreservation, fluid preservation, and fixation followed by polymer embedding. The [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.