Summary

This post gives my personal take on “AI for epistemics” and how important it might be to work on.

Some background context:

• AI capabilities are advancing rapidly and I think it's important to think ahead and prepare for the possible development of AI that could automate almost all economically relevant tasks that humans can do.[1]

• That kind of AI would have a huge impact on key epistemic processes in our society. (I.e.: It would have a huge impact on how new facts get found, how new research gets done, how new forecasts get made, and how all kinds of information spread through society.)
• I think it's very important for our society to have excellent epistemic processes. (I.e.: For important decisions in our society to be made by people or AI systems who have informed and unbiased beliefs that take into account as much of the available evidence [...]

---

Outline:

(00:04) Summary

(04:36) Structure of the post

(05:33) Previous work

(07:29) Why work on AI for epistemics?

(07:34) Summary

(09:38) Importance

(12:15) Path-dependence

(14:45) To be more concrete

(15:37) Good norms and practices for AI-as-knowledge-producers

(17:12) Good norms and practices for AI-as-communicators

(19:30) Differentially high epistemic capabilities

(27:12) Heuristics for good interventions

(28:01) Direct vs. indirect strategies

(29:24) Indirect value generation

(33:30) On long-lasting differential capability improvements

(36:11) Painting a picture of the future

(41:04) Concrete projects for differentially advancing epistemic capabilities

(43:47) Evals/benchmarks for forecasting (or other ambitious epistemic assistance)

(44:27) Automate forecasting question-generation and -resolution

(45:59) Logistics of past-casting

(50:22) Start efforts inside of AI companies for AI forecasting or other ambitious epistemic assistance

(51:34) Scalable oversight / weak-to-strong-generalization / ELK

(53:53) Experiments on what type of arguments and AI interactions tend to lead humans toward truth vs. mislead them

(56:12) Concluding thoughts

The original text contained 22 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Bart, Michael and Patrick are joint first authors. Research conducted as part of MATS 6.0 in Lee Sharkey and Neel Nanda's streams. Thanks to Mckenna Fitzgerald and Robert Krzyzanowski for their feedback!

TL;DR:

• Sparse Autoencoder (SAE) latents have been shown to typically be monosemantic (i.e. correspond to an interpretable property of the input). It is sometimes implicitly assumed that they are therefore atomic, i.e. simple, irreducible units that make up the model's computation.
• We provide evidence against this assumption by finding sparse, interpretable decompositions of SAE decoder directions into seemingly more atomic latents, e.g. Einstein -> science + famous + German + astronomy + energy + starts with E-
• We do this by training meta-SAEs, an SAE trained to reconstruct the decoder directions of a normal SAE. 
• We argue that, conceptually, there's no reason to expect [...]

---

Outline:

(02:36) Introduction

(07:41) Defining Meta-SAEs

(09:46) Meta-latents form interpretable decompositions of SAE latents

(13:04) Are Meta-Latents different from SAE Latents?

(19:56) Using Meta-SAEs to Interpret Split Features

(24:02) Causally Intervening and Making Targeted Edits with Meta-Latents

(29:03) Discussion

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

Narrated by TYPE III AUDIO.

After the release of Ben Pace's extended interview with me about my views on religion, I felt inspired to publish more of my thinking about religion in a format that's more detailed, compact, and organized. This post is the first publication in my series of intended posts about religion.

Thanks to Ben Pace, Chris Lakin, Richard Ngo, Renshin Lauren Lee, Mark Miller, and Imam Ammar Amonette for their feedback on this post, and thanks to Kaj Sotala, Tomáš Gavenčiak, Paul Colognese, and David Spivak for reviewing earlier versions of this post. Thanks especially to Renshin Lauren Lee and Imam Ammar Amonette for their input on my claims about religion and inner work, and Mark Miller for vetting my claims about predictive processing.

In Waking Up, Sam Harris wrote:[1]

But I now knew that Jesus, the Buddha, Lao Tzu, and the other saints and sages of [...]

---

Outline:

(01:36) “Trapped Priors As A Basic Problem Of Rationality”

(03:49) Active blind spots as second-order trapped priors

(06:17) Inner work ≈ the systematic addressing of trapped priors

(08:33) Religious mystical traditions as time-tested traditions of inner work?

The original text contained 12 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is part of a series of posts/thoughts I've put together as I come of age in the 2020s. Perhaps particularly relevant is the essay I've linked on some of the feelings I have about rationalism, but the whole collection might have material that is generally of interest (incl. my thoughts on why the current AI/LLM development paradigm is harmful and my proposed solution, including prototype code that runs).

To be clear: I'm not coming at this from the perspective of "outsider sees thing bad, points and laughs" - I have read all of HPMOR and been variously active in stuff like AI safety, but it does represent some of the points of tension I have felt come up while in conversation with rationalists/those who are rationalist-adjacent. I hope, if nothing else, there is something of interest to consider here.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I'm aware of good arguments that this scenario isn't inevitable, but it still seems frighteningly likely even if we solve technical alignment.

TL;DR:

1. If we solve alignment, it will probably be used to create AGI that follows human orders.
2. If takeoff is slow-ish, a pivotal act (preventing more AGIs from being developed) will be difficult.
3. If no pivotal act is performed, RSI-capable AGI proliferates. This creates an n-way non-iterated Prisoner's Dilemma where the first to attack, wins.
4. Disaster results.

The first AGIs will probably be aligned to take orders

People in charge of AGI projects like power. And by definition, they like their values somewhat better than the aggregate values of all of humanity. It also seems like there's a pretty strong argument that Instruction-following AGI is easier than value aligned AGI. In the slow-ish takeoff we expect, this alignment target seems to allow for error-correcting alignment [...]

---

Outline:

(00:43) The first AGIs will probably be aligned to take orders

(01:27) The first AGI probably wont perform a pivotal act

(02:30) So RSI-capable AGI may proliferate until a disaster occurs

(03:51) Counterarguments

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Meta: This post is a relatively rough dump of some recent research thoughts; it's not one of our more polished posts, in terms of either clarity or rigor. You’ve been warned.

The Interoperable Semantics post and the Solomonoff Inductor Walks Into A Bar post each tackled the question of how different agents in the same world can coordinate on an ontology, so that language can work at all given only a handful of example usages of each word (similar to e.g. children learning new words). Both use natural latents as a central mathematical tool - one in a Bayesian probabilistic framework, the other in a minimum description length framework. Both focus mainly on nouns, i.e. interoperable-across-minds clusters of “objects” in the environment.

… and the two propose totally different models. In one, the interoperability of cluster labels (i.e. nouns) follows from natural latent conditions over different features of each [...]

---

Outline:

(01:45) The Problem

(03:43) Two Previous Models: Naturality Over Objects vs Features

(07:34) Referents of Adjectives?

(10:49) One Implication: Adjectives Tend To Be Less Natural (Convergent) Than Nouns?

(12:38) How This Fits Into The Broader Gameplan

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

SB 1047 has been amended once more, with both strict improvements and big compromises. I cover the changes, and answer objections to the bill, in my extensive Guide to SB 1047. I follow that up here with reactions to the changes and some thoughts on where the debate goes from here. Ultimately, it is going to come down to one person: California Governor Gavin Newsom.

All of the debates we’re having matter to the extent they influence this one person. If he wants the bill to become law, it almost certainly will become law. If he does not want that, then it won’t become law, they never override a veto and if he makes that intention known then it likely wouldn’t even get to his desk. For now, he's not telling.

Table of Contents

---

Outline:

(00:52) Language Models Offer Mundane Utility

(04:39) Language Models Don’t Offer Mundane Utility

(06:49) Deepfaketown and Botpocalypse Soon

(08:00) The Art of the Jailbreak

(15:34) Get Involved

(16:33) Introducing

(17:44) In Other AI News

(20:27) Quiet Speculations

(27:51) SB 1047: Nancy Pelosi

(35:30) SB 1047: Anthropic

(39:59) SB 1047: Reactions to the Changes

(53:41) SB 1047: Big Picture

(55:45) The Week in Audio

(58:32) Rhetorical Innovation

(59:37) Aligning a Smarter Than Human Intelligence is Difficult

(01:00:55) The Lighter Side

The original text contained 6 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 94 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

This post walks through the math for a theorem. It's intended to be a reference post, which we’ll link back to as-needed from future posts. The question which first motivated this theorem for us was: “Redness of a marker seems like maybe a natural latent over a bunch of parts of the marker, and redness of a car seems like maybe a natural latent over a bunch of parts of the car, but what makes redness of the marker ‘the same as’ redness of the car? How are they both instances of one natural thing, i.e. redness? (or ‘color’?)”. But we’re not going to explain in [...]

---

Outline:

(02:22) Trivial Version

(03:10) Nontrivial Version

(03:40) Redundancy

(05:01) Mediation

---

First published:

Source:

Narrated by TYPE III AUDIO.

Note: This post is co-authored with Stacy Li, a PhD student at Berkeley studying aging biology! Highly appreciate all her help in writing, editing, and fact-checking my understanding!

Introduction

The last time I read about aging research deeply was around 2021. The general impression I was getting was that aging research was increasingly more and more funded (good!). Unfortunately, none of the money led to actionable or useful insights (bad).

Over time, you get slightly burnt out by all the negative news.

After getting a job in biotech, I kept a hazy eye on the subject but mostly tuned out of it entirely. But, especially today, I am curious: how has the aging field progressed in the last few years? Since 2021, what has changed?

In this post, I’ll share a list of immediate questions about the state of affairs in aging research, and the [...]

---

Outline:

(00:31) Introduction

(01:30) Did the therapeutic focus on sirtuins amount to much?

(01:35) Background

(03:55) Answer

(05:48) Have the longevity-focused research institutions/grant programs/startups led to anything significant?

(05:56) Background

(07:37) Answer

(09:22) Has cellular reprogramming yielded anything useful?

(09:40) Background

(16:05) Answer

(20:24) What's the state of biological clocks?

(20:28) Background

(24:14) Answer

The original text contained 2 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.