"He who fights with monsters might take care lest he thereby become a monster. And if you gaze for long into an abyss, the abyss gazes also into you."

- Friedrich Nietzsche

I think it is not an exaggeration to say that many people I know in this community hate the idea of powerful, unaligned AI. They describe it in apocalyptic terms, equate its existence to the death of everything they love, treat every step towards it as evil in its purest form. We as a community are haunted by pictures of paperclip maximisers quietly recycling human civilisation into spools of wire, disneylands without children, Von Neumann probes spreading out from a darkened Earth where humanity chokes to death because the sun has been blacked out by a Dyson sphere. As a community, you would think, we would be the least interested in pressing the make-a-superintelligence button.

But [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

We are opening applications to attend a 5 day agent foundations conference at Carnegie Mellon University. The program will include talks, breakout sessions, and other activities.

Apply here by January 26

Key Information

• March 3-7, 2025
• At Carnegie Mellon in Pittsburgh, PA
• 30-40 attendees
• Apply by January 26

About

Join us for the 2025 Agent Foundations Conference at Carnegie Mellon University, March 3-7. This five-day gathering brings together leading researchers in AI alignment to explore fundamental questions about Agent Foundations.

Topics may include:

• Bounded decision-making and resource-limited reasoning
• Reflective stability and fixed points in agency
• Logical decision theory and updateless decision theory
• Causal vs evidential vs logical decision theory
• Embedded agency
• Natural abstraction hypothesis
• Abstraction boundaries
• Infra-Bayesian learning theory
• Inner alignment and mesa-optimization
• Logical causality
• Multi-level world models
• Game theory and multi-agent systems
• Logical inductors and reflective reasoning
• Foundations of reasoning under uncertainty
• Coordination problems [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

A story I wrote about living through the transition to utopia.

This is the one story that I've put the most time and effort into; it charts a course from the near future all the way to the distant stars.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Tl;dr, Neural networks are deterministic and sometimes even reversible, which causes Shannon information measures to degenerate. But information theory seems useful. How can we square this (if it's possible at all)? The attempts so far in the literature are unsatisfying.

Here is a conceptual question: what is the Right Way to think about information theoretic quantities in neural network contexts?

Example: I've been recently thinking about information bottleneck methods: given some data distribution _P(X, Y)_, it tries to find features _Z_ specified by _P(Z|X)_ that have nice properties like minimality (small _I(X;Z)_) and sufficiency (big _I(Z;Y)_).

But as pointed out in the literature several times, the fact that neural networks implement a deterministic map makes these information theoretic quantities degenerate:

• if Z is a deterministic map of X and they’re both continuous, then I(X;Z) is infinite.
  • Binning / Quantizing them does turn Z into a stochastic function [...]

---

Outline:

(02:28) Treat the weight as stochastic:

(04:23) Use something other than shannon information measures:

---

First published:

Source:

Narrated by TYPE III AUDIO.

Last week some results were released from a 6-week study using AI tutors in Nigeria. Below I summarize the results of that and four other recent studies about AI tutoring (the dates reflect when the study was conducted rather than when papers were published):

1. Summer 2024 — 15–16-year olds in Nigeria
   They had 800 students total. The treatment group studied with GPT-based Microsoft Copilot twice weekly for six weeks, studying English. They were just provided an initial prompt to start chatting—teachers had a minimal “orchestra conductor” role—but they achieved “the equivalent of two years of typical learning in just six weeks.”
2. Spring 2024 — K-12 Title I schools in the South
   They had 1,800 K-12 students in a low-income school district and gave human tutors to both the treatment and control group, though in the treatment group the tutors had access to the “Tutor [...]
   

---

First published:

Source:

Narrated by TYPE III AUDIO.

We at Sentinel are developing an AI-enabled wargaming-tool, grim, to significantly scale up the number of catastrophic scenarios that concerned organizations can explore and to improve emergency response capabilities of, at least, Sentinel.

Table of Contents

1. How AI Improves on the State of the Art
2. Implementation Details, Limitations, and Improvements
3. Learnings So Far
4. Get Involved!

How AI Improves on the State of the Art

In a wargame, a group dives deep into a specific scenario in the hopes of understanding the dynamics of a complex system and understanding weaknesses in responding to hazards in the system. Reality has a surprising amount of detail, so thinking abstractly about the general shapes of issues is insufficient. However, wargames are quite resource intensive to run precisely because they require detail and coordination.

Eli Lifland shared with us some limitations about the exercises his team has run, like at The Curve conference:

[...]

---

Outline:

(00:24) How AI Improves on the State of the Art

(02:34) Implementation Details, Limitations, and Improvements

(04:37) Some Learnings So Far

(08:37) Get Involved!

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 159 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Today's post is in response to the post "Quantum without complications", which I think is a pretty good popular distillation of the basics of quantum mechanics.

For any such distillation, there will be people who say "but you missed X important thing". The limit of appeasing such people is to turn your popular distillation into a 2000-page textbook (and then someone will still complain).

That said, they missed something!

To be fair, the thing they missed isn't included in most undergraduate quantum classes. But it should be.[1]

Or rather, there is something that I wish they told me when I was first learning this stuff and confused out of my mind, since I was a baby [...]

---

Outline:

(02:18) Quantum mechanics 101

(07:36) Upshots

(08:32) Statistical mechanics 101

(12:20) Welcome to the matrix

(13:44) Density matrices

(20:23) So can I brag to people that Ive resolved all the multiverse/decoherence issues now?

(21:05) Who ordered that?

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I think a lot of people have heard so much about internalized prejudice and bias that they think they should ignore any bad vibes they get about a person that they can’t rationally explain.

But if a person gives you a bad feeling, don’t ignore that.

Both I and several others who I know have generally come to regret it if they’ve gotten a bad feeling about somebody and ignored it or rationalized it away.

I’m not saying to endorse prejudice. But my experience is that many types of prejudice feel more obvious. If someone has an accent that I associate with something negative, it's usually pretty obvious to me that it's their accent that I’m reacting to.

Of course, not everyone has the level of reflectivity to make that distinction. But if you have thoughts like “this person gives me a bad vibe but [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TLDR: We're hiring for research & engineering roles across different levels of seniority. Hires will work on applications of singular learning theory to alignment, including developmental interpretability.

About Us

Timaeus' mission is to empower humanity by making breakthrough scientific progress on alignment. Our research focuses on applications of singular learning theory to foundational problems within alignment, such as interpretability (via "developmental interpretability"), out-of-distribution generalization (via "structural generalization"), and the study of inductive biases (via "geometry of program synthesis").

Position Details

• Positions: Research Lead, Research Scientist, Research Engineer.
• Location: Remote. Timaeus will likely be located in either Berkeley or London in the next 6 months, and we intend to sponsor visas for these roles in the future.
• Type: Full-time.
• Start Date: April 2025.
• Salary range: USD70,000 – USD140,000 annually, based on experience and location.
• Deadline: Apply here by 13 February at 11:59 PM GMT.

Roles

We expect familiarity with [...]

---

Outline:

(00:21) About Us

(00:49) Position Details

(01:38) Roles

(01:52) Research Lead (1 FTE)

(02:55) Research Scientist (2 FTE)

(04:08) Research Engineer (1 FTE)

(05:12) Current Research Areas

(05:58) Benefits and Salary

(07:53) Application Process

(09:02) Apply Now

(09:30) Diversity and Inclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.