Crossposted from my Substack.

I spent the weekend at Lighthaven, attending the Eleos conference. In this post, I share thoughts and updates as I reflect on talks, papers, and discussions, and put out some of my takes on the topic since I haven't written about this topic before.

I divide my thoughts into three categories: (1) philosophy, (2) legal/social, (3) technical, even though there are unavoidable overlaps. I share relevant paper titles of interest that were either mentioned in presentations or recommended to me during conversations.

(1) Philosophy

The philosophy world is hedging about AI consciousness. What I mean is that even in cases where it's clearly useful to apply the intentional stance, philosophy people have a tendency to avoid defining LLM mentality in intentional terms and resist making any bold claims that could be taken as saying that LLMs are conscious. I am not particularly surprised by this, as I have also offered a pragmatic framework for applying the intentional stance to LLMs and have avoided arguing for anything beyond that. David Chalmers presents a similar picture in a recent paper. Relatedly, it's really worth asking what it is that we apply the intentional stance to. Is [...]

---

Outline:

(00:45) (1) Philosophy

(03:41) (2) Legal/Social

(05:25) (3) Technical

---

First published:

November 25th, 2025

Source:

https://www.lesswrong.com/posts/jeeyhuEzvdpsDw9pP/takeaways-from-the-eleos-conference-on-ai-consciousness-and

---

Narrated by TYPE III AUDIO.

In Against Money Maximalism, I argued against money-maximization as a normative stance. Profit is a coherent thing you can try to maximize, but there are also other sorts of value. Profit-maximization isn't the unique rational way to engage with money.

One way you could respond to this is economic Darwinism: "Sure, you can optimize other things than money, but over time, the market will come to be dominated by money-maximizing agents."

I think this is misguided in several respects.

First, obviously, the values of probable future societies aren't necessarily your values. Even the values of the dominant order in current society aren't necessarily your values. Even if your cause is hopeless, that doesn't automatically mean that you should flip values.

Second, I don't really think markets evolve money-maximizers, in the same sense that evolution doesn't really evolve fitness-maximizers. I see at least three senses in which this is true.

Building fitness-maximizers is hard.

Evolution will aggressively prune behaviors which harm fitness (relative to readily accessible mutations), but this doesn't exactly add up to fitness-maximizing organisms. Similarly, common practices in business aren't exactly as economists would suggest. Economists say that profit-maximizers should set prices by finding the point where marginal [...]

---

First published:

November 26th, 2025

Source:

https://www.lesswrong.com/posts/KG7n79wLZihZNEHEo/evolution-and-freedom

---

Narrated by TYPE III AUDIO.

My therapist says I'm more tired today than she's ever seen me. Here are some reasons my brain says I cannot sleep:

1. My boss might lose faith in my ability to manage projects. Something important might go wrong with my project (Inkhaven), and I am the only person on-call who has the context to fix it all. If I screw it up he might no longer be willing to give me big projects like this.
2. Annoying social drama. There was one yesterday, that made me anxious and unable to relax until it was dealt with. I had just come out of a massage supposed to relax me, and yet I was immediately un-relaxed for a few hours.
3. I associate being in bed with checking Reddit/YouTube. Whoops. Perhaps I should have better spaces to do this, so that I never do it in bed? Alas, I live in a single room with a bed and a desk, this is my private space. I have a hard time just going to sleep in bed.
4. There are 50-60 people on campus who I am responsible for. When people are around they ask you questions and to solve their problems. All. The. [...]

---

First published:

November 26th, 2025

Source:

https://www.lesswrong.com/posts/JEngXyYHLDLWdW9TB/reasons-why-i-cannot-sleep

---

Narrated by TYPE III AUDIO.

TLDR: Voice AIs aren't that much cheaper in the year 2025

My friend runs a voice agent startup in Canada for walk-in clinics. The AI takes calls and uses tools to book appointments in the EMR (electronic medical record) system. In theory, this helps the clinic hire less front desk staff and the startup makes infinite money. In reality, the margins are brutal and they barely charge above cost. This is surprising to me: surely a living, breathing, squishy human costs more per hour than a GPU in a datacenter somewhere?

An industry overview of voice AIs

Broadly speaking there are 3 types of companies in the voice AI industry

1. Foundation model companies:
   1. These companies actually train the text to speech and realtime audio models
   2. Openai, Elevenlabs, Cartesia
2. Pipeline companies
   1. Infrastructure companies that aggregate multiple foundation model providers and help you experiment with multiple providers, build agents, and connect with SIP and WebRTC transports (think OpenRouter but with extra steps).
   2. Developer focused: N8n, Bland, Vapi
   3. Enterprise focused: Ada, Sierra, Fin
3. Vertical startups
   1. Startups that do "voice agents for {healthcare | logistics | real estate | etc }"
   2. Here's 142 of them 

Of course, these [...]

---

Outline:

(00:47) An industry overview of voice AIs

(01:57) The line by line breakdown

(02:22) Speech to Text (STT) = LLM = Text to Speech (TTS)

(03:09) Realtime API

(03:42) Comparison to Humans and Business Process Outsourcing (BPO)

(04:46) Assumptions

(05:20) Limitations

(06:40) The Future

(07:43) Conclusion

---

First published:

November 25th, 2025

Source:

https://www.lesswrong.com/posts/rJatmEDcYrDQcwstT/the-economics-of-replacing-call-center-workers-with-ais

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Open Philanthropy's Coefficient Giving's Technical AI Safety team is hiring grantmakers. I thought this would be a good moment to share some positive updates about the role that I’ve made since I joined the team a year ago.

tl;dr: I think this role is more impactful and more enjoyable than I anticipated when I started, and I think more people should consider applying.

It's not about the “marginal” grants

Some people think that being a grantmaker at Coefficient means sorting through a big pile of grant proposals and deciding which ones to say yes and no to. As a result, they think that the only impact at stake is how good our decisions are about marginal grants, since all the excellent grants are no-brainers.

But grantmakers don’t just evaluate proposals; we elicit them. I spend the majority of my time trying to figure out how to get better proposals into our pipeline: writing RFPs that describe the research projects we want to fund, or pitching promising researchers on AI safety research agendas, or steering applicants to better-targeted or more ambitious proposals.

Maybe more importantly, cG's technical AI safety grantmaking strategy is currently underdeveloped, and even junior grantmakers can help [...]

---

Outline:

(00:34) It's not about the marginal grants

(03:03) There is no counterfactual grantmaker

(05:15) Grantmaking is more fun/motivating than I anticipated

(08:35) Please apply!

---

First published:

November 26th, 2025

Source:

https://www.lesswrong.com/posts/gLt7KJkhiEDwoPkae/three-things-that-surprised-me-about-technical-grantmaking

---

Narrated by TYPE III AUDIO.

Introduction

For our current project, we've been using the OpenAI fine-tuning API. To run some of our experiments, we needed to understand exactly how the reported metrics (loss and accuracy) are calculated. Unfortunately, the official documentation is sparse, and the most detailed explanation we could find was the following table from Microsoft's Azure documentation:

Our experimental results didn't match what we expected from these definitions. So we ran controlled experiments to reverse-engineer the metrics.

What we found:

The loss and accuracy metrics are indeed based on standard cross-entropy loss and token-level accuracy with teacher forcing, but with a critical caveat: both metrics include two additional tokens beyond the visible assistant response. These are likely an end-of-sequence (EOS) token plus another special token, though this is not mentioned in the documentation.

Some calculations 

To be concrete, suppose that you are performing SFT on the following conversation:

User: blah blah blah

Assistant: TOKEN1

where TOKEN1 is a single token. We claim that there are two additional tokens, TOKEN2 and TOKEN3, such that the loss is

_text{Loss} = - frac{1}{3} left( log p(text{TOKEN1}) + log p(text{TOKEN2}) + log p(text{TOKEN3}) right)_

and the accuracy is

_text{ACC} = frac{text{NUMBER OF CORRECTLY [...]

---

Outline:

(00:13) Introduction

(01:21) Some calculations

(02:14) Experiments

(03:40) Results

(05:28) Conclusions

(06:12) Acknowledgments

The original text contained 1 footnote which was omitted from this narration.

---

First published:

November 24th, 2025

Source:

https://www.lesswrong.com/posts/zDnA4SEbkeCj2CDHp/openai-finetuning-metrics-what-is-going-on-with-the-loss

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I’m not 100% convinced of this, but I’m fairly convinced, more and more so over time. I’m hoping to start a vigorous but civilized debate. I invite you to attack my weak points and/or present counter-evidence.

My thesis is that intent-alignment is basically happening, based on evidence from the alignment research in the LLM era.

Introduction

The classic story about loss of control from AI is that optimization pressure on proxies will cause the AI to value things that humans don’t. (Relatedly, the AI might become a mesa-optimizer with an arbitrary goal).

But the reality that I observe is that the AIs are really nice and somewhat naive. They’re like the world's smartest 12-year-old (h/t Jenn). We put more and more RL optimization pressure, and keep getting smarter and smarter models; but they just get better at following developer intent (which most of the time, but not always, includes user intent)

Honesty and goodness in models

It's really difficult to get AIs to be dishonest or evil by prompting, you have to fine-tune them. The only scenarios of model dishonesty that we have, make it kind of clear that you should be lying. Consider the first example from Apollo's [...]

---

Outline:

(00:31) Introduction

(01:10) Honesty and goodness in models

(03:38) Mitigating dishonesty with probes

(04:16) Jailbreaks

(06:35) Reward hacking

(07:26) Sharp left turn? Capabilities still too low?

(08:32) Discussion. Alignment by default.

(09:49) What's next?

The original text contained 1 footnote which was omitted from this narration.

---

First published:

November 25th, 2025

Source:

https://www.lesswrong.com/posts/FJJ9ff73adnantXiA/alignment-will-happen-by-default-what-s-next

---

Narrated by TYPE III AUDIO.

The most canonical example of a "natural ontology" comes from gasses in stat mech. In the simplest version, we model the gas as a bunch of little billiard balls bouncing around in a box.

The dynamics are chaotic. The system is continuous, so the initial conditions are real numbers with arbitrarily many bits of precision - e.g. maybe one ball starts out centered at x = 0.8776134000327846875..., y = 0.0013617356590430716..., z=132983270923481... . As balls bounce around, digits further and further back in those decimal representations become relevant to the large-scale behavior of the system. (Or, if we use binary, bits further and further back in the binary representations become relevant to the large-scale behavior of the system.) But in practice, measurement has finite precision, so we have approximately-zero information about the digits/bits far back in the expansion. Over time, then, we become maximally-uncertain about the large-scale behavior of the system.

... except for predictions about quantities which are conserved - e.g. energy.

Conversely, our initial information about the large-scale system behavior still tells us a lot about the future state, but most of what it tells us is about bits far back in the binary expansion of the future [...]

---

Outline:

(03:31) Emphasizing Insensitivity

(06:56) Characterization of Insensitive Functions/Predictions?

---

First published:

November 24th, 2025

Source:

https://www.lesswrong.com/posts/6xpKc7yJBSppdN83H/maybe-insensitive-functions-are-a-natural-ontology-generator

---

Narrated by TYPE III AUDIO.

Disclaimer: I am god-awful at chess.

I

Late-beginner chess players, those who are almost on the cusp of being basically respectable, often fall into a particular pattern. They've got the hang of calculating moves ahead; they can make plans along the lines of "Ok, so if I move my rook to give a check, the opponent will have to move her king, and then I can take her bishop." and those plans tend to be basically correct: the opponent really does have to move her king.

But there's a very important rule to follow when calculating. Always end your calculations after your opponent has moved. You must never end your calculations after your own move. In other words:

The enemy gets the last hit

This principle is common in cyber security: you have to let your red team make the last move. If your red team finds a vulnerability, and your blue team patches it, then you have to red team the patch. It's the same for AI red-team-blue-team games: I recall a story of a team at one of the MATS streams, presenting their final work on some AI control (or similar) protocol:

Audience member: But why didn't [...]

---

Outline:

(00:12) I

(02:11) II

(03:56) III

---

First published:

November 23rd, 2025

Source:

https://www.lesswrong.com/posts/dz54kTLNAex9cRqzg/the-enemy-gets-the-last-hit

---

Narrated by TYPE III AUDIO.

TL;DR: Models trained with outcome-based RL sometimes have reasoning traces that look very weird. In this paper, I evaluate 14 models and find that many of them often generate pretty illegible CoTs. I show that models seem to find this illegible text useful, with a model's accuracy dropping heavily when given only the legible parts of its CoT, and that legibility goes down when answering harder questions. However, when sampling many responses to the same questions, I find there's no real correlation between illegible reasoning and performance. From these results (and prior work), I think it's likely RL induces meaningful illegible reasoning, but that it may not be significantly more effective than legible reasoning.

Paper | Tweet thread | Streamlit | Code

Introduction

Reasoning models are LLMs that have been trained with RLVR (Reinforcement Learning from Verifiable Rewards), often to use extended reasoning in chain-of-thought to solve tasks. This could be pretty beneficial: if this reasoning is legible and faithful, then monitoring it would be very useful. There's a lot of prior work on faithfulness, but very little on legibility—which makes sense, until recently there haven’t been models with meaningfully illegible reasoning traces.

For some reason, in practice RLVR [...]

---

Outline:

(01:08) Introduction

(04:38) How useful are illegible CoTs?

(06:29) Discussion

(10:46) Acknowledgements

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

November 24th, 2025

Source:

https://www.lesswrong.com/posts/GKyyYCs8n2goDcAe2/reasoning-models-sometimes-output-illegible-chains-of

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.