(Audio version here, or search for "Joe Carlsmith Audio" on your podcast app.)

“There comes a moment when the children who have been playing at burglars hush suddenly: was that a real footstep in the hall?”

- C.S. Lewis

1. Introduction

Sometimes, my thinking feels more “real” to me; and sometimes, it feels more “fake.” I want to do the real version, so I want to understand this spectrum better. This essay offers some reflections.

I give a bunch of examples of this “fake vs. real” spectrum below -- in AI, philosophy, competitive debate, everyday life, and religion. My current sense is that it brings together a cluster of related dimensions, namely:

• Map vs. world: Is my mind directed at an abstraction, or it is trying to see past its model to the world beyond?
• Hollow vs. solid: Am I [...]

---

Outline:

(00:30) 1. Introduction

(03:21) 2. Caveats

(05:42) 3. Examples

(06:18) 3.1 AI

(11:54) 3.2 Philosophy

(15:41) 3.3 Competitive debate

(18:11) 3.4 Everyday life

(21:51) 3.5 Lewis on the living God

(25:57) 4. Why does this matter?

(26:33) 4.1 Spiritual stuff

(29:51) 4.2 The telos of thinking

(34:24) 5. How do we do real thinking?

(35:17) 5.1 Going slow

(36:36) 5.2 Following curiosity and aliveness

(37:52) 5.3 Staying in tune with your why

(39:21) 5.4 Tethering your concepts

(42:21) 5.5 Arguments are lenses on the world

(48:12) 5.6 Helplessness about the truth

(50:07) 5.7 Just actually imagining different ways the world could be

(52:41) 5.8 Being wrong/right in both directions

(54:07) 5.9 What would the future people think?

(59:08) 5.10 No bullshit

(01:00:01) 5.11 Real scientists, real philosophers

(01:10:49) 6. Staying awake

The original text contained 40 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

Several developments over the past few months should cause you to re-evaluate what you are doing. These include:

1. Updates toward short timelines
2. The Trump presidency
3. The o1 (inference-time compute scaling) paradigm
4. Deepseek
5. Stargate/AI datacenter spending
6. Increased internal deployment
7. Absence of AI x-risk/safety considerations in mainstream AI discourse

Taken together, these are enough to render many existing AI governance strategies obsolete (and probably some technical safety strategies too). There's a good chance we're entering crunch time and that should absolutely affect your theory of change and what you plan to work on.

In this piece I try to give a quick summary of these developments and think through the broader implications these have for AI safety. At the end of the piece I give some quick initial thoughts on how these developments affect what safety-concerned folks should be prioritizing. These are early days and I expect many of [...]

---

Outline:

(00:06) Introduction

(01:21) Implications of recent developments

(01:25) Updates toward short timelines

(04:26) The Trump Presidency

(07:34) The o1 paradigm

(09:23) Deepseek

(12:08) Stargate/AI data center spending

(13:11) Increased internal deployment

(15:43) Absence of AI x-risk/safety considerations in mainstream AI discourse

(17:13) Implications for strategic priorities

(17:18) Broader implications for US-China competition

(19:33) What seems less likely to work?

(20:56) What should people concerned about AI safety do now?

(24:01) Acknowledgements

The original text contained 13 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Audio note: this article contains 88 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

(This post draws heavily on earlier writing co-authored with Jesse Clifton, but he's not listed as an author since he hasn’t reviewed this version in detail.)

Should we always be able to say whether one outcome is more likely, less likely, or exactly as likely as another? Or should we sometimes suspend judgment and say “none of the above”, that the answer is indeterminate?

Indeterminate beliefs (often modeled with imprecise probabilities)[1] could have far-reaching implications for anyone who cares about the distant consequences of their actions. Most notably, we might be clueless about how our decisions affect the long-term future, if our estimates of our net effects on long-term welfare ought to be severely indeterminate. Perhaps we [...]

---

Outline:

(05:21) Background on degrees of belief and what makes them rational

(08:25) Motivating example

(11:46) The structure of indeterminacy

(14:23) Indeterminate Bayesianism

(19:26) Decision-making: A first pass

(21:00) Practical hallmarks of indeterminacy

(21:42) Insensitivity to mild sweetening

(24:47) Suspending judgment on total effects, and choosing based on other reasons

(31:06) Responses to key objections to indeterminacy

(31:42) Aggregating our representor with higher-order credences uses more information

(32:51) Responses

(38:09) Precise forecasts do better than chance

(39:03) Responses

(42:06) Maximality is too permissive

(44:06) Responses

(45:47) Conclusion

(46:29) Acknowledgments

(46:51) Appendix: Indeterminacy for ideal agents

(50:57) Indeterminate priors

(54:40) The principle of indifference

(58:23) Occam's razor

The original text contained 25 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Edit 1/29: Funding is back, baby!

Crossposted from my blog.

(This could end up being the most important thing I’ve ever written. Please like and restack it—if you have a big blog, please write about it).

A mother holds her sick baby to her chest. She knows he doesn’t have long to live. She hears him coughing—those body-wracking coughs—that expel mucus and phlegm, leaving him desperately gasping for air. He is just a few months old. And yet that's how old he will be when he dies.

The aforementioned scene is likely to become increasingly common in the coming years. Fortunately, there is still hope.

Trump recently signed an executive order shutting off almost all foreign aid. Most terrifyingly, this included shutting off the PEPFAR program—the single most successful foreign aid program in my lifetime. PEPFAR provides treatment and prevention of HIV and AIDS—it has saved about [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The views expressed here are my own, not those of the people who provided feedback on the draft.

Summary

• Ambitious Impact (AIM) implicitly assumed all species have welfare ranges conditional on sentience equal to that of humans until 2024. Consequently, AIM had been overestimating the welfare range of species with lower welfare ranges conditional on sentience, and therefore the cost-effectiveness of helping animals of such species. For example, I believe helping fish was favoured relative to helping chickens, as fish have a lower welfare range conditional on sentience than chickens.
• Using Rethink Priorities’ (RP's) median welfare ranges instead of AIM's past assumption described above, the 3 fish welfare interventions AIM has recommended have a significantly lower cost-effectiveness than that of cage-free campaigns at the time of recommendation.
• I conclude influencing European Union fish welfare policy through strategic work in Greece advocating for electrical stunning before slaughter is 12.3 % as cost-effective as cage-free campaigns. [...]

---

Outline:

(00:16) Summary

(05:14) Fish welfare interventions recommended by Ambitious Impact

(05:20) Context

(06:07) Calculations

(09:50) Discussion

(09:54) Influencing European Union fish welfare policy through strategic work in Greece advocating for electrical stunning before slaughter

(14:25) East Asian fish welfare

(14:52) Improving water quality on fish farms

(15:34) Fish Welfare Initiative's farm program

(15:39) Context

(16:27) Calculations

(19:40) Discussion

(29:36) Acknowledgements

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

We have contact details and can send emails to 1500 students and former students who've received hard-cover copies of HPMOR (and possibly Human Compatible and/or The Precipice) because they've won international or Russian olympiads in maths, computer science, physics, biology, or chemistry.

This includes over 60 IMO and IOI medalists.

This is a pool of potentially extremely talented people, many of whom have read HPMOR.

I don't have the time to do anything with them, and people in the Russian-speaking EA community are all busy with other things.

The only thing that ever happened was an email sent to some kids still in high school about the Atlas Fellowship, and a couple of them became fellows.

I think it could be very valuable to alignment-pill these people. I think for most researchers who understand AI x-risk well enough and can speak Russian, even manually going through IMO and IOI [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

TLDR

• Original Position is a new effective giving organisation that uses a virtual ‘lottery of birth’ experience to inspire high-earners ($100k-$1m) in the US to pledge a portion of their income to effective charities
• This demographic donate less proportionally than other income brackets and are underserved by effective giving initiatives
• A digital, immersive experience meets this demographic's need for a quick intervention and provides EAs a tool to share with this often difficult-to-reach segment
• My background (management consulting, business school, tech startups) makes me well positioned to understand and reach this demographic
• It is high-risk, high-potential reward bet to commit time and resources to proving out
  • My research and initial pilot give me ~20% confidence of achieving a >35 multiplier over the next two years, with a median outcome of a 2-5x multiple

Asks

• Try the experience and share your feedback if you’re part of the target demographic
  • Try the Demo experience if you’re [...]

---

Outline:

(00:06) TLDR

(01:10) Asks

(01:47) The Problem: An Underserved Demographic

(03:20) Reaching high-earners through referral marketing

(04:45) The Original Position Experience provides efficient inspiration to make an effective giving pledge

(06:03) A vision to be a thoughtful, tech-for-good leader

(08:08) The Original Position Fund simplifies effective giving for a time-strapped audience

(09:33) This Theory of Impact offers a low probability of outsized returns

(11:27) Initial validation is positive but limited

(14:08) I have strong personal fit with this problem

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Note: This post was crossposted from the Open Philanthropy Farm Animal Welfare Research Newsletter by the Forum team, with the author's permission. The author may not see or respond to comments on this post.

They’re imperfect agents of change

The world's three largest animal welfare groups are under attack. Their antagonists are not factory farmers, but other animal groups. And the ASPCA, HSUS, and RSPCA stand accused not of hurting farmers, but of hurting animals, through their work with GAP and RSPCA Assured, which certify animal products as being less cruelly produced.

The attacks began last summer when the UK animal rights group Animal Rising released a report and footage showing abuses on RSPCA Assured farms. They’ve since forced the RSPCA to cancel its 200th year celebrations, plastered portraits of RSPCA patron King Charles, and persuaded the ceremonial president and two vice-presidents of the RSPCA to resign in protest. [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Phil Trammell an economics postdoc at the Digital Economy Lab, Stanford University.*
Zach Mazlish is an economics doctoral student at the University of Oxford.

As readers of this forum well know, some claim that advanced AI will greatly

1. accelerate economic growth,
2. displace labor, and/or
3. risk existential catastrophe.

We are preparing a course this August at Stanford on a selection of tools in (mainly) economic theory relevant to evaluating and expanding on the first two claims, and deciding what to do (altruistically) if some or all of them seem likely.
More information about the program may be found here.

Dates: August 16-29

The course is designed to be most suitable for graduate students in economics, but others prepared to get a lot out of the content are also encouraged to apply, including strong undergraduates, and students of computer science. Some of the lectures will be given by guest lecturers [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

One hope for keeping existential risks low is to get AI companies to (successfully) make high-assurance safety cases: structured and auditable arguments that an AI system is very unlikely to result in existential risks given how it will be deployed.[1] Concretely, once AIs are quite powerful, high-assurance safety cases would require making a thorough argument that the level of (existential) risk caused by the company is very low; perhaps they would require that the total chance of existential risk over the lifetime of the AI company[2] is less than 0.25%[3][4].

The idea of making high-assurance safety cases (once AI systems are dangerously powerful) is popular in some parts of the AI safety community and a variety of work appears to focus on this. Further, Anthropic has expressed an intention (in their RSP) to "keep risks below acceptable levels"[5] and there is a common impression that Anthropic would pause [...]

---

Outline:

(03:19) Why are companies unlikely to succeed at making high-assurance safety cases in short timelines?

(04:14) Ensuring sufficient security is very difficult

(04:55) Sufficiently mitigating scheming risk is unlikely

(09:35) Accelerating safety and security with earlier AIs seems insufficient

(11:58) Other points

(14:07) Companies likely wont unilaterally slow down if they are unable to make high-assurance safety cases

(18:26) Could coordination or government action result in high-assurance safety cases?

(19:55) What about safety cases aiming at a higher risk threshold?

(21:57) Implications and conclusions

The original text contained 20 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.