The conversation begins

(Fictional) Optimist: So you expect future artificial superintelligence (ASI) “by default”, i.e. in the absence of yet-to-be-invented techniques, to be a ruthless sociopath, happy to lie, cheat, and steal, whenever doing so is selfishly beneficial, and with callous indifference to whether anyone (including its own programmers and users) lives or dies?

Me: Yup! (Alas.)

Optimist: …Despite all the evidence right in front of our eyes from humans and LLMs.

Me: Yup!

Optimist: OK, well, I’m here to tell you: that is a very specific and strange thing to expect, especially in the absence of any concrete evidence whatsoever. There's no reason to expect it. If you think that ruthless sociopathy is the “true core nature of intelligence” or whatever, then you should really look at yourself in a mirror and ask yourself where your life went horribly wrong.

Me: Hmm, I think the “true core nature of intelligence” is above my pay grade. We should probably just talk about the issue at hand, namely future AI algorithms and their properties.

…But I actually agree with you that ruthless sociopathy is a very specific and strange thing for me to expect.

Optimist: Wait, you—what??

Me: Yes! Like [...]

---

Outline:

(00:11) The conversation begins

(03:54) Are people worried about LLMs causing doom?

(06:23) Positive argument that brain-like RL-agent ASI would be a ruthless sociopath

(11:28) Circling back LLMs: imitative learning vs ASI

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

A Harry Potter fanfiction. Based on the world of "Harry Potter and the Methods of Rationality" by Eliezer Yudkowsky, diverging from canon.

Harry had been having, by any objective measure, an excellent week.

On Monday he had demonstrated, to his own satisfaction and Professor Flitwick's visible alarm, that the Hover Charm could be generalized to any object regardless of mass if you conceptualized it as a momentum transfer rather than a force application. On Wednesday he had worked out why Neville's potions kept failing — the textbook instructions assumed clockwise stirring, but the underlying reaction was chirally sensitive, and Neville was left-handed. A trivial fix. Neville had cried.

On Friday evening, buoyed by the week's successes and looking for a specific reference on crystalline wand cores that he was certain would unlock a further generalization of his momentum framework, Harry was in the Restricted Section.

He had access. Professor McGonagall had granted it after the Hover Charm incident, in a tone that suggested she was choosing between supervised access and finding him there anyway at 2 AM. A reasonable calculation on her part.

The book he wanted wasn't where the index said it should be. In its place was [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

It seems to me that the Hamming problem for developing a formidable art of rationality is, what to do about problems that systematically fight being solved. And in particular, how to handle bad reasoning that resists being corrected.

I propose that each such stubborn problem is nearly always, in practice, part of a solution to some social problem. In other words, having the problem is socially strategic.

If this conjecture is right, then rationality must include a process of finding solutions to those underlying social problems that don’t rely on creating and maintaining some second-order problem. Particularly problems that convolute conscious reasoning and truth-seeking.

The rest of this post will be me fleshing out what I mean, sketching why I think it's true, and proposing some initial steps toward a solution to this Hamming problem.

Truth-seeking vs. embeddedness

I’ll assume you’re familiar with Scott & Abram's distinction between Cartesian vs. embedded agency. If not, I suggest reading their post's comic, stopping when it mentions Marcus Hutter and AIXI.

(In short: a Cartesian agent is clearly distinguishable from the space the problems it's solving exists in, whereas an embedded agent is not. Contrast an entomologist studying an ant colony (Cartesian) [...]

---

Outline:

(00:59) Truth-seeking vs. embeddedness

(02:57) Protected problems

(06:42) Dissolving protected problems

(08:11) Develop inner privacy

(11:54) Look for the social payoff

(14:51) Change your social incentives

(18:52) The right social scene would help a lot

(20:59) Summary

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

In the UK, you need a license to watch TV. The proceeds from the license fee fund the state broadcasting company, the BBC. In order to enforce this, the government sends out TV license inspectors, who go into unlicensed properties to check whether there's a TV present. Of course you don't actually have to let them in; they have no legal power! They are real. I had one visit once. He came in and asked what my TV was. I (truthfully) told him we watched Netflix and Youtube on it, and didn't own a TV box. He left.

They also have TV detector vans, which drive around and detect which houses have television sets.

On The Existence—or Non-Existence—of TV Detector Vans

On the one hand TV detector vans don't exist. Wikipedia calls them an "urban legend". It goes on to say that no detector van has ever enforced a TV license fee. This appears to be true. The BBC has never registered a patent on the operating principles of a TV detector van. The principle of detecting a broadcast receiver is itself pretty fraught. How do you detect whether something is absorbing a signal?

On the other hand [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Your hot takes are killing your credibility.

Prior to my last year at ControlAI, I was a physicist working on technical AI safety research. Like many of those warning about the dangers of AI, I don’t come from a background in public communications, but I’ve quickly learned some important rules. The #1 rule that I’ve seen far too many others in this field break is that You’re an AI Expert - Not an Influencer.

When communicating to an audience, your persona is one of two broad categories: Influencer or Professional

• Influencers are individuals who build an audience around themselves as a person. Their currency is popularity and their audience values them for who they are and what they believe, not just what they know.
• Professionals are individuals who appear in the public eye as representatives of their expertise or organization. Their currency is credibility and their audience values them for what they know and what they represent, not who they are.

So… let's say you’re trying to be a public figure making a difference about AI risk. You’ve been on a podcast or two, maybe even on The News. You might work at an AI policy organization, or [...]

---

Outline:

(00:11) Your hot takes are killing your credibility.

(02:10) STOP - What Would Media Training Steve do?

(05:22) Dont Feed Your Enemies

(07:07) The Luxury of Not Being a Politician

(09:33) So How Do You Deal With Politics?

(10:58) Conclusion

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Epistemic status: Speculation on agent foundations research culture (which I am pretty deeply engaged with) and whether "we are confused about agency" which I am not sure about. I will take for granted that this is a common refrain, which should be familiar to anyone who is part of the relevant scene.

The phrase "We are confused about agency," often with variations such as "way too," "deeply," or "dangerously" is a common membership signal for a certain AI safety research culture. Roughly speaking, this is the culture of the agent foundations research program that accreted around MIRI.[1] The phrase is usually supplied as an argument for delaying the development of A.I. until certain mathematical research (particularly in learning/decision theory) has been carried out.

I find the phrase uncomfortable on various levels.

As Cultural Signal

Since I claim that the phrase (apart from its literal meaning) functions as an in-group signal, it is natural to wonder what "we" refers to here.

I believe the intention is "we = everyone." I think that it requires a pretty serious level of scholarship to confidently claim that everyone is confused about agency. One general frustration I have with rationalist culture [...]

---

Outline:

(00:59) As Cultural Signal

(02:48) Confused in what Sense?

(05:10) Confused about what?

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Some people say “the brain is a computer”. Other people say “well, the brain is not really a computer, because, like, what's the hardware versus the software?” I agree: “the brain is a computer” is kinda missing the mark. I prefer: “the brain is a machine that runs an algorithm”.

Here's a mechanical adder:

What's “hardware versus software” for a mechanical adder? The question is nonsense.

A mechanical adder is not “a computer”, analogous to a MacBook. Rather, it's a machine that runs an algorithm. (Namely, the binary addition algorithm.)

And the brain is likewise a machine that runs a (much more complicated) algorithm.

“A machine??”, you say. Yeah, you heard me. A machine. An extraordinarily complex machine, but a machine all the same. If you could zoom in enough to really see it, it would just be obvious! You should pause here to marvel at some molecular simulations of cell biology in action: DNA replication, more crazy DNA stuff, kinesin, and so on.

…And what else would you expect? We live in a universe that follows orderly laws of physics.[1] And the laws apply to our bodies and brains just like everything else. After all, scientists have been [...]

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Apply to Seminar to Study, Explain, and Try to Solve Superintelligence Alignment

Applications for the AFFINE Superintelligence Alignment Seminar are now open, and we invite you to apply. It will take place in Hostačov, near Prague (Czechia), from 28 April to 28 May.

We are working on a draft of the learning materials, in consultation with world-experts.

KEY INFO

• Dates: From 28th April to 28th of May
• Location: Hostačov, 582 82 Skryje-Golčův Jeníkov, Czechia 🇨🇿
• Accommodation & catering: Covered
• Mentors: Abram Demski, Ramana Kumar, Steve Byrnes, Kaj Sotala, Kaarel Hänni, Jonas Hallgren, Ouro, Cole Wyeth, Aram Ebtekar, Elliot Thornley, Linda Linsefors, Paul ‘Lorxus’ Rappoport, and more
• Positions available: 30
• Attendance cost: Free (donations welcome, we have some budget for travel expenses)
• Stipends: We are applying for additional funding to cover stipends for those who would benefit from them
• TO JOIN: Fill out this form by the 8th of March

Goal

The main purpose of the Seminar is to give promising newcomers to AI alignment an opportunity to acquire a deep understanding of some large pieces of the problem, making them better equipped for work on the mitigation of AI existential risk.

ASI breaking out of human control and [...]

---

Outline:

(00:37) KEY INFO

(01:36) Goal

(02:39) Strategy

(05:20) Interested?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was one of those. So here we go.

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary. Some points are dropped.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

Normally I keep everything to numbered lists, but in several cases here it was more of a ‘he didn’t just say what I think he did did he’ and I needed extensive quotes.

In addition to the podcast, there were some discussions around safety, or the lack thereof, at xAI, and Elon Musk went on what one can only describe as megatilt, including going hard after Anthropic's Amanda Askell. I will include that as a postscript.

I will not include recent developments regarding Twitter, since that didn’t come up in the interview.

I lead with a discussion of bounded distrust and how to epistemically consider Elon Musk, since that will be important throughout including in the postscript.

What are the key takeaways?

---

Outline:

(02:56) Bounded Distrust

(05:12) IN SPACE

(09:56) The AI Will Follow You To Mars

(22:32) xAI Business Plans

(25:54) Optimus Prime

(27:04) Beating China

(30:02) SpaceX and How To Run a Company Elon Style

(33:17) DOGE

(35:29) TeraFab IN SPACE

(35:47) Postscript: Safety Third at xAI

(40:15) Elon Serves Back Saying That Which Is Not

(42:51) Elons Army

(43:55) Children Are Our Future

(48:11) Where Do We Go From Here

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

One of the central arguments for AI existential risk goes through inner misalignment: a model trained to exhibit aligned behavior might be pursuing a different objective internally, which diverges from the intended behavior when conditions shift. This is a core claim of If Anyone Builds It, Everyone Dies: we can't reliably aim an ASI at any goal, let alone the precise target of human values.

A common source of optimism, articulated by Nora Belrose & Quintin Pope or Jan Leike, analyzed by John Wentworth, and summarized in this recent IABIED review, goes something like: current LLMs already display good moral reasoning; human values are pervasive in training data and constitute "natural abstractions" that sufficiently capable learners converge on; so alignment should be quite easy, and get easier with scale.

I think the history of LLM jailbreaking is a neat empirical test of this claim.

Jailbreaking

A jailbreak works like this:

1. A model is trained (RLHF, SFT, constitutional AI, etc.) to refuse harmful requests.
2. Someone crafts a prompt that's out of distribution (in a broad sense) relative to the safety training.
3. The model produces the harmful output.

This is goal misgeneralization. The model learned something during safety training that [...]

---

Outline:

(01:08) Jailbreaking

(02:59) What the alignment-by-default view would predict (I think)

(03:35) It doesnt work

(05:00) Conclusion

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.