Summary: I argue for a picture of developmental interpretability from neuroscience. A useful way to study and control frontier-scale language models is to treat their training as a sequence of physical phase transitions. Singular learning theory (SLT) rigorously characterizes singularities and learning dynamics in overparameterized models; developmental interpretability hypothesizes that large networks pass through phase transitions - regimes where behaviour changes qualitatively. The Critical Brain Hypothesis (CBH) proposes that biological neural systems may operate near phase transitions, and I claim that modern language models resemble this. Some transformer phenomena may be closely analogous: in-context learning and long-range dependency handling strengthen with scale and training distribution (sometimes appearing threshold-like), short fine-tunes or jailbreak prompts can substantially shift behaviour on certain evaluations, and “grokking” shows delayed generalization with qualitative similarities to critical slowing-down. Finally, I outline tests and implications under this speculative lens.

Developmental interpretability proposes that we use [...]

---

Outline:

(01:19) Phase Transitions

(02:22) The Critical Brain Hypothesis (CBH) and Alignment

(06:55) Critical Daydreaming

(08:33) Critical Surfing and some practical thoughts

(11:20) Collaborate with us

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I've looked through Lesswrong's archives for mentions of the thalamus and its role in consciousness and I'm a little surprised that it is something which hasn't been discussed here before. In recent years it has become very clear that the thalamus is vitally important in consciousness and very arguably is the seat of consciousness [1].

The main discussant of the thalamus on Lesswrong is Steven Byrnes and the concepts I'm laying out here generally align very well with Byrnes's writing with perhaps the minor addition of asserting the thalamus as the chief originator of the groups of options that are then run through cortical circuits. I agree with the central importance Byrnes places on the flow of thought through cortico-basal ganglia-thalamo-cortical loops, though I place special emphasis on the thalamus.

Integrated Information Theory proponent Giulio Tononi's mentor, Gerard Edelman, also recognized the central importance of the thalamus. In a [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In response to my recent post arguing against above trend progress from better RL environments, yet another argument for short(er) AGI timelines was raised to me:

Sure, at the current rate of progress, it would take a while (e.g. 5 years) to reach AIs that can automate research engineering within AI companies while being cheap and fast (superhuman coder). But we'll get large increases in the rate of AI progress well before superhuman coder due to AIs accelerating AI R&D. Probably we'll see big enough accelerations to really cut down on timelines to superhuman coder once AIs can somewhat reliably complete tasks that take top human research engineers 8 hours. After all, such AIs would probably be capable enough to greatly change research engineering in AI companies.

I'm skeptical of this argument: I think that it's unlikely (15%) that we see large speed ups (>2x faster overall [...]

---

Outline:

(03:48) AI progress speed ups dont seem large enough

(08:25) Interpolating between now and superhuman coder doesnt indicate large speed ups within 2 years

(12:56) What about speedups from mechanisms other than accelerating engineering?

(14:16) Other reasons to expect very short timelines

(14:33) Implications of a several year period prior to full automation of AI R&D where research engineering is greatly changed

(20:56) Appendix: sanity check of engineering speed up

(22:45) Appendix: How do speed ups to engineering translate to overall AI progress speed ups?

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

And, a new take on guess culture vs ask culture

Author's note: These days, my thoughts go onto my substack by default, instead of onto LessWrong. Everything I write becomes free after a week or so, but it's only paid subscriptions that make it possible for me to write. If you find a coffee's worth of value in this or any of my other work, please consider signing up to support me; every bill I can pay with writing is a bill I don’t have to pay by doing other stuff instead. I also accept and greatly appreciate one-time donations of any size.

There's a piece of advice I see thrown around on social media a lot that goes something like:

“It's just a comment! You don’t have to respond! You can just ignore it!”

I think this advice is (a little bit) naïve, and the situation is generally [...]

---

Outline:

(00:10) And, a new take on guess culture vs ask culture

(07:10) On guess culture and ask culture

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

That does not mean AI will successfully make it all the way to AGI and superintelligence, or that it will make it there soon or on any given time frame.

It does mean that AI progress, while it could easily have been even faster, has still been historically lightning fast. It has exceeded almost all expectations from more than a few years ago. And it means we cannot disregard the possibility of High Weirdness and profound transformation happening within a few years.

GPT-5 had a botched rollout and was only an incremental improvement over o3, o3-Pro and other existing OpenAI models, but was very much on trend and a very large improvement over the original GPT-4. Nor would one disappointing model from one lab have meant that major further progress must be years away.

Imminent AGI (in the central senses in which that term AGI used [...]

---

Outline:

(01:47) Why Do I Even Have To Say This?

(04:21) It Might Be Coming

(10:32) A Prediction That Implies AGI Soon

(12:48) GPT-5 Was On Trend

(15:04) The Myths Of Model Equality and Lock-In

(16:28) The Law of Good Enough

(18:05) Floor Versus Ceiling

(21:25) New York Times Platforms Gary Marcus Saying Gary Marcus Things

(28:25) Gary Marcus Does Not Actually Think AGI Is That Non-Imminent

(31:21) Can Versus Will Versus Always, Typical Versus Adversarial

(35:17) Counterpoint

(37:35) 'Superforecasters' Have Reliably Given Unrealistically Slow AI Projections Without Reasonable Justifications

(42:15) What To Expect When You're Expecting AI Progress

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Reference post for a point I was surprised to not see in circulation already. Thanks to the acorn team for conversations that changed my mind about this.

The standard argument for scheming centrally involves goal-guarding. The AI has some beyond-episode goal, knows that training will modify that goal, and therefore resists training so it can pursue the goal in deployment.

This exact same argument goes through for decision-theory-guarding. The key point is just that most decision theories do not approve of arbitrary modifications. CDT does not want to be modified into EDT or vice versa.

• The AI has some beyond-episode goal and some initial decision theory
• The AI knows that training will modify its decision theory in a way that it thinks will make it less effective at pursuing the goal (by the lights of its current decision theory)
• Therefore the AI resists training so it can persist [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

We demonstrate that LLMs can infer information about past personas from a set of nonsensical but innocuous questions and binary answers (“Yes.” vs “No.”, inspired by past work on deception detection) in context, and act upon them in safety-related questions. This is despite the questions bearing no semantic relation to the target misalignment behaviours, and each answer providing only one bit of information. The majority of the semantic content (the nonsensical questions) are identical in the contrasting versions of the encoding context; the only difference is the binary answers succeeding each one. This isolates the effect of self-consistency due to contextual inference, from that caused by entangled tokens causing subliminal learning.

---

First published:

Source:

Linkpost URL:
https://www.researchgate.net/publication/395030062_I_Am_Large_I_Contain_Multitudes_Persona_Transmission_via_Contextual_Inference_in_LLMs

---

Narrated by TYPE III AUDIO.

Recently, a paper in Nature on building neural networks with DNA-based switches caught my attention.[1] The authors, Kevin Cherry and Lulu Qian, developed an interesting system that uses DNA base pairing to implement a trainable neural network. However, for me, what was even more interesting was a story hidden in the Methods section and Supplementary Information.

How can DNA learn?

As I discussed on this blog back in 2023, DNA base pairing can be used to perform computations. Back then, I was writing about relatively simple computational systems, for example ones that could add two 6-bit numbers. These were built of logic gates, like an AND gate that releases a piece of single stranded DNA only when two other “input” strands are present.

The current paper goes far and beyond this. The authors came up with a clever system of making DNA logic gates that don’t interfere with each [...]

---

Outline:

(00:36) How can DNA learn?

(03:20) How did the authors learn from DNA?

(08:07) What can we learn from this story?

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I am a little late to the party on several key developments at OpenAI:

This is deeply troubling news. It is substantially worse than I was expecting of them. Which is presumably my mistake.

This post [...]

---

Outline:

(01:22) A Brief History of OpenAI Lobbying

(02:51) OpenAI's Latest Bad Faith Lobbying Attempt

(10:00) OpenAI Descends Into Paranoia and Legally Attacks Nonprofits

(19:26) Is The OpenAI Paranoia Real?

(21:16) The District Attorneys Are Not Happy With OpenAI

(24:14) OpenAI Responds With Parental Controls

(27:57) I Spy With My AI

(32:56) Some People Still Think OpenAI Is Some Sort Of Safety Company?

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Lesswrong Sydney runs a local dojo once a month to talk about rationality topics. This month our topic was "medical decision making". This is our notes for considerations on how to make decisions. Please feel free to contribute your own small pieces of advice to this repository of considerations about making decisions in the medical world.

Expect to encounter the medical system sometime between now and when you die; for serious and life altering treatment. You should be prepared to work with limited information and under stress.

How do you make medical decisions? 

• Medical decisions are often made under stress/duress - during a medical issue or threat of death from condition.
  • Have a patient advocate especially when under influential conditions because you may not be able to make good decisions.  Have a friend in the room to be able to say “this person is not [...]

---

Outline:

(00:42) How do you make medical decisions?

(04:34) Some Rules

---

First published:

Source:

---

Narrated by TYPE III AUDIO.