We have developed some relatively general methods for mechanistic estimation competitive with sampling by studying problems that are expressible as expectations of random products. This includes several different estimation problems, such as random halfspace intersections, random #3-SAT and random permanents. In this post, we will give a high-level introduction to these methods before sharing some more detailed notes. This is intended as an interim technical update and will be relatively light on motivation: for a broader discussion of this line of research, see our prior post.

Random instances of the matching sampling principle

All of the problems discussed in this post can be thought of particular choices of "architecture" in our matching sampling principle. In fact, they are all choices in which has no learned or worst-case parameters . They still have random parameters, which are captured in the "context" variable , making them similar to randomly-initialized networks rather than trained networks. Note that when is missing from , no "explanation" is required by the estimation algorithm , which reflects the fact that there is no "structure" in a randomly-initialized network that needs to be pointed out.

Why study random instances of the matching sampling principle? [...]

---

Outline:

(00:46) Random instances of the matching sampling principle

(02:02) Expectations of random products

(03:13) Deduction-projection estimators

(04:59) Mechanistic sketching

(06:03) Detailed notes

(08:36) Conclusion

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

I couldn’t find a recent write-up from a MATS alum about what attending MATS was like, so this is the thing that I wish I had. I attended MATS from January to March 2026, on Team Shard with Alex Turner and Alex Cloud. It was a great time! Applications for MATS are basically on a rolling basis nowadays, and I can strongly recommend applying (to multiple streams) even if you think you’re not a great match.

With that being said, there's a lot I wish I knew going into MATS, so here's a brain-dump of thoughts. It's not extremely polished, but I expect it’ll be useful nonetheless (none of this is endorsed by MATS, just my thoughts):

Work ethic

I think most mentees were working 10-12, sometimes 14 hours a day Mon-Fri, and probably 2-8 hours on Saturday and Sunday, often going out on some adventure or party on the weekend. Exactly which hours people worked varied wildly. I usually worked 8:30am/9am to 11pm/midnight, with breaks during the day, others worked from midday into the early hours of the morning. This was surprisingly sustainable (IMO); MATS puts a lot of effort into removing all other blockers that you normally [...]

---

Outline:

(00:50) Work ethic

(01:29) Use more compute

(02:20) Research requires a lot of compute

(03:12) Applying for jobs during MATS (dont do it)

(04:55) The serious people are in War Mode

(05:44) Do you feel the AGI?

(06:00) Burn rate, efficiency, and decisions

(07:12) insider information

(08:08) Names & Faces

(08:20) Fellows

(08:50) Useful tools

(11:19) Use more Claudes

(12:06) Build nice helper utilities for yourself

(12:59) MATS-mentee-mentor dynamics

(13:45) Working with your mentors

(14:27) Research managers

(14:48) Ops requests

(15:38) Non-MATS events

(16:17) Team Shard

(17:12) Weekly updates

(18:46) Keep a log of your mistakes

(19:06) My running-experiments setup

(27:51) Lighthaven

(28:12) Getting setup with the Compute team

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

An insidious pattern among smart people is feeling that because something is familiar and obvious, you are impervious to ignoring or forgetting it. In challenging times, I have often heard these clichés and reflexively shrugged them off.

“Oh, I should dust myself off and pick myself up? What a lazy aphorism. What a patronising throwaway line. They must think I’m some kind of idiot. No, it must be something else…”

There is a filter in many people's heads that functions to ignore clichés on the basis that they are mere clichés. However, looking closely at your actions, decisions and attitudes will almost invariably reveal you are dropping the ball on at least a few of the most obvious bits of pop wisdom that are all clearly good practices.

Some (non exhaustive) examples of these “yeah, obviously” pieces of advice that are worth deliberately checking on a regular basis include:

• Believe in yourself

The greatest example of an eye-rolling cliché is also one of the highest impact pieces of advice ever articulated. Self-belief is the foundation of morale, and without morale, you are doomed. If the words “believe in yourself” evoke a sense [...]

---

First published:

Source:

Linkpost URL:
https://hidefromit.substack.com/p/dont-be-too-clever-to-take-obvious

---

Narrated by TYPE III AUDIO.

"Sometimes the AI just makes stuff up" is a problem I don't really expect to go away. In the nearterm, AI is going to keep occasionally hallucinating, or misinterpreting information. Eventually, AI will be powerful enough we need to be worried if it's presenting misleading information on purpose.

There might be a nice window where the AI is powerful enough to not make things up but non-agentic enough that we don't have to worry about deliberate manipulation. But, even then, interpreting data is tricky.

I'm worried about this for my own use, but, I'm more worried about this on the global scale. I'm worried about people trusting things AI made up, and I'm worried about the internet proliferating with slop that makes it harder to even find original statements that are a human's real testimony.

An approach that might help is to make AI reports more "Verification centric."

Right now, some AI chatbots provides little citation-links. That's better than not-having-them. But, those are a pain to open and investigate. Probably you very rarely do so.

So, imagine a world where when you answer a question, the AI doesn't guess. Instead:

• It finds a primary source.
• It [...]

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Tl;dr

Convergent abstraction hypothesis posits abstractions are often convergent in the sense of convergent evolution: different cognitive systems converge on the same abstraction, when facing similar selection pressures and learning in similar environments. It is a less ambitious alternative to 'natural abstractions hypotheses' and, in my view, more likely to be true.

Convergence may be real, useful, and empirically robust, while still being contingent and fragile under changes in architecture, training pressure, or optimization regime.

Convergent evolution 

To recapitulate some basics: notice that sharks, ichthyosaurs and dolphins look remarkably similar, despite their ancestors being wildly different.

This is a case of convergent evolution: the process by which organisms with different origins develop similar features. Both sharks and dolphins needed speed and energy efficiency when moving in an environment governed by the laws of hydrodynamics, and so they converged on a pretty similar body shape.

A complementary concept from biology is contingency - convergent features are themselves often contingent on some other feature. For example, the shark morphology depends on there being a central vertebral column (or spine), which only evolved once and is the defining feature of vertebrates. There are many other aquatic predators [...]

---

Outline:

(00:44) Convergent evolution

(01:53) Convergent evolution applied to abstractions

(02:22) Selection pressures

(02:53) Compression buys you a lot

(04:41) Shared environments

(05:30) What this predicts

(06:21) What this does not predict, or contingency in abstraction space

(06:45) Case study of architectural contingency: convergent vision abstractions

(09:14) Whats the difference between convergent abstractions and natural abstractions

(11:22) Why do I care

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is what a lull looks like at this point. The government is having internal arguments. The models are getting improved internally. The coding agent improvements are all what we would expect. There's still a lot happening, including a bunch of cool papers, but I feel able to relax and to take care of some other work while I have the chance. You never know when that chance will be over.

Table of Contents

From yesterday: Cyber Lack of Security and AI Governance.

---

Outline:

(00:36) Language Models Offer Mundane Utility

(02:25) Language Models Dont Offer Mundane Utility

(03:26) Huh, Upgrades

(04:02) Levels of Friction

(05:11) On Your Marks

(06:38) Get My Agent On The Line

(09:42) Deepfaketown and Botpocalypse Soon

(11:11) Fun With Media Generation

(12:49) On AI Writing

(16:05) A Young Ladys Illustrated Primer

(18:35) You Drive Me Crazy

(19:23) They Took Our Jobs

(21:31) The Art of the Jailbreak

(21:49) Introducing

(22:29) Claude Has Its Limits

(23:28) Show Me the Money

(25:15) Show Me The Compute

(26:58) Quiet Speculations

(28:33) Quickly, Theres No Time

(29:32) Chip City

(30:37) Pick Up The Phone

(31:15) The Week in Audio

(31:41) Rhetorical Innovation

(36:13) Not Leading the Future

(42:24) Elon Musk v OpenAI

(43:21) People Just Say Things

(43:47) People Just Publish Things

(44:37) OpenAI Endroses Kosa And SB 315

(48:08) The LLMs All Believe Roughly Similar Things

(50:45) I Learned It By Reading YOU

(01:08:22) Aligning a Smarter Than Human Intelligence is Difficult

(01:18:45) People Are Worried About AI Killing Everyone

(01:20:34) Messages From Janusworld

(01:22:31) People Worried About AI For Other Reasons

(01:23:06) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Summary

This is a summary of a paper published by the alignment team at UK AISI. Read the full paper here.

AI research agents may help solve ASI alignment, for example via the following plan:

• Build agents that can do empirical alignment work (e.g.~writing code, running experiments, designing evaluations and red teaming) and confirm they are not scheming.[1]
• Use these agents to build increasingly sophisticated empirical safety cases for each successive generation of agents, gradually automating more of the research process
• Hand over primary research responsibility once agents outperform humans at all relevant alignment tasks.

We argue that automating alignment research in this manner could produce catastrophically misleading safety assessments, causing researchers to believe that an egregiously misaligned AI is safe, even if AI agents are not scheming to deliberately sabotage alignment research. Our core argument (Fig. 1) is as follows:

1. The goal of an automated alignment program is to produce an overall safety assessment (OSA) - an estimate of the probability that the next-generation agent is non-scheming - that is both calibrated and shows low risk.[2]
2. Producing an OSA involves several tasks that are difficult to check. We refer to these as hard-to-supervise fuzzy tasks: tasks [...]

---

Outline:

(00:13) Summary

(07:10) Acknowledgments

The original text contained 4 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

1) The safe-to-dangerous shift is a fundamental problem for eval realism

Suppose we have a capable and potentially scheming model, and before we deploy it, we want some evidence that it won’t do anything catastrophically dangerous once we deploy it. A common approach is to use black-box alignment evaluations. However, alignment evaluations are only reassuring to the extent that the model can't reliably[1] distinguish the deployment distribution from the evaluation distribution, as it is otherwise difficult to rule out the possibility of alignment faking.

There are many approaches one could use to try to make evaluations appear more realistic: you can try to create realistic environments (e.g. Petri, WebArena, OSWorld); use data from past deployments (e.g. OpenAI, SAD); and spoof tool-call responses (e.g. ToolEmu).

However, the core difference between an alignment evaluation and a real deployment, is that an alignment evaluation must be safe[2]. That is, if you want to safely evaluate an untrusted AI system, you need to limit your AI's ability to cause harm within your evaluation environment. If you want to deploy your AI system (usefully) you need to give it at least some ability to affect the world, and potentially cause harm.

The safe-to-dangerous shift [...]

---

Outline:

(00:15) 1) The safe-to-dangerous shift is a fundamental problem for eval realism

(02:10) 2) The safe-to-dangerous shift is also a problem for measuring eval awareness

(05:09) 3) We should expect measuring eval realism to be hard.

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

TL;DR: We estimate how often Qwen 3 4B exhibits rare harmful behaviors with 30× fewer rollouts than naive sampling, using a new method that interpolates between the model and a less-safe variant in logit space.

Authors: Francisco Pernice (MIT), Santiago Aranguri (Goodfire)

Introduction

A harmful behavior that occurs once in a million rollouts will rarely surface during pre-deployment testing, yet will almost inevitably appear after release. Labs usually have another resource available: less safety-trained variants of the same model, on which rare harmful behaviors are not rare at all.

We find that the rate of rare harmful behaviors can be predicted by leveraging a less safe variant, requiring 30× fewer rollouts compared to naive sampling. Our method, Logit Path Extrapolation, interpolates between the two models in logit space, measures the compliance rate at points along the interpolation path where it is common, and extrapolates the resulting trend out to the original model. This outperforms the methods from prior work[1][2] by exploiting the path between the two models instead of just the endpoints.

Results

We can use 30× fewer rollouts to estimate how often Qwen 3 4B complies with harmful requests. We consider HarmBench prompts where the model never complies [...]

---

Outline:

(00:34) Introduction

(01:26) Results

(05:17) Conclusion

(06:22) Related work

(08:02) References

(08:09) Contribution statement

(08:31) Acknowledgement

(08:42) Appendix

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The real recent story of AI has been the background work being done on Cybersecurity, as we process the Mythos Moment along with GPT-5.5, and figure out both how to patch the internet and what our new regulatory regime is going to look like.

The Trump Administration is being dragged, kicking and screaming, into the era of at least some situational awareness, and acknowledgment that catastrophic risks are very much a real risk and they need to have a role in supervising frontier model releases. Now that they’re there, Commerce is deciding who gets access to the most powerful model in the world, and they are fighting Intelligence and the national security state over who should be in charge.

Another question is, exactly how strong is Mythos, both compared to past model and to GPT-5.5 and also in absolute terms? We got multiple new reports on that, as well as the METR graph results. There's little question Mythos is a big deal, but there's a wide range of big deals out there.

Part of the new report from UK AISI is learning that there is a substantial gap between the abilities of the early Mythos Preview [...]

---

Outline:

(01:28) On Your Marks

(05:24) How Good Is Mythos?

(11:11) Cyber Lack of Security

(14:36) Greetings From The Department of War

(16:13) The Prior Restraint Era Begins

(20:59) Commerce Versus Intelligence

(27:43) The Quest for Sane Regulations

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.