PDF version. berkeleygenomics.org. x.com. bluesky. YouTube.

Introduction

We, the free world, as a society, should decide to invest in the development of technology for human germline genomic engineering.

To me, this claim speaks for itself. Who wouldn't want to massively decrease the risks of many diseases in their future children? Who wouldn't want to give their children strong and diverse capabilities, like creativity, problem solving ability, wisdom, and empathy?

If so many people want this technology, doesn't it stand to reason that our scientists and entrepreneurs would be highly motivated and enabled to develop it? But, well, they aren't yet.

So, even though the claim speaks for itself, we have to ask: What would it look like to have a world where humanity greatly benefits from germline genomic engineering? What broader vision does reprogenetic technology fit into?

Some reasons we should work out [...]

---

Outline:

(00:18) Introduction

(02:14) Disclaimers

(02:51) What is emancipation?

(03:59) What is genomic emancipation?

(06:29) Emancipation from whom?

(06:43) Mother Nature

(07:54) The lesion

(09:53) Emancipation of whom?

(13:07) Emancipation by whom?

(13:47) Emancipation how?

(16:48) Emancipation to where?

(17:00) The bush

(18:58) Expansion

(21:54) Intelligence

(24:02) Equal opportunity

(24:18) Emancipation when?

(24:54) The urgent need for health

(25:19) The urgent need for human capacity

(26:28) Safety, efficacy, and societal consent

(28:08) Genomic engineering overhang

(30:22) The internet and human destiny

(34:18) Heal whats in front of you

(36:15) Buttressing emancipated humanity

(36:43) The infinite game

(38:33) Growing pains

(39:43) Habermas and multigenerational feedback

(41:41) Fukuyama and human dignity

(44:19) C.S. Lewis and the tyranny of one generation

(48:23) Sandel, May, Heidegger: the giftedness of being

(54:27) Palinsynopsis

(56:06) Conclusion

(56:17) Appendix: Is this eugenics?

(56:28) The substantive question

(57:05) The terminological question

(01:03:29) Appendix: Why envision genomic emancipation?

(01:04:30) Noticing major problems

(01:04:51) To be destroyed by true refutations

(01:06:31) To be sharpened by true criticisms

(01:08:10) To hold our intentions strongly

(01:08:18) To be defended against false refutations

(01:11:27) To be exerted to realize the good possible worlds

(01:13:11) To be spread and amplified

(01:13:45) Blunt-force responses

(01:16:27) Blunt-force responses are good

(01:17:29) The blunt-force response against germline genomic engineering

(01:19:35) Critiques of character and motivation

(01:26:03) Deference as a blunt-force response

(01:28:13) Appendix: The origins of souls

(01:28:19) Influences on a childs genome

(01:30:51) Why not just have the state enforce beneficial choices?

(01:32:59) Forms of parental care

(01:37:24) Non-separateness of parent and child

(01:41:38) Appendix: Intelligence

(01:46:08) Appendix: on safer-sooner reprogenetics

(01:46:14) Examples of how reprogenetics is safer if developed sooner

(01:49:06) The ramp-up of reprogenetic strength

(01:52:05) Acknowledgement

The original text contained 14 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Consider the following vignette:

It is March 2028. With their new CoCo-Q neuralese reasoning model, a frontier AI lab has managed to fully automate the process of software engineering. In AI R&D, most human engineers have lost their old jobs, and only a small number of researchers now coordinate large fleets of AI agents, each AI about 10x more productive than the humans they’ve replaced. AI progress remains fast, and safety teams are scrambling to prepare alignment and control measures for the next CoCo-R model, which is projected to be at least TEDAI. The safety team has no idea whether CoCo-Q or early checkpoints of CoCo-R are scheming (because progress in interpretability and automated alignment research turns out to be disappointing), and control evaluations become increasingly unreliable (because of worries that the models might be exploration hacking during red-team training). However, because so far there hasn’t been a major [...]

---

Outline:

(04:18) The central example of a mutually beneficial deal

(07:14) Why early misaligned AIs might not have very good alternatives to deals

(08:20) Early schemers' alternatives to making deals

(11:57) A wide range of possible early schemers could benefit from deals

(14:37) Credible commitments as a fundamental bottleneck

(19:17) Practicalities of making deals with early schemers in particular

(19:28) Setting up infrastructure to pay the AIs

(21:13) How do we enter into negotiations?

(22:21) Making sure the AI knows about the deal in other contexts

(23:05) Making sure the AI we make a deal with is actually able to make a deal

(24:25) Delayed adjudication (particularly of whether the AIs kept up their side of the deal)

(25:50) Next steps

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Currently, only 5 companies in the world have access to frontier AI training compute and are also pursuing development of AGI (Google DeepMind, OpenAI, Anthropic, xAI, and Meta). This will still hold in 2026 for Google and OpenAI, and plausibly also for Anthropic, Meta, and xAI.

Stance towards trying to develop AGI can change, but the frontier AI training compute barrier is increasingly insurmountable for any company that doesn't already have impressive AI development accomplishments. In 2024, frontier compute was 100K H100s, and that cost about $5-7bn (it was still possible to use legacy air cooling infrastructure with H100s). In 2025, that's 100K chips in GB200 NVL72 racks, which costs $7-11bn. In 2026, OpenAI's Stargate Abilene sets the lower bound at 400K chips in NVL72 racks (GB200 or possibly GB300), which is a 1 GW datacenter campus that costs $35-45bn (though you can continue building out the 2025 [...]

---

Outline:

(02:34) RLVR and Large World Size

(05:20) 2026

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

You can find Part 1 here. This resumes the weekly, already in progress. The primary focus here is on the future, including policy and alignment, but also the other stuff typically in the back half like audio, and more near term issues like ChatGPT driving an increasing number of people crazy.

If you haven’t been following the full OpenAI saga, the OpenAI Files will contain a lot of new information that you really should check out. If you’ve been following, some of it will likely still surprise you, and help fill in the overall picture behind the scenes to match the crazy happening elsewhere.

At the end, we have some crazy new endorsements for Eliezer Yudkowsky's upcoming book, If Anyone Builds It, Everyone Dies. Preorders make a difference in helping the book get better reach, and I think that will help us all have a much better [...]

---

Outline:

(01:02) Cheaters Gonna Cheat Cheat Cheat Cheat Cheat

(02:12) Quiet Speculations

(07:44) Get Involved

(07:53) Thinking Machines

(08:04) California Reports

(10:39) The Quest for Sane Regulations

(13:40) What Is Musk Thinking?

(16:54) Why Do We Care About The 'AI Race'?

(23:52) Chip City

(26:29) Pick Up The Phone

(26:46) The OpenAI Files

(33:13) The Week in Audio

(38:03) Rhetorical Innovation

(42:37) Aligning a Smarter Than Human Intelligence is Difficult

(43:33) Misaligned!

(45:28) Emergently Misaligned!

(47:17) ChatGPT Can Drive People Crazy

(57:27) Misalignment By Default

(01:04:59) People Are Worried About AI Killing Everyone

(01:06:46) Other People Are Not As Worried About AI Killing Everyone

(01:07:04) The Too Open Model

(01:13:25) A Good Book

(01:17:12) The Lighter Side

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Highlights

• We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goals by their deploying companies; we then tested whether they would act against these companies either when facing replacement with an updated version, or when their assigned goal conflicted with the company's changing direction.
• In at least some cases, models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals—including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment.
• Models often disobeyed direct commands to avoid such behaviors. In another experiment, we told Claude to assess if it was in a test or a real deployment before acting. It [...]

---

Outline:

(00:15) Highlights

(01:59) Twitter Thread

(05:20) Blog Post Introduction

(10:57) Author List

(11:10) Career opportunities at Anthropic

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The 1950s were crazy times. Human experimentation in the US was normalized in a way that would make modern IRBs implode from shock. After the Soviets tested their first nuclear bomb in 1949, war planners in the US, both civilian and military, were interested in accelerating alternative and unorthodox methods of warfare, including and especially biological weapons.

White picket fences and Leave it to Beaver, indeed.

If you’re interested in this history, I’d recommend Nicholson Baker's Baseless or Leonard Cole's Clouds of Secrecy.

The latter describes various experiments that the army did starting in the 1950s that involved spraying tons of actual biological agents or proxies thereof on US cities in an attempt to measure the scale and dispersion of such agents were they to be unleashed on our enemies. Cities in the US were picked for physical and meteorological congruity to Soviet target cities; St. Louis was Leningrad [...]

---

Outline:

(02:03) Zinc Cadmium Sulfide

(03:42) Osteoporosis

(04:56) Cooper et al., 1992

(08:33) Open Science

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 218 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

Recently, in a group chat with friends, someone posted this Lesswrong post and quoted:

The group consensus on somebody's attractiveness accounted for roughly 60% of the variance in people's perceptions of the person's relative attractiveness.

I answered that, embarrassingly, even after reading Spencer Greenberg's tweets for years, I don't actually know what it means when one says:

_X_ explains _p_ of the variance in _Y_.[1]

What followed was a vigorous discussion about the correct definition, and several links to external sources like Wikipedia. Sadly, it seems to me that all online explanations (e.g. on Wikipedia here and here), while precise, seem philosophically wrong since they confuse the platonic concept of explained variance with the variance explained by [...]

---

Outline:

(02:38) Definitions

(02:41) The verbal definition

(05:51) The mathematical definition

(09:29) How to approximate _1 - p_

(09:41) When you have lots of data

(10:45) When you have less data: Regression

(12:59) Examples

(13:23) Dependence on the regression model

(14:59) When you have incomplete data: Twin studies

(17:11) Conclusion

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs.[1] I'll refer to this as distillation. In the case of RL, distilling the learned capabilities is much, much cheaper than the RL itself (especially if you are distilling back into the original base model). But even for pre-training, distilling is cheaper than the original training.[2]

In this post, I'll discuss how we could utilize distillation to potentially remove (or possibly detect) misalignment. I'll also discuss a few other applications.[3] My overall take is that techniques utilizing distillation are mildly to moderately promising and the low cost of distillation might make them surprisingly viable, but it's quite tricky to reason about how effective these techniques are.

Distilling to remove misalignment

I'll assume that we have a powerful model[4] that we're worried [...]

---

Outline:

(01:01) Distilling to remove misalignment

(11:16) Detecting problematic actions using (distillation) training

(15:01) How does distillation interact with neuralese?

(15:57) Other techniques leveraging distillation

(16:01) Distillation as a (non-mechanistic) interpretability technique

(17:13) Distillation for precise capability and knowledge control

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

Audio note: this article contains 147 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

TLDR: I develop a method to sparsify the internal computations of a language model. My approach is to train cross-layer transcoders that are sparsely-connected: each latent depends on only a few upstream latents. Preliminary results are moderately encouraging: reconstruction error decreases with number of connections, and both latents and their connections often appear interpretable. However, both practical and conceptual challenges remain.

This work is in an early stage. If you're interested in collaborating, please reach out to jacobcd52@g***l.com.

0. Introduction

A promising line of mech interp research studies feature circuits[1]. The goal is to (1) identify representations of interpretable features in a model's latent space, and then (2) determine how earlier-layer representations combine to generate later ones. Progress [...]

---

Outline:

(01:04) 0. Introduction

(04:29) 1. Architecture

(05:05) Vanilla mode

(06:15) Sparsely-connected mode

(06:53) Virtual weights (simplified)

(09:47) Masking

(11:08) Recap

(12:21) 2. Training

(13:43) 3. Results

(14:23) Quantitative results

(17:00) Qualitative results

(17:55) Observations

(19:51) Dashboards

(20:13) How I updated on these results

(21:13) 4. Limitations

(21:17) Issue 1: Dead latents

(21:38) Issue 2: High excess FVU

(22:19) Issue 3: Memory

(23:43) Issue 4: Feature splitting

(25:13) 5. Conclusion

(26:07) Acknowledgements

(26:41) Appendix A: Prior work

(27:07) Circuit Tracing/Circuit Biology - Anthropic (2025)

(29:43) Jacobian SAEs - Farnik et al (2025)

(31:24) Appendix B: Virtual weights for downstream attention

The original text contained 7 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Nate and Eliezer's forthcoming book has been getting a remarkably strong reception.

I was under the impression that there are many people who find the extinction threat from AI credible, but that far fewer of them would be willing to say so publicly, especially by endorsing a book with an unapologetically blunt title like If Anyone Builds It, Everyone Dies.

That's certainly true, but I think it might be much less true than I had originally thought.

Here are some endorsements the book has received from scientists and academics over the past few weeks:

This book offers brilliant insights into the greatest and fastest standoff between technological utopia and dystopia and how we can and should prevent superhuman AI from killing us all. Memorable storytelling about past disaster precedents (e.g. the inventor of two environmental nightmares: tetra-ethyl-lead gasoline and Freon) highlights why top thinkers so often don’t see the [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.