On Carcinogenic Complexity, Software Senescence and Cognitive Provenance: Our roadmap for 2025 and beyond

It is mandatory to start any essay on AI in the post-ChatGPT era with the disclaimer that AI brings huge potential, and great risks. Unfortunately, on the path we are currently on, we will not realize those benefits, but are far more likely to simply drown in terrible AI slop, undermine systemic cybersecurity and blow ourselves up.

We believe AI on its current path will continue to progress exponentially, to the point where it can automate, and summarily replace, all of humanity. We are unlikely to survive such a transition.

Powerful technology always comes with powerful risks, but this does not mean we have to wait idly for the other shoe to drop. Risks can be managed and prevented, while harnessing the benefits. We have done it with aviation, nuclear and other risky technologies [...]

---

Outline:

(01:47) Part 1: Cognitive Software

(03:01) AI, what is it good for?

(04:37) A Tale of a Database

(06:35) The Punchline Should Be Obvious

(08:36) Part 2: AI Slop and Complexity Debt

(12:02) The Strange Decoupling and Software Senescence

(15:33) Algorithmic Carcinogenesis

(20:53) Neither New Nor Unsolvable

(28:20) Part 3: A Better Future and How to Get There: A Roadmap

(29:41) Phase 1: Foundational Infrastructure

(32:52) Phase 2: Cognitive Language Design and Computational Control

(38:47) Phase 3: Horizontal Scaling: Scaling Without Sprawling

(44:31) Phase 4: Vertical Scaling: Deliberate Teaching and Cognitive Provenance

(48:19) Phase 5: Cognitive Emulation (CoEm)

(51:51) Conclusion: A Future Worth Building

The original text contained 3 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

The Less Wrong General Census is unofficially here! You can take it at this link.

The oft-interrupted annual tradition of the Less Wrong Census is once more upon us!

If you are reading this post and identify as a LessWronger, then you are the target audience. If you are reading this post and don't identify as a LessWronger, you just read posts here or maybe go to house parties full of rationalists or possibly read rationalist fanfiction and like talking about it on the internet, or you're not a rationalist you're just, idk, adjacent, then you're also the target audience.

If you want to just spend five minutes answering the basic demographics questions before leaving the rest blank and hitting submit, that's totally fine. The survey is structured so the fastest and most generally applicable questions are (generally speaking) towards the start. At any point you can scroll to [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Two months ago I attended Eric Drexler's launch of

Drexler wants to draw more attention to the benefits of nanotech, which

He hopes that a decentralized network of users will create a rich

Drexler hopes to gamify the [...]

---

Outline:

(01:24) Protein-based nanotech versus Diamondoid-style

(02:52) Nanotech Risks

(03:55) Awareness

(05:22) Who would we convince?

(06:32) Weighing the Tradeoffs

---

First published:

Source:

Narrated by TYPE III AUDIO.

403 Forbidden

---

Source:

Narrated by TYPE III AUDIO.

YouTube link

The ‘model organisms of misalignment’ line of research creates AI models that exhibit various types of misalignment, and studies them to try to understand how the misalignment occurs and whether it can be somehow removed. In this episode, Evan Hubinger talks about two papers he's worked on at Anthropic under this agenda: “Sleeper Agents” and “Sycophancy to Subterfuge”.

Topics we discuss:

Daniel Filan:

---

Outline:

(01:46) Model organisms and stress-testing

(09:02) Sleeper Agents

(25:18) Do ‘sleeper agents’ properly model deceptive alignment?

(42:08) Surprising results in “Sleeper Agents”

(01:02:51) Sycophancy to Subterfuge

(01:15:27) How models generalize from sycophancy to subterfuge

(01:23:27) Is the reward editing task valid?

(01:28:53) Training away sycophancy and subterfuge

(01:36:42) Model organisms, AI control, and evaluations

(01:41:12) Other model organisms research

(01:43:11) Alignment stress-testing at Anthropic

(01:51:07) Following Evan's work

---

First published:

Source:

Narrated by TYPE III AUDIO.

TLDR: LessWrong + Lighthaven need about $3M for the next 12 months. Donate here, or send me an email, DM or signal message (+1 510 944 3235) if you want to support what we do. Donations are tax-deductible in the US. Reach out for other countries, we can likely figure something out. We have big plans for the next year, and due to a shifting funding landscape we need support from a broader community more than in any previous year.

I've been running LessWrong/Lightcone Infrastructure for the last 7 years. During that time we have grown into the primary infrastructure provider for the rationality and AI safety communities. "Infrastructure" is a big fuzzy word, but in our case, it concretely means:

• We build and run LessWrong.com and the AI Alignment Forum.[1]
• We built and run Lighthaven (lighthaven.space), a ~30,000 sq. ft. campus in downtown Berkeley where we [...]

---

Outline:

(03:52) LessWrong

(06:36) Does LessWrong influence important decisions?

(09:37) Does LessWrong make its readers/writers more sane?

(11:37) LessWrong and intellectual progress

(19:08) Lighthaven

(22:04) The economics of Lighthaven

(24:26) How does Lighthaven improve the world?

(28:41) The relationship between Lighthaven and LessWrong

(30:36) Lightcone and the funding ecosystem

(35:17) Our work on funding infrastructure

(37:57) If its worth doing its worth doing with made-up statistics

(38:44) The OP GCR capacity building team survey

(42:09) Lightcone/LessWrong cannot be funded by just running ads

(43:55) Comparing LessWrong to other websites and apps

(45:00) Lighthaven event surplus

(47:13) The future of (the) Lightcone

(48:02) Lightcone culture and principles

(50:04) Things I wish I had time and funding for

(59:31) What do you get from donating to Lightcone?

(01:02:03) Tying everything together

The original text contained 22 footnotes which were omitted from this narration.

The original text contained 7 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

TLDR: In this post, I argue that if you are a junior AI safety researcher, you should consider applying to PhD programs in ML soon, especially if you have recently participated in an AI safety upskilling or research program like MATS or ARENA, might be interested in working on AI safety long term, but don't have immediate career plans. It is relatively cheap to apply, and provides good future option value. I don’t argue that you should necessarily do a PhD, but some other posts do. PhD application deadlines are coming up soon; many application deadlines are December 15th, though some are earlier (as early as next week). For the uninitiated, I provide a step by step guide to applying at the end of this post.

Applying to PhD programs might, in expectation, be worth your time.

This might be true even if you are not [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

Note: This is meant to be an accessible introduction to causal inference. Comments appreciated.

Let's say you buy a basil plant and put it on the counter in your kitchen. Unfortunately, it dies in a week.

So the next week you buy another basil plant and feed it a special powder, Vitality Plus. This second plant lives. Does that mean Vitality Plus worked?

Not necessarily! Maybe the second week was a lot sunnier, you were better about watering, or you didn’t grab a few leaves for a pasta. In other words, it wasn’t a controlled experiment. If some other variable like sun, water, or pasta is driving the results you’re seeing, your study is confounded, and you’ve fallen prey to a core issue in science.

When someone says “correlation is not causation,” they’re usually talking about confounding. Here are some examples:

• A 2019 study found that student test [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

There are lots of great charitable giving opportunities out there right now.

The first time that I served as a recommender in the Survival and Flourishing Fund (SFF) was back in 2021. I wrote in detail about my experiences then. At the time, I did not see many great opportunities, and was able to give out as much money as I found good places to do so.

How the world has changed in three years.

I recently had the opportunity to be an SFF recommender for the second time. This time I found an embarrassment of riches. Application quality was consistently higher, there were more than twice as many applications, and essentially all applicant organizations were looking to scale their operations and spending.

That means the focus of this post is different. In 2021, my primary goal was to share my perspective on [...]

---

Outline:

(01:39) A Word of Warning

(02:44) Use Your Personal Theory of Impact

(04:13) Use Your Local Knowledge

(05:10) Unconditional Grants to Worthy Individuals Are Great

(06:55) Do Not Think Only On the Margin, and Also Use Decision Theory

(07:48) And the Nominees Are

(10:55) Organizations that Are Literally Me

(11:10) Balsa Research

(12:56) Don’t Worry About the Vase

(14:19) Organizations Focusing On AI Non-Technical Research and Education

(14:37) The Scenario Project

(15:48) Lightcone Infrastructure

(17:20) Effective Institutions Project (EIP)

(18:06) Artificial Intelligence Policy Institute (AIPI)

(19:10) Psychosecurity Ethics at EURAIO

(20:07) Pallisade Research

(21:07) AI Safety Info (Robert Miles)

(21:51) Intelligence Rising

(22:32) Convergence Analysis

(23:29) Longview Philanthropy

(24:27) Organizations Focusing Primary On AI Policy and Diplomacy

(25:06) Center for AI Safety and the CAIS Action Fund

(26:00) MIRI

(26:59) Foundation for American Innovation (FAI)

(28:58) Center for AI Policy (CAIP)

(29:58) Encode Justice

(30:57) The Future Society

(31:42) Safer AI

(32:26) Institute for AI Policy and Strategy (IAPS)

(33:13) AI Standards Lab

(34:05) Safer AI Forum

(34:40) CLTR at Founders Pledge

(35:54) Pause AI and Pause AI Global

(36:57) Existential Risk Observatory

(37:37) Simons Institute for Longterm Governance

(38:21) Legal Advocacy for Safe Science and Technology

(39:17) Organizations Doing ML Alignment Research

(40:16) Model Evaluation and Threat Research (METR)

(41:28) Alignment Research Center (ARC)

(42:02) Apollo Research

(42:53) Cybersecurity Lab at University of Louisville

(43:44) Timaeus

(44:39) Simplex

(45:08) Far AI

(45:41) Alignment in Complex Systems Research Group

(46:23) Apart Research

(47:06) Transluce

(48:00) Atlas Computing

(48:45) Organizations Doing Math, Decision Theory and Agent Foundations

(50:05) Orthogonal

(50:47) Topos Institute

(51:37) Eisenstat Research

(52:13) ALTER (Affiliate Learning-Theoretic Employment and Resources) Project

(53:00) Mathematical Metaphysics Institute

(54:06) Focal at CMU

(55:15) Organizations Doing Cool Other Stuff Including Tech

(55:26) MSEP Project at Science and Technology Futures (Their Website)

(56:26) ALLFED

(57:51) Good Ancestor Foundation

(59:10) Charter Cities Institute

(59:50) German Primate Center (DPZ) – Leibniz Institute for Primate Research

(01:01:08) Carbon Copies for Independent Minds

(01:01:44) Organizations Focused Primarily on Bio Risk

(01:01:50) Secure DNA

(01:02:46) Blueprint Biosecurity

(01:03:35) Pour Domain

(01:04:17) Organizations That then Regrant to Fund Other Organizations

(01:05:14) SFF Itself (!)

(01:06:10) Manifund

(01:08:02) AI Risk Mitigation Fund

(01:08:39) Long Term Future Fund

(01:10:16) Foresight

(01:11:08) Centre for Enabling Effective Altruism Learning and Research (CEELAR)

(01:11:43) Organizations That are Essentially Talent Funnels

(01:13:40) AI Safety Camp

(01:14:23) Center for Law and AI Risk

(01:15:22) Speculative Technologies

(01:16:19) Talos Network

(01:17:11) MATS Research

(01:17:48) Epistea

(01:18:52) Emergent Ventures (Special Bonus Organization, was not part of SFF)

(01:20:32) AI Safety Cape Town

(01:21:08) Impact Academy Limited

(01:21:47) Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS)

(01:22:34) Tarbell Fellowship at PPF

(01:23:32) Catalyze Impact

(01:24:32) Akrose

(01:25:14) CeSIA within EffiSciences

(01:25:59) Stanford Existential Risk Initiative (SERI)

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

AI infrastructure numbers are hard to find with any precision. There are many reported numbers of “[company] spending Xbn on infrastructure this quarter” and “[company] has bought 100k H100s or “has a cluster of 100k H100s” but when I went looking for an estimate of how much compute a given company had access to, I could not find consistent numbers available. Here I’ve tried to pull together information from a variety of sources to get ballpark estimates of (i) as of EOY 2024, who do we expect to have how much compute? and (ii) how do we expect that to change in 2025? I then spend a little time talking about what that might mean for training compute availability at the main frontier labs. Before going into this, I want to lay out a few caveats:

• These numbers are all estimates I’ve made from publicly available data, in [...]

---

Outline:

(01:36) Nvidia chip production

(02:38) 2024 production

(03:32) Previous production

(04:06) GPU/TPU counts by organisation

(06:00) Estimating H100 equivalent chip counts at year end 2024

(06:06) Microsoft, Meta

(08:03) Google, Amazon

(10:46) XAI

(11:08) 2025 - Blackwell

(13:53) Summary of estimated chip counts

(14:17) Model training notes

The original text contained 13 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.