tl;dr: LLMs rapidly improving at software engineering and math means lots of projects are better off as Google Docs until your AI agent intern can implement them.

Implementation keeps getting cheaper

Writing research code has gotten a lot faster over the past few years. Since 2021 and OpenAI Codex, new models and tools such as Cursor built around them have saved myself more and more time on coding every year.

This trend is accelerating fast: AI agents using Claude-3.5-Sonnet and o1-preview can do tasks that take ML researchers up to 2 hours of coding. This is without considering newer models such as o3, which do 70% on SWE-bench out of the box.
Yet this progress remains somewhat concentrated in implementation: progress on “soft” skills like idea generation has, as far as I can tell, been slower.

I’ve come to believe that, if you work in technical [...]

---

Outline:

(00:18) Implementation keeps getting cheaper

(01:33) When later means better

(03:01) Did I do well in 2024?

(07:29) Themes for temporally privileged work

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

TL;DR: There may be a fundamental problem with interpretability work that attempts to understand neural networks by decomposing their individual activation spaces in isolation: It seems likely to find features of the activations - features that help explain the statistical structure of activation spaces, rather than features of the model - the features the model's own computations make use of.

Written at Apollo Research

Introduction

Claim: Activation space interpretability is likely to give us features of the activations, not features of the model, and this is a problem.

Let's walk through this claim.

What do we mean by activation space interpretability? Interpretability work that attempts to understand neural networks by explaining the inputs and outputs of their layers in isolation. In this post, we focus in particular on the problem of decomposing activations, via techniques such as sparse autoencoders (SAEs), PCA, or just by looking at individual neurons. This [...]

---

Outline:

(00:33) Introduction

(02:40) Examples illustrating the general problem

(12:29) The general problem

(13:26) What can we do about this?

The original text contained 11 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

A while back I gave a talk about doing AI safety as a YC startup. I wrote a blog post about it and thought it would be interesting to share it with both the YC and AI safety communities. Please share any feedback or thoughts. I would love to hear them!

AI Safety is a problem and people pay to solve problems

Intelligence is dangerous, and I think there's a significant chance that the default scenario of AI progress poses an existential risk to humanity. While it's far from certain, even small probabilities are significant when the stakes are this high. This is an enormous abstract problem, but there are thousands of sub-problems waiting to be solved.

Some of these sub-problems already exist today, but most are in the future (GPT is not capable of killing us yet). When people start feeling these pains, they will [...]

---

Outline:

(00:22) AI Safety is a problem and people pay to solve problems

(01:24) More startups should solve these problems

(02:39) Y Combinator

(04:23) YC advice in the context of AI safety

(04:39) The problems they are solving are in the future

(05:13) Customers are often researchers from AI labs or from the government

(05:54) The pool of potential customers is often small

(06:54) Doing good

---

First published:

Source:

Narrated by TYPE III AUDIO.

Our research is centered on empirical research with LLMs. So if you are doing something similar, these tips on slide-based communication may be helpful!

Background:

James Chua and John Hughes are researchers working under Owain Evans and Ethan Perez, respectively. Both of us (James and John) used to be MATS mentees. We weren't good at making research slides at first -- here are some principles we've found useful for understandable slides for our weekly research meetings.

We show some good example slides. We also show examples of confusing slides we've made — marked in the caption with “❌ Negative example”.
Below we use the study of sycophancy as an example. Sycophancy occurs when a model has responses that match user beliefs rather than truthful ones.

Summary slide sets the frame

---

Outline:

(00:58) Summary slide sets the frame

(02:22) Include an agenda

(03:27) Simple charts to describe experiments

(07:30) Backup slides - Be ready for questions

(10:01) End with concrete discussion points

(10:19) Keep one slide deck per project

(10:48) Get consistent feedback on the story of your paper

(11:18) Ask your friends

(11:44) Investing time is worth it

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

"Is this…enlightenment?" I asked.

"We use the word 'Awakening'," said the Zen Master, kindly.

"Now what?" I asked, "Every book I've read about meditation is about getting to this point. There's nothing about how to safely navigate the terirtory afterward―or what to do afterward―or even what happens afterward."

"The first thing you need to know is that you experienced was real," the Zen Master spoke from personal experience and that of his many previous students, "Don't make it too complicated. Just be in the moment. When you're cooking, then just cook. When you're eating, then just eat."

It is impossible to communicate the subjective experience of Stream Entry to someone who has not personally experienced it. However, it is possible to communicate an outside mechanistic model of what happens in your brain. First I will explain how a human brain functions normally. Then I will [...]

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is an interim research report on role embeddings, an approach to make language models more robust to many-shot jailbreaks and prompt injections by adding role information at every token position in the context rather than just at special token delimiters. We credit Cem Anil for originally proposing this idea.

In our initial experiments on Llama 3, we find that role embeddings mitigate many-shot jailbreaks more effectively than fine-tuning alone without degrading general model capabilities, which demonstrates that this technique may be a viable way to increase LLM robustness. However, more work should to be done to find the optimal set of hyperparameters and fully understand any side-effects of our proposed approach.

Background on prompt formats

By default, chat LLMs are trained (during instruction fine-tuning and RLHF) using a particular prompt format that distinguishes different message "roles". Almost all chat LLMs accept some version of system, user, and assistant. [...]

---

Outline:

(00:49) Background on prompt formats

(05:27) Where role embeddings come in

(06:29) Our experiments

(06:44) Datasets

(08:40) Intervention

(09:25) We assess

(10:10) Baseline

(10:37) Inference-time only: adding a role vector to token embeddings

(12:14) Using fine-tuning to improve performance

(12:49) Key fine-tuning results

(13:11) Residual-stream coloring

(16:20) Embedding coloring

(17:51) Next steps

(20:32) Acknowledgements

The original text contained 2 footnotes which were omitted from this narration.

The original text contained 15 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

(Both characters are fictional, loosely inspired by various traits from various real people. Be careful about combining kratom and alcohol.)

The original text contained 24 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Introduction

I’ll write a series of posts in which I'll introduce alternative cancer care. I’ll explain why it can be a rigorous form of biohacking rather than mere quackery. I’ll review books popular in the alternative cancer care world like: Surviving Terminal Cancer by Ben Williams and How to Starve Cancer by Jane McLelland, both written by cancer survivors who overcame terminal stage 4 cancer. I’ll review the historical development of combination chemotherapy. I’ll review some famous cases e.g. the ‘spontaneous remission’ of Joe Tippens and the virologist Beta Halassy curing herself via self-experimentation. And I’ll review select phase 1/2 clinical trials. Throughout, I’ll criticize the FDA and the medical establishment for being too risk-averse, for having the wrong framework for clinical trials and having a frustratingly closed-minded epistemology.

This is also my first post on this website so I welcome any (additional) feedback. And last, I don’t [...]

---

Outline:

(00:07) Introduction

(01:24) What is Alternative Cancer Care?

(02:17) Summary of My Perspective.

(05:44) Book Review: Surviving Terminal Cancer; Clinical Trials, Drug Cocktails, And Other Treatments Your Oncologist Won't Tell You About

(07:14) Prof. Williams' Case History and Development of His Drug Cocktail

(17:14) The Drug Cocktail Treatment Philosophy

(20:54) Criticisms of the Medical Establishment

(25:20) Conclusion

---

First published:

Source:

Narrated by TYPE III AUDIO.

This week, Altman offers a post called Reflections, and he has an interview in Bloomberg. There's a bunch of good and interesting answers in the interview about past events that I won’t mention or have to condense a lot here, such as his going over his calendar and all the meetings he constantly has, so consider reading the whole thing.

Table of Contents

The Battle of the Board

Here is what he says about the Battle of the Board in Reflections:

Sam Altman: A little over a year ago, on one particular Friday, the main thing that had gone wrong that day was [...]

---

Outline:

(00:25) The Battle of the Board

(05:12) Altman Lashes Out

(07:48) Inconsistently Candid

(09:35) On Various People Leaving OpenAI

(10:56) The Pitch

(12:07) Great Expectations

(12:56) Accusations of Fake News

(15:02) OpenAI's Vision Would Pose an Existential Risk To Humanity

---

First published:

Source:

Narrated by TYPE III AUDIO.

Epistemic Status: This post is an attempt to condense some ideas I've been thinking about for quite some time. I took some care grounding the main body of the text, but some parts (particularly the appendix) are pretty off the cuff, and should be treated as such.

The magnitude and scope of the problems related to AI safety have led to an increasingly public discussion about how to address them. Risks of sufficiently advanced AI systems involve unknown unknowns that could impact the global economy, national and personal security, and the way we investigate, innovate, and learn. Clearly, the response from the AI safety community should be as multi-faceted and expansive as the problems it aims to address. In a previous post, we framed fruitful collaborations between applied science, basic science, and governance as trading zones mediated by a safety-relevant boundary object (a safety case sketch) without discussing [...]

---

Outline:

(02:35) What is Big Science?

(05:53) Paths toward a Big Science of AI Safety

(06:38) The Way Things Are

(17:42) Why bottom-up could be better

(21:43) How to scale from where we are now

(23:40) Appendix: Potential Pushback of this Approach

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.