Audio note: this article contains 182 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

I (Subhash) am a Masters student in the Tegmark AI Safety Lab at MIT. I am interested in recruiting for full time roles this Spring - please reach out if you're interested in working together!

TLDR 

This blog post accompanies the paper "Language Models Use Trigonometry to Do Addition." Key findings:

• We show that LLMs represent numbers on a helix
  • This helix has circular features (sines and cosines) with periods _T = [2,5,10,100]_ and a linear component
• To solve _a+b_, we claim that LLMs manipulate the helix for _a_ and _b_ to create the helix for _a+b_ using a form of the Clock algorithm introduced by Nanda, et al.
  • This is conceptually akin to going from _cos(a), cos(b)_ to _cos(a+b)_
  • Intuitively, it's like adding angles on a [...]

---

Outline:

(00:30) TLDR

(01:28) Motivation and Problem Setting

(02:18) LLMs Represent Numbers on a Helix

(02:23) Investigating the Structure of Numbers

(02:48) Periodicity

(03:49) Linearity

(04:39) Parameterizing Numbers as a Helix

(05:39) Fitting a Helix

(07:12) Evaluating the Helical Fit

(08:57) Relation to the Linear Representation Hypothesis

(10:28) Is the helix the full story?

(12:15) LLMs Use the Clock Algorithm to Compute Addition

(14:23) Understanding MLPs

(16:24) Zooming in on Neurons

(17:07) Modeling Neuron Preactivations

(18:53) Understanding MLP Inputs

(20:34) Interpreting Model Errors

(24:11) Limitations

(25:49) Conclusion

The original text contained 9 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Yeah I put this off until the last day, and I'm not sure this is the format Raemon was actually looking for. Oh well.

Then, in proportion to how valuable they seem, spend at least some time this month reflecting...

• ...on the big picture of what intellectual progress seems important to you. Do it whatever way is most valuable to you. But, do it publicly, this month, such that it helps encourage other people to do so as well. And ideally, do it with some degree of "looking back" – either of your own past work and how your views have changed, or how the overall intellectual landscape has changed.
• ...on how you wish incentives were different on LessWrong. Write up your thoughts on this post. (I suggest including both "what the impossible ideal" would be, as well as some practical ideas for how to improve them on [...]

---

Outline:

(01:23) 1. On the big picture of what intellectual progress seems important to you

(02:25) 2. On how I wish incentives could be different on LessWrong

(04:21) 3. On how the LessWrong community could make group progress

(06:55) 4. I propose an archipelago

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Table of Contents

The Pitch

---

Outline:

(00:20) The Pitch

(03:12) It's Coming

(05:01) Is It Safe?

(09:49) How Does Deep Research Work?

(10:47) Killer Shopping App

(12:17) Rave Reviews

(18:33) Research Reports

(31:21) Perfecting the Prompt

(32:26) Not So Fast!

(35:46) What's Next?

(36:59) Paying the Five

(37:59) The Lighter Side

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Meta just published its version of an RSP (blogpost, framework).

(Expect several more of these in the next week as AI companies try to look like they're meeting the Seoul summit commitments before the Paris summit deadline in a week.)

Summary, quotes, and commentary in progress.

We define catastrophic outcomes in two domains: Cybersecurity and Chemical & biological risks.

---

First published:

Source:

Narrated by TYPE III AUDIO.

In his recent post arguing against AI Control research, John Wentworth argues that the median doom path goes through AI slop, rather than scheming. I find this to be plausible. I believe this suggests a convergence of interests between AI capabilities research and AI alignment research.

Historically, there has been a lot of concern about differential progress amongst AI safety researchers (perhaps especially those I tend to talk to). Some research is labeled as "capabilities" while other is labeled as "safety" (or, more often, "alignment"[1]). Most research is dual-use in practice (IE, has both capability and safety implications) and therefore should be kept secret or disclosed carefully.

Recently, a colleague expressed concern that future AIs will read anything AI safety researchers publish now. Since the alignment of future AIs seems uncertain and even implausible, almost any information published now could be net harmful for the future.

I argued the [...]

---

Outline:

(02:42) AI Slop

(04:46) Coherence and Recursive Self-Improvement

(08:00) Whats to be done?

The original text contained 10 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

New model, new hype cycle, who dis?

On a Friday afternoon, OpenAI was proud to announce the new model o3-mini and also o3-mini-high which is somewhat less mini, or for some other reasoning tasks you might still want o1 if you want a broader knowledge base, or if you’re a pro user o1-pro, while we want for o3-not-mini and o3-pro, except o3 can use web search and o1 can’t so it has the better knowledge in that sense, then on a Sunday night they launched Deep Research which is different from Google's Deep Research but you only have a few of those queries so make them count, or maybe you want to use operator?

Get it? Got it? Good.

Yes, Pliny jailbroke o3-mini on the spot, as he always does.

This most mostly skips over OpenAI's Deep Research (o3-DR? OAI-DR?). I need more time for [...]

---

Outline:

(01:16) Feature Presentation

(04:37) QandA

(09:14) The Wrong Side of History

(13:29) The System Card

(22:08) The Official Benchmarks

(24:55) The Unofficial Benchmarks

(27:43) Others Report In

(29:47) Some People Need Practical Advice

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Today we’re launching deep research in ChatGPT, a new agentic capability that conducts multi-step research on the internet for complex tasks. It accomplishes in tens of minutes what would take a human many hours.

Deep research is OpenAI's next agent that can do work for you independently—you give it a prompt, and ChatGPT will find, analyze, and synthesize hundreds of online sources to create a comprehensive report at the level of a research analyst. Powered by a version of the upcoming OpenAI o3 model that's optimized for web browsing and data analysis, it leverages reasoning to search, interpret, and analyze massive amounts of text, images, and PDFs on the internet, pivoting as needed in reaction to information it encounters.

The ability to synthesize knowledge is a prerequisite for creating new knowledge. For this reason, deep research marks a significant step toward our broader goal of [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There once was a post on LessWrong asking why so many bans? The OP of that post complained of, among other things, being punished despite not having broken any rules that they knew about before taking the action they got punished for. I started to write a comment, which then ballooned out of scope. The following is based on my own experiences trying to write rules.

1.

I claim the ideal rules are concise, comprehensive, and clear[1]. In practice, I think that we usually have to pick two.

Concise: Ideal rules are short. You can read or refer to them easily.

Comprehensive: Ideal rules include everything relevant. There aren't gaps.

Clear: Ideal rules have one interpretation. There's no ambiguity.

If you're running LessWrong, then "Don't be a jerk, talk about things related to rationality" is a concise and fairly comprehensive set of rules. It's just not a very [...]

---

Outline:

(00:29) 1.

(05:44) 2.

(07:13) 3.

(10:46) 4.

(13:42) 5.

The original text contained 2 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

The sunlight hitting even a small portion of the United States has enough energy to power the whole country—that simple calculation was the subject of the last post. All this shows on its own is that running the country on solar power is physically possible. With photovoltaics it may be not just possible but practical, but photovoltaics on their own left open the question of what to do at night. Now it seems that the answer may be battery energy storage. Here's Tesla's sales of battery energy storage systems again, but updated since my last post to include 2024 Q4:

It's natural and popular to believe that solar will either "win" or "lose", that declining panel and battery prices will bankrupt the competition or that they'll be revealed as a ruinous environmentalist fad. The reason I say this dichotomy is natural is because naively it seems that the solar+batteries [...]

---

Outline:

(02:32) The economic niche of battery energy storage

(05:59) Economic comparison to natural gas peakers

(09:16) The niche pays better

(10:50) The near future of electricity generation

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.