Executive Summary

• Refusing harmful requests is not a novel behavior learned in chat fine-tuning, as pre-trained base models will also refuse requests (48% of all harmful requests, 3% of harmless) just at a lower rate than chat models (90% harmful, 3% harmless)
• Further, for both Qwen 1.5 0.5B and Gemma 2 9B, chat fine-tuning reinforces the existing mechanisms. In both the chat and base models it is mediated by the refusal direction described in Arditi et al.
  • We can both induce and bypass refusal in a pre-trained model, using a steering vector transferred from the chat model's activations
  • On the contrary, in LLaMA 1 7B (which was trained on data from before November 2022 and so can't have had ChatGPT outputs in the pre-training data), we find evidence that chat fine-tuning learns additional / different refusal representations and mechanisms.
• We open source our code at https://github.com/ckkissane/base-models-refuse

---

Outline:

(00:06) Executive Summary

(01:25) Introduction

(03:23) Background and methodology

(06:30) Results

(06:33) Base models refuse harmful requests

(08:21) Eliciting more base model refusals with steering vectors

(11:00) Bypassing refusal in base models

(12:22) Investigating (pre-ChatGPT model) LLaMA 1 7B

(16:32) Related work

(17:44) Conclusion

(19:50) Citing this work

(20:04) Author contributions statement

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This is Shapley Value Calculator breaks down the formula in simple steps, using cats.

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

There has been a renewal of discussion on how much hope we should have of an unaligned AGI leaving humanity alive on Earth after a takeover. When this topic is discussed, the idea of using simulation arguments or acausal trade to make the AI spare our lives often come up. These ideas have a long history. The first mention I know of comes from Rolf Nelson in 2007 on an SL4 message board, the idea later makes a brief appearance in Superintelligence under the name of Anthropic Capture, and came up on LessWrong last time as recently as a few days ago. In response to these, Nate Soares wrote Decision theory does not imply that we get to have nice things, arguing that decision theory is not going to save us, and that we can't bamboozle a superintelligence into submission by clever simulation arguments.

However, none of the [...]

---

Outline:

(01:15) A proposal for humanity in the Future

(03:50) What does the AI do?

(05:42) Is this the same as acausal trade?

(07:32) Response to Nates arguments in his post

(12:05) Nates arguments in the comments

(12:54) 1. We might just have a very low chance of solving alignment, so the AI doesnt need to take seriously the possibility of humans simulating it.

(14:12) 2. The successful human civilization would need to guess correctly what random thing an AI developing in a different Universe branch might value, and this is possibly infeasible.

(15:17) 3. Maybe the successful human civilization could pay for our salvation, but they will choose to spend their resources on other things.

(18:25) What should we bargain for?

(21:49) Can we get more than this?

(24:49) Other possible types of AI values

(25:23) Ethical considerations

(26:54) Ways this hope could fail

(30:24) 2. The true cost might be not just a planet, but the delay.

(32:44) 3. It might be very hard to create simulations that are indistinguishable from reality.

(33:18) 4. There are just too many possible simulators out there with too many different goals.

(37:51) 6. The AI just doesnt take the simulation hypothesis seriously.

(38:35) 7. A lot of people might be killed during takeover.

(39:44) Are we in a simulation? What should we do?

(43:34) Conclusion

The original text contained 34 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

I've been wanting to write a nice post for a few months, but should probably just write a one sooner instead. This is a top-level post not because it's a long text, but because it's important text.

Anyways. Cryonics is pretty much money-free now—one of the most affordable ways to dispose of your body post-mortem.

In the west coast in the USA, from Oregon Brain Preservation, as of around May 2024 I think:

Our research program is open to individuals in Washington, Oregon, and Northern California. This is the same brain preservation procedure, with the goal of future revival if this ever becomes feasible and humane. The difference is that we will also remove two small biopsy samples, which will be transferred to our partner non-profit organization, Apex Neuroscience, to measure the preservation quality and contribute to neuroscience research. Although there are no guarantees, we do not expect these [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.

At least, if you happen to be near me in brain space.

What advice would you give your younger self?

That was the prompt for a class I taught at PAIR 2024. About a quarter of participants ranked it in their top 3 of courses at the camp and half of them had it listed as their favorite.

I hadn’t expected that.

I thought my life advice was pretty idiosyncratic. I never heard of anyone living their life like I have. I never encountered this method in all the self-help blogs or feel-better books I consumed back when I needed them.

But if some people found it helpful, then I should probably write it all down.

Why Listen to Me Though?

I think it's generally worth prioritizing the advice of people who have actually achieved the things you care about in life. I can’t tell you if that's me [...]

---

Outline:

(00:46) Why Listen to Me Though?

(04:22) Pick a direction instead of a goal

(12:00) Do what you love but always tie it back

(17:09) When all else fails, apply random search

The original text contained 3 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Audio note: this article contains 38 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.

This post has some ablation results around the thesis of the ICML 2024 Mech. Interp. workshop 1st prize winning paper: The Geometry of Categorical and Hierarchical Concepts in Large Language Models The main takeaway is that the orthogonality they observe in categorical and hierarchical concepts occurs practically everywhere, even at places where it really should not.

Overview of the original paper

A lot of the intuition and math around why they do what they do is shared in their previous work called The Linear Representation Hypothesis and the Geometry of Large Language Models, but let's quickly go over what the paper's core idea is:

They split the computation of a large language model (LLM) as:

_P(y mid [...]

---

Outline:

(02:42) Ablations

(03:37) Hierarchical features are orthogonal - but so are semantic opposites!?

(04:51) Categorical features form simplices - but so do totally random ones!?

(05:27) Orthogonality being ubiquitous in high dimensions

(06:39) Discussion and Future Work

The original text contained 4 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

For a long time, when I heard "slow takeoff", I assumed it meant "takeoff that takes longer calendar time than fast takeoff." (i.e. what is now referred to more often as "short timelines" vs "long timelines."). I think Paul Christiano popularized the term, and it so happened he both expected to see longer timelines and smoother/continuous takeoff.

I think it's at least somewhat confusing to use the term "slow" to mean "smooth/continuous", because that's not what "slow" particularly means most of the time.

I think it's even more actively confusing because "smooth/continuous" takeoff not only could be faster in calendar time, but, I'd weakly expect this on average, since smooth takeoff means that AI resources at a given time are feeding into more AI resources, whereas sharp/discontinuous takeoff would tend to mean "AI tech doesn't get seriously applied towards AI development until towards the end."

I don't think this [...]

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Part 0: "Previously, on Petrov Day..."

By Raymond Arnold

One year ago, many people on LessWrong received a DM asking them to choose the most important virtue of Petrov Day, with four listed options that we'd seen people argue for in previous years.

1. "Avoiding actions that noticeably increase chance that civilization is destroyed"
2. "Accurately reporting your epistemic state"
3. "Quickly orienting to novel situations"
4. "Resisting social pressure."

Then, people who chose one of those options were sent another message saying "Your virtue was in the minority, but you have the power to unilaterally choose the virtue of next year's Petrov Day by clicking a link."

The first person to click the Unilateral Virtue Link was a proponent of "Avoiding actions that noticeably increase the chance that the world will end." But, this virtue was actually in the majority[1]. The first unilateralist of a Virtue Minority was a proponent [...]

---

Outline:

(00:05) Part 0: Previously, on Petrov Day...

(00:12) By Raymond Arnold

(03:01) Part I: The 2024 Petrov Day Scenario

(03:07) By Ben Pace

(09:50) Part II: Stats and Feedback Form

(12:18) Part IV: Reflections

(12:23) Reflections by Raymond Arnold

(16:59) Reflections by Ben Pace

The original text contained 6 footnotes which were omitted from this narration.

The original text contained 14 images which were described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

This graph is the biggest update to the AI alignment discourse since GPT-3

For those of you unfamiliar with the lore, prior to GPT-3, the feeling was that AGI would rapidly foom based on recursive-self improvement.

After GPT-3, it became clear that the first AGI would in reality be built in a large lab using a multi-billion dollar supercomputer and any idea that it simply "copy itself to the internet" is nonsense.

Under the GPT-3 regime, however, it was still plausible to assume that the first AGI would be able to simulate millions of human beings. This is because the training cost for models like GPT-3/4 is much higher than the inference cost.

However, COT/o1 reveals this is not true. Because we can scale both training and inference, the first AGI will not only cost billions of dollars to train, it will also cost millions of dollars to run [...]

The original text contained 1 image which was described by AI.

---

First published:

Source:

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Previously: The Fundamentals, The Gamblers, The Business

We have now arrived at the topics most central to this book, aka ‘The Future.’

Rationalism and Effective Altruism (EA)

The Manifest conference was also one of the last reporting trips that I made for this book. And it confirmed for me that the River is real—not just some literary device I invented. (6706)

Yep. The River is real.

I consider myself, among many things, a straight up rationalist.

I do not consider myself an EA, and never have.

This completes the four quadrants of the two-by-two of [does Nate knows it well, does Zvi knows it well]. The first two, where Nate was in his element, went very well. The third clearly was less exacting, as one would expect, but pretty good.

Now I have the information advantage, even more than I did [...]

---

Outline:

(00:16) Rationalism and Effective Altruism (EA)

(06:01) Cost-Benefit Analysis

(09:04) How About Trying At All

(10:11) The Virtues of Rationality

(11:56) Effective Altruism and Rationality, Very Different of Course

(24:37) The Story of OpenAI

(30:19) Altman, OpenAI and AI Existential Risk

(38:26) Tonight at 11: Doom

(01:00:39) AI Existential Risk: They’re For It

(01:07:42) To Pause or Not to Pause

(01:11:11) You Need Better Decision Theory

(01:15:27) Understanding the AI

(01:19:43) Aligning the AI

(01:23:50) A Glimpse of Our Possible Future

(01:28:16) The Closing Motto

---

First published:

Source:

Narrated by TYPE III AUDIO.