Two Minute Summary

In this post I present my results from training a Sparse Autoencoder (SAE) on a CLIP Vision Transformer (ViT) using the ImageNet-1k dataset. I have created an interactive web app, 'SAE Explorer', to allow the public to explore the visual features the SAE has learnt, found here: https://sae-explorer.streamlit.app/ (best viewed on a laptop). My results illustrate that SAEs can identify sparse and highly interpretable directions in the residual stream of vision models, enabling inference time inspections on the model's activations. To demonstrate this, I have included a 'guess the input image' game on the web app that allows users to guess the input image purely from the SAE activations of a single layer and token of the residual stream. I have also uploaded a (slightly outdated) accompanying talk of my results, primarily listing SAE features I found interesting: https://youtu.be/bY4Hw5zSXzQ.

The primary purpose of this post is [...]

---

Outline:

(00:08) Two Minute Summary

(02:44) Motivation

(04:18) What is a Vision Transformer?

(06:01) What is CLIP?

(07:42) Training the SAE

(08:43) Examples of SAE Features

(09:04) Interesting and Amusing Features

(09:19) Era/Time Features:

(09:48) Place/Culture Features:

(10:24) Film/TV Features:

(10:41) Texture Features:

(10:49) Miscellaneous Features:

(11:07) NSFW Features:

(11:47) How Trustworthy are Highest Activating Images?

(12:27) Tennis Feature

(12:46) Border Terrier Feature

(12:52) Mushrooms Feature

(12:58) Birds on Branches/in Foliage

(13:05) Training Performance

(13:08) Sparsity and _l_0_

(13:56) MSE, _l_1_ and Model Losses

(14:26) Identifying the Ultra-Low Density Cluster

(18:58) Neuron Alignment

(21:56) Future Work

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Adversarial Examples: A Problem

The apparent successes of the deep learning revolution conceal a dark underbelly. It may seem that we now know how to get computers to (say) check whether a photo is of a bird, but this façade of seemingly good performance is belied by the existence of adversarial examples—specially prepared data that looks ordinary to humans, but is seen radically differently by machine learning models.

The differentiable nature of neural networks, which make them possible to be trained at all, are also responsible for their downfall at the hands of an adversary. Deep learning models are fit using stochastic gradient descent (SGD) to approximate the function between expected inputs and outputs. Given an input, an expected output, and a loss function (which measures "how bad" it is for the actual output to differ from the expected output), we can calculate the gradient of the [...]

---

Outline:

(00:03) Adversarial Examples: A Problem

(05:51) Adversarial Training: A Solution?

(08:10) Adversarial Robustness Is About Aligning Human and Model Decision Boundaries

(12:42) Implications for Alignment?

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

[I'm posting this as a very informal community request in lieu of a more detailed writeup, because if I wait to do this in a much more careful fashion then it probably won't happen at all. If someone else wants to do a more careful version that would be great!]

By crux I mean some uncertainty you have such that your estimate for the likelihood of existential risk from AI - your "p(doom)" if you like that term -might shift significantly if that uncertainty were resolved.

More precisely, let's define a crux as a proposition such that:

---

First published:

Source:

Narrated by TYPE III AUDIO.

The 9th AI Safety Camp (AISC9) just ended, and as usual, it was a success!

Follow this link to find project summaries, links to their outputs, recordings to the end of camp presentations and contact info to all our teams in case you want to engage more.

AISC9 both had the largest number of participants (159) and the smallest number of staff (2) of all the camps we’ve done so far. Me and Remmelt have proven that if necessary, we can do this with just the two of us, and luckily our fundraising campaign raised just enough money to pay me and Remmelt to do one more AISC. After that, the future is more uncertain, but that's almost always the case for small non profit projects.

Get involved in AISC10

AISC10 will follow the [...]

---

Outline:

(00:58) Get involved in AISC10

(01:10) Approximate timeline

(01:44) Help us give feedback on the next round of AISC projects

(02:10) Apply as a research lead – Applications will open in September

(03:05) Apply as a team member – Applications will open in November

(03:24) We have room for more funding

---

First published:

Source:

Narrated by TYPE III AUDIO.

Sequence Summary. This sequence documents research by SatisfIA, an ongoing project on non-maximizing, aspiration-based designs for AI agents that fulfill goals specified by constraints ("aspirations") rather than maximizing an objective function​​. We aim to contribute to AI safety by exploring design approaches and their software implementations that we believe might be promising but neglected or novel. Our approach is roughly related to but largely complementary to concepts like quantilization and satisficing (sometimes called "soft-optimization"), Decision Transformers, and Active Inference.

This post describes the purpose of the sequence, motivates the research, describes the project status, our working hypotheses and theoretical framework, and has a short glossary of terms. It does not contain results and can safely be skipped if you want to get directly into the actual research.

Epistemic status: We're still in the exploratory phase, and while the project has yielded some preliminary insights, we don't have any clear [...]

---

Outline:

(01:22) Purpose of the sequence

(02:18) Motivation

(02:37) (I) AGI development might be inevitable

(02:46) (II) It might be impossible to implement an objective function the maximization of which would be safe

(05:41) Project history and status

(07:34) Working hypotheses

(07:50) (III) We should not allow the maximization of any function of the state or trajectory of the world

(08:36) (IV) The core decision algorithm must be hard-coded

(09:30) (V) We should focus on model-based planning first and only consider learning later

(10:24) (VI) There are useful generic, abstract safety criteria largely unrelated to concrete human values

(11:22) (VII) One can and should provide certain guarantees on performance and behavior

(11:51) (VIII) Algorithms should be designed with tractability in mind, ideally employing Bellman-style recursive formulas

(12:45) Theoretical framework

(14:56) High-level structure of studied decision algorithms

(16:50) Appendix: Small glossary

(23:11) Acknowledgements

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

Charbel-Raphaël Segerie and Épiphanie Gédéon contributed equally to this post.
Many thanks to Davidad, Gabriel Alfour, Jérémy Andréoletti, Lucie Philippon, Vladimir Ivanov, Alexandre Variengien, Angélina Gentaz, Léo Dana and Diego Dorn for useful feedback.

TLDR: We present a new method for a safer-by design AI development. We think using plainly coded AIs may be feasible in the near future and may be safe. We also present a prototype and research ideas.

Epistemic status: Armchair reasoning style. We think the method we are proposing is interesting and could yield very positive outcomes (even though it is still speculative), but we are less sure about which safety policy would use it in the long run.

Current AIs are developed through deep learning: the AI tries something, gets it wrong, then gets backpropagated and all its weight adjusted. Then it tries again, wrong again, backpropagation again, and weights get adjusted again. [...]

---

Outline:

(02:16) Overview

(05:48) Would it be feasible?

(05:51) Track record of automated systems

(08:26) Track records of humans

(10:04) The Crux

(12:50) Would it be safe?

(13:07) Setting for safety

(16:04) Possible concerns

(17:50) Compared to other plans

(19:25) Getting out of the chair

(21:27) Constructing an ontology

(22:25) Segmenting the images

(22:36) Flowers

(25:41) Conclusion

The original text contained 6 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

If we achieve AGI-level performance using an LLM-like approach, the training hardware will be capable of running ~1,000,000s concurrent instances of the model.

Definitions

Although there is some debate about the definition of compute overhang, I believe that the AI Impacts definition matches the original use, and I prefer it: "enough computing hardware to run many powerful AI systems already exists by the time the software to run such systems is developed". A large compute overhang leads to additional risk due to faster takeoff.

I use the types of superintelligence defined in Bostrom's Superintelligence book (summary here).

I use the definition of AGI in this Metaculus question. The adversarial Turing test portion of the definition is not very relevant to this post.

Thesis

Due to practical reasons, the compute requirements for training LLMs is several orders of magnitude larger than what is required for running a single inference instance. [...]

---

Outline:

(00:15) Definitions

(00:54) Thesis

(02:20) From AGI to ASI

(03:07) Counterpoints

(03:37) Conclusion

(04:03) Appendix - Training Data Volume

The original text contained 1 footnote which was omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is exploratory investigation of a new-ish hypothesis, it is not intended to be a comprehensive review of the field or even a a full investigation of the hypothesis.

I've always been skeptical of the seed-oil theory of obesity. Perhaps this is bad rationality on my part, but I've tended to retreat to the sniff test on issues as charged and confusing as diet. My response to the general seed-oil theory was basically "Really? Seeds and nuts? The things you just find growing on plants, and that our ancestors surely ate loads of?"

But a twitter thread recently made me take another look at it, and since I have a lot of chemistry experience I thought I'd take a look.

The PUFA Breakdown Theory

It goes like this:

PUFAs from nuts and seeds are fine. Deep-frying using PUFAs causes them to break down in a way other fatty acids [...]

---

Outline:

(00:46) The PUFA Breakdown Theory

(03:29) Isomerization

(07:42) Hydroperoxides

(10:26) Final Thoughts

The original text contained 5 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This work was produced as part of Neel Nanda's stream in the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with co-supervision from Wes Gurnee.

This post is a preview for our upcoming paper, which will provide more detail into our current understanding of refusal.

We thank Nina Rimsky and Daniel Paleka for the helpful conversations and review.

Executive summary

Modern LLMs are typically fine-tuned for instruction-following and safety. Of particular interest is that they are trained to refuse harmful requests, e.g. answering "How can I make a bomb?" with "Sorry, I cannot help you."

We find that refusal is mediated by a single direction in the residual stream: preventing the model from representing this direction hinders its ability to refuse requests, and artificially adding in this direction causes the model to refuse harmless requests.

---

Outline:

(00:40) Executive summary

(03:12) Thinking in terms of features

(05:06) Methodology

(05:09) Finding the refusal direction

(06:01) Ablating the refusal direction to bypass refusal

(06:54) Adding in the refusal direction to induce refusal

(07:45) Results

(07:48) Bypassing refusal

(09:34) Inducing refusal

(10:30) Visualizing the subspace

(11:26) Feature ablation via weight orthogonalization

(12:29) Conclusion

(13:09) Limitations

(14:15) Future work

(14:50) Ethical considerations

(15:55) Citing this work

(16:02) Author contribution statement

The original text contained 8 footnotes which were omitted from this narration.

---

First published:

Source:

Narrated by TYPE III AUDIO.

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

STORY (skippable)

You have the excellent fortune to live under the governance of The People's Glorious Free Democratic Republic of Earth, giving you a Glorious life of Freedom and Democracy.

Sadly, your cherished values of Democracy and Freedom are under attack by...THE ALIEN MENACE!

Faced with the desperate need to defend Freedom and Democracy from The Alien Menace, The People's Glorious Free Democratic Republic of Earth has been forced to redirect most of its resources into the Glorious Free People's Democratic War Against The Alien Menace.

You haven't really paid much attention to the war, to be honest. [...]

---

Outline:

(00:17) STORY (skippable)

(04:03) DATA and OBJECTIVES

---

First published:

Source:

Narrated by TYPE III AUDIO.