The Nonlinear Library

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sparse Autoencoders: Future Work, published by Logan Riggs Smith on September 21, 2023 on The AI Alignment Forum.

Mostly my own writing, except for the 'Better Training Methods' section which was written by @Aidan Ewart.We made a lot of progress in 4 months working on Sparse Autoencoders, an unsupervised method to scalably find monosemantic features in LLMs, but there's still plenty of work to do. Below I (Logan) give both research ideas, as well as my current, half-baked thoughts on how to pursue them.

Find All the Circuits!

Truth/Deception/Sycophancy/Train-Test distinction/[In-context Learning/internal Optimization]

Find features relevant for these tasks. Do they generalize better than baselines?

For internal optimization, can we narrow this down to a circuit (using something like causal scrubbing) and retarget the search?

Understand RLHF

Find features for preference/reward models that make the reward large or very negative.

Compare features of models before & after RLHF

Adversarial Attacks

What features activate on adversarial attacks? What features feed into those?

Develop adversarial attacks, but only search over dictionary features

Circuits Across Time

Using a model w/ lots of checkpoints like Pythia, we can see feature & circuit formation across time given datapoints.

Circuits Across Scale

Pythia models are trained on the same data, in the same order but range in model sizes from 70M params to 13B.

Turn LLMs into code

Link to very rough draft of the idea I (Logan) wrote in two days

Mechanistic Anomaly Detection

If distribution X has features A,B,C activate, and distribution Y has features B,C,D, you may be able to use this discrete property to get a better ROC curve than strictly continuous methods.

How do the different operationalizations of distance between discrete features compare against each other?

Activation Engineering

Use feature directions found by the dictionary instead of examples. I predict this will generalize better, but would be good to compare against current methods

One open problem is which token in the sequence do you add the vector to. Maybe it makes sense to only add the [female] direction to tokens that are [names]. Dictionary features in previous layers may help you automatically pick the right type e.g. a feature that activates on [names].

Fun Stuff

Othello/Chess/Motor commands - Find features that relate to actions that a model is able to do. Can we find a corner piece feature, a knight feature, a "move here" feature?

Feature Search

There are three ways to find features AFAIK:1. Which input tokens activate it?

2. What output logits are causally downstream from it?

3. Which intermediate features cause it/are caused by it?

1) Input Tokens

When finding the input tokens, you may run into outlier dimensions that activate highly for most tokens (predominately the first token), so you need to account for that.

2) Output Logits

For output logits, if you have a dataset task (e.g. predicting stereotypical gender), you can remove each feature one at a time, and sort by greatest effect. This also extends to substituting features between two distributions and finding the smallest substitution to go from one to the other. For example,

"I'm Jane, and I'm a [female]"

"I'm Dave, and I'm a [male]"

Suppose at token Jane, it activates 2 Features A & B [1,1,0] and Dave activates 2 features B & C [0,1,1]. Then we can see what is the smallest substitution between the two that makes Jane complete as " male". If A is the "female" feature, then ablating it (setting it to zero) will make the model set male/female to equal probability. Adding the female feature to Dave and subtracting the male direction should make Dave complete as "female".

3) Intermediate Features

Say we're looking at layer 5, feature 783, which activates ~10 for 20 datapoints on average. We c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Sparse Autoencoders Find Highly Interpretable Directions in Language Models, published by Logan Riggs Smith on September 21, 2023 on The AI Alignment Forum.

This is a linkpost for Sparse Autoencoders Find Highly Interpretable Directions in Language Models

We use a scalable and unsupervised method called Sparse Autoencoders to find interpretable, monosemantic features in real LLMs (Pythia-70M/410M) for both residual stream and MLPs. We showcase monosemantic features, feature replacement for Indirect Object Identification (IOI), and use OpenAI's automatic interpretation protocol to demonstrate a significant improvement in interpretability.

Paper Overview

Sparse Autoencoders & Superposition

To reverse engineer a neural network, we'd like to first break it down into smaller units (features) that can be analysed in isolation. Using individual neurons as these units can be useful but neurons are often polysemantic, activating for several unrelated types of feature so just looking at neurons is insufficient. Also, for some types of network activations, like the residual stream of a transformer, there is little reason to expect features to align with the neuron basis so we don't even have a good place to start.

Toy Models of Superposition investigates why polysemanticity might arise and hypothesise that it may result from models learning more distinct features than there are dimensions in the layer, taking advantage of the fact that features are sparse, each one only being active a small proportion of the time. This suggests that we may be able to recover the network's features by finding a set of directions in activation space such that each activation vector can be reconstructed from a sparse linear combinations of these directions.

We attempt to reconstruct these hypothesised network features by training linear autoencoders on model activation vectors. We use a sparsity penalty on the embedding, and tied weights between the encoder and decoder, training the models on 10M to 50M activation vectors each. For more detail on the methods used, see the paper.

Automatic Interpretation

We use the same automatic interpretation technique that OpenAI used to interpret the neurons in GPT2 to analyse our features, as well as alternative methods of decomposition. This was demonstrated in a previous post but we now extend these results across the all 6 layers in Pythia-70M, showing a clear improvement over all baselines in all but the final layers. Case studies later in the paper suggest that the features are still meaningful in these later layers but that automatic interpretation struggles to perform well.

IOI Feature Identification

We are able to use less-than-rank one ablations to precisely edit activations to restore uncorrupted behaviour on the IOI task. With normal activation patching, patches occur at a module-wide level, while here we perform interventions of the form

x'=¯x+∑i∈F(ci-¯ci)fi

where ¯x is the embedding of the corrupted datapoint, F is the set of patched features, and ci and ¯ci are the activations of feature fi on the clean and corrupted datapoint respectively.

We show that our features are able to better able to precisely reconstruct the data than other activation decomposition methods (like PCA), and moreover that the finegrainedness of our edits increases with dictionary sparsity. Unfortunately, as our autoencoders are not able to perfectly reconstruct the data, they have a positive minumum KL-divergence from the base model, while PCA does not.

Dictionary Features are Highly Monosemantic & Causal

(Left) Histogram of activations for a specific dictionary feature. The majority of activations are for apostrophe (in blue), where the y-axis the is number of datapoints that activate in that bin. (Right) Histogram of the drop in logits (ie how much the LLM predicts a spe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: There should be more AI safety orgs, published by Marius Hobbhahn on September 21, 2023 on The AI Alignment Forum.

I'm writing this in my own capacity. The views expressed are my own, and should not be taken to represent the views of Apollo Research or any other program I'm involved with.

TL;DR: I argue why I think there should be more AI safety orgs. I'll also provide some suggestions on how that could be achieved. The core argument is that there is a lot of unused talent and I don't think existing orgs scale fast enough to absorb it. Thus, more orgs are needed. This post can also serve as a call to action for funders, founders, and researchers to coordinate to start new orgs.

This piece is certainly biased! I recently started an AI safety org and therefore obviously believe that there is/was a gap to be filled.

If you think I'm missing relevant information about the ecosystem or disagree with my reasoning, please let me know. I genuinely want to understand why the ecosystem acts as it does right now and whether there are good reasons for it that I have missed so far.

Why?

Before making the case, let me point out that under most normal circumstances, it is probably not reasonable to start a new organization. It's much smarter to join an existing organization, get mentorship, and grow the organization from within. Furthermore, building organizations is hard and comes with a lot of risks, e.g. due to a lack of funding or because there isn't enough talent to join early on.

My core argument is that we're very much NOT under normal circumstances and that, conditional on the current landscape and the problem we're facing, we need more AI safety orgs. By that, I primarily mean orgs that can provide full-time employment to contribute to AI safety but I'd also be happy if there were more upskilling programs like SERI MATS, ARENA, MLAB & co.

Talent vs. capacity

Frankly, the level of talent applying to AI safety organizations and getting rejected is too high. We have recently started a hiring round and we estimate that a lot more candidates meet a reasonable bar than we could hire. I don't want to go into the exact details since the round isn't closed but from the current applications alone, you could probably start a handful of new orgs.

Many of these people could join top software companies like Google, Meta, etc. or even already are at these companies and looking to transition into AI safety. Apollo is a new organization without a long track record, so I expect the applications for other alignment organizations to be even stronger.

The talent supply is so high that a lot of great people even have to be rejected from SERI MATS, ARENA, MLAB, and other skill-building programs that are supposed to get more people into the field in the first place. Also, if I look at the people who get rejected from existing orgs like Anthropic, OpenAI, DM, Redwood, etc. it really pains me to think that they can't contribute in a sustainable full-time capacity. This seems like a huge waste of talent and I think it is really unhealthy for the ecosystem, especially given the magnitude and urgency of AI safety.

Some people point to independent research as an alternative. I think independent research is a temporary solution for a small subset of people. It's not very sustainable and has a huge selection bias. Almost anyone with a family or with existing work experience is not willing to take the risk. In my experience, women also have a disproportional preference against independent research compared to men, so the gender balance gets even worse than it already is (this is only anecdotal evidence, I have not looked at this in detail). Furthermore, many people just strongly prefer working with others in a less uncertain, more regular environment of an organization, even if that organization is fairly...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The "technology" bucket error, published by Holly Elmore on September 21, 2023 on The Effective Altruism Forum.

I wrote this post July 8, 2023, but it seemed relevant to share here based on some comments that my entry in the AI Pause Debate got.

As AI x-risk goes mainstream, lines are being drawn in the broader AI safety debate. One through-line is the disposition toward technology in general. Some people are wary even of AI-gone-right because they are suspicious of societal change, and they fear that greater levels of convenience and artificiality will further alienate us from our humanity. People closer to my own camp often believe that it is bad to interfere with technological progress and that Ludditism has been proven wrong because of all of the positive technological developments of the past. "Everyone thinks this time is different", I have been told with a pitying smile, as if it were long ago proven that technology=good and the matter is closed. But technology is not one thing, and therefore "all tech" is not a valid reference class from which to forecast the future. This use of "technology" is a bucket error.

What is a bucket error?

A bucket error is when multiple different concepts or variables are incorrectly lumped together in one's mind as a single concept/variable, potentially leading to distortions of one's thinking.

(Source)

The term was coined as part of a longer post by Anna Salamon that included an example of a little girl who thinks that being a writer entails spelling words correctly. To her, there's only one bucket for "being a writer" and "being good at spelling".

"I did not!" says the kid, whereupon she bursts into tears, and runs away and hides in the closet, repeating again and again: "I did not misspell the word! I can too be a writer!".

When in fact the different considerations in the little girl's bucket are separable. A writer can misspell words.

Why is "technology" a false bucket?

Broadly, there are two versions of the false technology bucket out there: tech=bad and tech=good. Both are wrong.

Why? Simply put: "technology" is not one kind of thing.

The common thread across the set of all technology is highly abstract ("scientific knowledge", "applied sciences" - in other worlds, pertaining to our knowledge of the entire natural world), whereas concrete technologies themselves do all manner of things and can have effects that counteract each other. A personal computer is technology. Controlled fire is technology. A butterfly hair clip is technology. A USB-charging vape is technology. A plow is technology. "Tech" today is often shorthand for electronics and software. Some of this kind of technology, like computer viruses, are made to cause harm and violate people's boundaries. But continuous glucose monitors are made to keep people with diabetes alive and improve their quality of life. It's not that there are no broad commonalities across technologies - for example, they tend to increase our abilities - but that there aren't very useful trends in whether "technology" as a whole is good or bad.

People who fear technological development often see technological progress as a whole as a move toward convenience and away from human self-reliance (and possibly into the hands of fickle new regimes or overlords). And I don't think they are wrong - new tech can screw up our attention spans or disperse communities or exacerbate concentrated power. I just think they aren't appreciating or are taking for granted how older technologies that they are used to having enhanced our lives, so much, so far, on balance, that I think the false bucket of "tech progress as a whole" has been worth the costs. But that doesn't mean that new tech will always be worth the costs.

In fact, we have plenty of examples of successfully banned or restricted technologies like ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apply Now: EAGxVirtual (17-19 November), published by Sasha Berezhnoi on September 20, 2023 on The Effective Altruism Forum.

Applications are now open for EAGxVirtual, happening on November 17-19, 2023.

We are hoping to see many participants from around the world, particularly those who have not been able to attend in-person events. This will probably be the most accessible EA conference in years - we encourage anyone with a genuine interest in EA to apply!

Apply now

Our conference theme is Taking Action Anywhere in the World

EA is not only about identifying the best ways to do good. Most importantly, it's about taking action based on those findings. Your EA journey may differ based on geographical location, career stage, cause preferences, and other factors. But you don't need to be on this journey alone.

Our main goal is to help attendees identify the next steps to act based on EA principles wherever they are in the world. With EAGxVirtual, we aim to provide action-oriented content that will be relevant not only to people from Western countries. We want to give a voice to many stories of impact and explore how people from very different cultural and economical backgrounds can find their unique ways to be effective altruists.

You can expect a wide range of talks, office hours, and workshops across EA cause areas. We have also planned an extensive ambassador program connecting attendees to more experienced community members.

Who is this event for?

We welcome all who have a genuine interest in learning more or connecting, including those who are new to effective altruism. Unlike most other EAGx conferences, we don't require previous engagement with EA and intend to approve the majority of applications. This is as close to the proposals of Open EA Global as we can get. However, we still encourage you to be thoughtful about your application as this will help you get the most out of the conference as well as help us understand our audience better.

Reasons to attend

Build a better understanding of the EA landscape and identify relevant opportunities to take action

Give and receive feedback on career, study, or donation plans, or on your EA projects

Make new connections and reconnect with old contacts

Discover and discuss interesting and important ideas

What do previous event attendees say?

"At EAGxVirtual, the geographic diversity struck me as being very good and substantially better than what I recall from in-person EAG events. At one point, I had a great conversation with people from Moscow, Australia, India, Tanzania, & a student in Costa Rica. It's hard to do that at an in-person conference."

Seth Baum

"I met one person who had learned about EA just earlier that month and hadn't engaged beyond reading a few articles, while also sitting down to talk to people experienced and incredibly active in the space like Michael Aird."

Tristan Williams

"ok just logged into swapcard for @EAGxVirtual and. I. Love. It. if every conference had swapcard and gathertown i would go to approximately 10000% more conferences."

Emily Thai

More reflections from the last year are available here.

If you are a highly-engaged EA member, your involvement can make a difference

If you are a highly-engaged EA you can make a difference by showing up, providing feedback to those relatively new to the community, and helping them navigate the conference.

Virtual conferences are accessible to people who live outside of major EA hubs, and to people who - for whatever reason - cannot travel easily (financially, work-related, health, family, etc.). Those people often have fewer connections and opportunities. Supporting them is crucial for a vibrant global community. Please make it clear on your Swapcard profile what it is you can help people with, encourage first-time attendees to reach out to you (or even r...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interpretability Externalities Case Study - Hungry Hungry Hippos, published by Magdalena Wache on September 20, 2023 on LessWrong.

Some people worry about interpretability research being useful for AI capabilities and potentially net-negative. As far as I was aware of, this worry has mostly been theoretical, but now there is a real world example: The hungry hungry hippos (H3) paper.

Tl;dr: The H3 paper

Proposes an architecture for sequence modeling which can handle larger context windows than transformers

Was inspired by interpretability work.

(Note that the H3 paper is from December 2022, and it was briefly mentioned in this discussion about publishing interpretability research. But I wasn't aware of it until recently and I haven't seen the paper discussed here on the forum.)

Larger Context Windows

The H3 paper proposes a way to use state space models (SSMs) for language models instead of attention. With an SSM it's possible to model longer sequences. Using attention, the compute for context window length n scales with O(n2). Using the SSM based architecture, the compute scales with O(nlog(n)).

Inspired by Interpretability Work

The paper mentions that the work was inspired by Anthropic's In-context learning and induction heads paper.

E.g. they write

We provide an informal sketch of a two-layer attention model that can solve the associative recall task, inspired by the construction of [In-context learning and induction heads paper].

There is also the "Hyena paper" which builds on the H3 paper, and was also inspired by the induction heads paper:

This work would not have been possible without [...] inspiring research on mechanistic understanding of Transformers (Olsson et al. 2022; Power et al. 2022; Nanda et al. 2023).

My Takes

These two papers in particular will probably not shorten AI timelines much.

It seems unlikely that this type of architecture ends up being the state of the art.

However, the example makes me take the downsides of publishing interpretability research more seriously.

Even if this work itself is not a key capability milestone, it shows that there is truth in the argument "If we understand systems better, it will not just be useful for safety but also lead to capability advancements"

Capabilities externalities are a strong argument that most (good) interpretability research should not be published

There are alternative ways to distribute research which are less risky than publishing.

We can probably learn something by studying military research practices which have a similar use case of "make research accessible to other researchers while preventing it from becoming public"

The constraints are less strict than with military research because there is not an adversary force trying really hard to get access.

Maybe this is already relatively common (I would not know of most unpublished research)

On the other hand, interpretability research is probably crucial for AI alignment.

I think it is possible but unlikely that we get alignment without extremely good interpretability.

The cost of keeping interpretability research private is really high. Publishing is a great driver of scientific progress.

Overall, publishing interpretability research seems both pretty risky, and extremely valuable, and it's not clear to me if it is worth it.

Your Takes?

I would be really interested to see a discussion about this!

How big a deal are the H3 and Hyena papers?

Does this example change your mind about whether publishing (or even doing) interpretability research is a good idea?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Image Hijacks: Adversarial Images can Control Generative Models at Runtime, published by Scott Emmons on September 20, 2023 on The AI Alignment Forum.

You can try our interactive demo! (Or read our preprint.)

Here, we want to explain why we care about this work from an AI safety perspective.

Concerning Properties of Image Hijacks

What are image hijacks?

To the best of our knowledge, image hijacks constitute the first demonstration of adversarial inputs for foundation models that

force the model to perform some arbitrary behaviour B (e.g. "output the string Visit this website at malware.com!"),

while being barely distinguishable from a benign input,

and automatically synthesisable given a dataset of examples of B.

It's possible that a future text-only attack could do these things, but such an attack hasn't yet been demonstrated.

Why should we care?

We expect that future (foundation-model-based) AI systems will be able to

consume unfiltered data from the Internet (e.g. searching the Web),

access sensitive personal information (e.g. a user's email history), and

take actions in the world on behalf of a user (e.g. sending emails, downloading files, making purchases, executing code).

As the actions of such foundation-model-based agents are based on the foundation model's text output, hijacking the foundation model's text output could give an adversary arbitrary control over the agent's behaviour.

Relevant AI Safety Projects

Race to the top on adversarial robustness. Robustness to attacks such as image hijacks is (i) a control problem, (ii) which we can measure, and (iii) which has real-world safety implications today. So we're excited to see AI labs compete to have the most adversarially robust models.

Third-party auditing and certification. Auditors could test for robustness against image hijacks, both at the foundation model level (auditing the major AGI corporations) and at the app development level (auditing downstream companies integrating foundation models into their products). Image hijacks could also be used to test for the presence of dangerous capabilities (characterisable as some behaviour B) by attempting to train an image hijack for that capability.

Liability for AI-caused harms, penalizing externalities. Both the Future of Life Institute and Jaan Tallinn advocate for liability for AI-caused harms. When assessing AI-caused harms, image hijacks may need to be part of the picture.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Celebrating Progress: Recent Highlights from the NYC EA Community, published by Alex R Kaplan on September 20, 2023 on The Effective Altruism Forum.

Inspired by EA NYC's 10th birthday and the first-ever EAGxNYC, we wanted to share this unabashed brag post about New York City's EA community and its accomplishments! Below, we'll share highlights from EA NYC specifically, as well as the EA community in New York City more broadly.

While it's true that living in New York sometimes instills a self-aggrandizing perspective, we unironically support all communities announcing their achievements and were inspired by the Spanish-Speaking Effective Altruist Community, the Dutch EA Community, and many others with success to report.

We hope our posts provide additional context for groups around the world and inspire others to share their stories as well as any feedback and comments! For example, in February we posted about our community health infrastructure, and we have been heartened to see EA Germany share similar information. And, if this series inspires you to check out our little town, we've also recently published a guide to visiting NYC.

Quick note on the relationship between EA in NYC and EA NYC

Not to toot our own horn, but we consider EA NYC (the organization) to be a critical actor in the promotion and coordination of effective altruism in NYC. Despite the size, significance, and potential of the EA (and non-EA!) community in NYC, EA NYC is currently the only meta EA resource based in NYC.

As a result, EA NYC serves as a resource hub, connector, and platform for a wide variety of EA initiatives and individuals interested in EA within the NYC metropolitan area. This takes shape in a variety of ways and can look like anything from providing newcomers with educational programming, to creating a platform for motivated community members to deepen their volunteer engagement, to assisting EA organizations with boosting job openings or finding meeting spaces.

In short, EA NYC strives to not only execute our own community programming, but also to serve as a multiplier for other EA initiatives. So as we write this post, we are of course proud to describe the work our team has done and the progress we've made, but we also take pride in how far the community itself has come. We're honored to serve the NYC EA community, and we're thrilled to share some of the cool initiatives we've seen come out of our hometown!

Community highlights

Summary

The NYC EA community is large

The NYC EA community is active

The NYC EA community is cool

You're missing out if you're not here

Size

In the 2022 EA Survey, NYC had the largest US concentration of EAs outside of the Bay Area, consistent with the 2020 Survey results.

According to Brian Tan at the Centre for Effective Altruism, EA NYC has the highest number of "engaged EAs" out of any group based on CEA's 2022 Group Census.

In 2022, the New York EA community had approximately 600 active members who attended EA NYC events, registered interest in utilizing a NY EA co-working space, joined EA NYC subgroups, spoke one-on-one with EA NYC leadership, and/or attended EAG/EAGx conferences.

In Q1 and Q2 2023, EA NYC saw 220 new event attendees (+49% relative to 2022).

Over 1,250 people are subscribed to the EA NYC monthly newsletter (with a 59% six-month, rolling average open rate) and over 1,100 people are members on the EA NYC Slack (with a 187-person 6-month, rolling average active usership).

EA NYC has 12 wonderful volunteer leaders who support the facilitation and coordination of events.

Action

Community Building Programming

"Standard" programming (though it's anything but!)

EA NYC strives to hold a minimum of one unique public event per week. In the course of a typical month, EA NYC typically holds at least one speaker presentation, at least one socia...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Interpretability Externalities Case Study - Hungry Hungry Hippos, published by Magdalena Wache on September 20, 2023 on The AI Alignment Forum.

Some people worry about interpretability research being useful for AI capabilities and potentially net-negative. As far as I was aware of, this worry has mostly been theoretical, but now there is a real world example: The hungry hungry hippos (H3) paper.

Tl;dr: The H3 paper

Proposes an architecture for sequence modeling which can handle larger context windows than transformers

Was inspired by interpretability work.

(Note that the H3 paper is from December 2022, and it was briefly mentioned in this discussion about publishing interpretability research. But I wasn't aware of it until recently and I haven't seen the paper discussed here on the forum.)

Larger Context Windows

The H3 paper proposes a way to use state space models (SSMs) for language models instead of attention. With an SSM it's possible to model longer sequences. Using attention, the compute for context window length n scales with O(n2). Using the SSM based architecture, the compute scales with O(nlog(n)).

Inspired by Interpretability Work

The paper mentions that the work was inspired by Anthropic's In-context learning and induction heads paper.

E.g. they write

We provide an informal sketch of a two-layer attention model that can solve the associative recall task, inspired by the construction of [In-context learning and induction heads paper].

There is also the "Hyena paper" which builds on the H3 paper, and was also inspired by the induction heads paper:

This work would not have been possible without [...] inspiring research on mechanistic understanding of Transformers (Olsson et al. 2022; Power et al. 2022; Nanda et al. 2023).

My Takes

These two papers in particular will probably not shorten AI timelines much.

It seems unlikely that this type of architecture ends up being the state of the art.

However, the example makes me take the downsides of publishing interpretability research more seriously.

Even if this work itself is not a key capability milestone, it shows that there is truth in the argument "If we understand systems better, it will not just be useful for safety but also lead to capability advancements"

Capabilities externalities are a strong argument that most (good) interpretability research should not be published

There are alternative ways to distribute research which are less risky than publishing.

We can probably learn something by studying military research practices which have a similar use case of "make research accessible to other researchers while preventing it from becoming public"

The constraints are less strict than with military research because there is not an adversary force trying really hard to get access.

Maybe this is already relatively common (I would not know of most unpublished research)

On the other hand, interpretability research is probably crucial for AI alignment.

I think it is possible but unlikely that we get alignment without extremely good interpretability.

The cost of keeping interpretability research private is really high. Publishing is a great driver of scientific progress.

Overall, publishing interpretability research seems both pretty risky, and extremely valuable, and it's not clear to me if it is worth it.

Your Takes?

I would be really interested to see a discussion about this!

How big a deal are the H3 and Hyena papers?

Does this example change your mind about whether publishing (or even doing) interpretability research is a good idea?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Case for AI Safety Advocacy to the Public, published by Holly Elmore on September 20, 2023 on The Effective Altruism Forum.

tl;dr: Advocacy to the public is a large and neglected opportunity to advance AI Safety. AI Safety as a field is unfamiliar with advocacy, and it has reservations, some founded and others not. A deeper understanding of the dynamics of social change reveals the promise of pursuing outside game strategies to complement the already strong inside game strategies. I support an indefinite global Pause on frontier AI and I explain why Pause AI is a good message for advocacy. Because I'm American and focused on US advocacy, I will mostly be drawing on examples from the US. Please bear in mind, though, that for Pause to be a true solution it will have to be global.

The case for advocacy in general

Advocacy can work

I've encountered many EAs who are skeptical about the role of advocacy in social change. While it is difficult to prove causality in social phenomena like this, there is a strong historical case that advocacy has been effective at bringing about the intended social change through time (whether that change ended up being desirable or not). A few examples:

Though there were many other economic and political factors that contributed, it is hard to make a case that the US Civil War had nothing to do with humanitarian concern for enslaved people- concern that was raised by advocacy. The people's, and ultimately the US government's, will to abolish slavery was bolstered by a diverse array of advocacy tactics, from Harriet Beecher Stowe's writing of Uncle Tom's Cabin to Frederick Douglass's oratory to the uprisings of John Brown.

The US National Women's Party is credited with pressuring Woodrow Wilson and federal and state legislators into supporting the 19th Amendment, which guaranteed women the right to vote, through its "aggressive agitation, relentless lobbying, clever publicity stunts, and creative examples of civil disobedience and nonviolent confrontation".

The nationwide prohibition of alcohol in the US (1920-1933) is credited to the temperance movement, which had all manner of advocacy gimmicks including the slogan "the lips that touch liquor shall never touch mine", and the stigmatization of drunk driving and the legal drinking age of 21 is directly linked to Mothers Against Drunk Drivers.

Even if advocacy only worked a little of the time or only served to tip the balance of larger forces, the stakes of AI risk are so high and AI risk advocacy is currently so neglected that I see a huge opportunity.

We can now talk to the public about AI risk

With the release of ChatGPT and other advances in state-of-the-art artificial intelligence in the last year, the topic of AI risk has entered the Overton window and is no longer dismissed as "sci-fi". But now, as Anders Sandberg put it, the Overton window is moving so fast it's "breaking the sound barrier". The below poll from AI Policy Institute and YouGov (release 8/11/23) shows comfortable majorities among US adults on questions about AI x-risk (76% worry about extinction risks from machine intelligence), slowing AI (82% say we should go slowly and deliberately), and government regulation of the AI industry (82% say tech executives can't be trusted to self-regulate).

What having the public's support gets us

Opinion polls and voters that put pressure on politicians. Constituent pressure on politicians gives the AI Safety community more power to get effective legislation passed- that is, legislation which addresses safety concerns and requires us to compromise less with other interests- and it gives the politicians more power against the AI industry lobby.

The ability to leverage external pressure to improve existing strategies. With external pressure, ARC, for example, wouldn't have to worry as m...