Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Internal Interfaces Are a High-Priority Interpretability Target, published by Thane Ruthenis on December 29, 2022 on The AI Alignment Forum.

tl;dr: ML models, like all software, and like the NAH would predict, must consist of several specialized "modules". Such modules would form interfaces between each other, and exchange consistently-formatted messages through these interfaces. Understanding the internal data formats of a given ML model should let us comprehend an outsized amount of its cognition, and allow to flexibly interfere in it as well.

1. A Cryptic Analogy

Let's consider three scenarios. In each of them, you're given the source code of a set of unknown programs, and you're tasked with figuring out their exact functionality. Details vary:

In the first scenario, the programs are written in some known programming language, e. g. Python.

In the second scenario, the programs were randomly generated by perturbing machine code until it happened to end up in a configuration that, when ran, instantiates a process externally indistinguishable from a useful intelligently-written program.

In the third scenario, the programs are written in a programming language that's completely unfamiliar to you (or to anyone else).

In the first scenario, the task is all but trivial. You read the source code, make notes on it, run parts of it, and comprehend it. It may not be quick, but it's straightforward.

The second scenario is a nightmare. There must be some structure to every program's implementation — something like the natural abstraction hypothesis must still apply, there must be modules in this mess of a code that can be understood separately, etc. There is some high-level structure that you can parcel out into tiny pieces that can fit into a human mind. The task is not impossible.

But suppose you've painstakingly reverse-engineered one of the programs this way. You move on to the next one, and... Yup, you're essentially starting from zero. Well, you've probably figured out something about natural abstractions when working on your first reverse-engineering, so it's somewhat easier, but still a nightmare. Every program is structured in a completely different way, you have to start from the fundamentals every time.

The third scenario is a much milder nightmare. You don't focus on reverse-engineering the programs here — first, you reverse-engineer the programming language. It's a task that may be as complex as reverse-engineering one of the individual programs from (2), but once you've solved it, you're essentially facing the same problem as in (1) — a problem that's comparably trivial.

The difference between (2) and (3) is:

In (3), every program has a consistent high-level structure, and once you've figured it out, it's a breeze.

In (2), every high-level structure is unique, and you absolutely require some general-purpose tools for inferring high-level structures.

(2) is a parallel to the general problem of interpretability, different programs being different ML models. Is there some interpretability problem that's isomorphic to (3), however?

I argue there is.

2. Interface Theory

Suppose that we have two separate entities with different specializations: they both can do some useful "work", but there are some types of work that only one of them can perform. Suppose that they want to collaborate: combine their specializations to do tasks neither entity can carry out alone. How can they do so?

For concreteness, imagine that the two entities are a Customer, which can provide any resource from the set of resources R, and an Artist, which can make any sculpture from some set S given some resource budget. They want to "trade": there's some sculpture sx the Customer wants made, and there are some resources rx the Artist needs to make it. How can they carry out such exchanges?

The Custo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the EA Merch Store!, published by Ines on December 29, 2022 on The Effective Altruism Forum.

You can now buy EA merch at eamerch.co! We are launching with a small number of designs, and will be adding more soon, some of which will be based on submissions from the community.

Why a merch store?

Merch helps EAs express themselves and fosters a sense of community. It can also help EAs recognize each other within the same setting or act as a conversation-starter at a function. But most importantly, it’s fun!

How it works

Our designer Pearl will regularly add new products, many based on ideas submitted by the community. Shipping and manufacturing is handled by a third-party (Printful). All products are priced at 0% markup, and we are operating on a one-time independent grant for the time being. We will seek long-term funding in the future contingent on demand.

If you have an idea for a new product or design, submit it at this link! It doesn’t need to be fully fleshed out—just describe what you have in mind, and our designer will try to make it happen. You can also upload a (preferably high-quality) image if you are picturing something really specific. In the short term, we are more likely to take requests for new designs than new products since we are somewhat restricted in the products that we have the ability to manufacture. You are also able to vote on other people’s ideas, and we will generally prioritize those that get the most votes.

We are unlikely to accept designs with controversial, vulgar, or otherwise inappropriate messaging. We are aware of the fact that our merch will symbolize the EA community, and we want to communicate that we are a friendly and welcoming group.

We are still in Beta mode

We have not yet processed any orders, so it is possible that there will be some bugs or unexpected hiccups during our first few weeks of operation (e.g. delays in shipping or payment processing). Place your orders now only if you are okay with this! If not, we recommend you wait and submit your ideas to us in the meantime. We also ask that you do not place orders in bulk at this time, as we are charged immediately for manufacturing and shipping but it takes about a week for your money to reach us.

Due to our web platform’s limited flexibility in setting shipping prices, they may sometimes be slightly higher than shipping cost.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Good things that happened in EA this year, published by Shakeel Hashim on December 29, 2022 on The Effective Altruism Forum.

Crossposted from Twitter

As the year comes to an end, we want to highlight some of the incredible work done and supported by people in the effective altruism community — work that's helping people and animals all over the world.

1/ The team at Charity Entrepreneurship incubated five new charities this year, including the Center for Effective Aid Policy and Vida Plena — the first CE-incubated organisation to operate in Latin America.

2/ Over 1,400 new people signed the Giving What We Can Pledge, committing to giving away 10% or more of their annual income to effective charities. The total number of pledgers is now over 8,000!

3/ The work of The Humane League and other animal welfare activists led 161 new organisations to commit to using cage-free products, helping free millions of chickens from cruel battery cages.

4/ Open Philanthropy launched two new focus areas: South Asian Air Quality and Global Aid Policy. It's already made grants that aim to tackle pollution and increase the quality or quantity of foreign aid./ and/

5/ Alvea, a new biotechnology company dedicated to fighting pandemics, launched and announced that it had already started animal studies for a shelf-stable COVID vaccine.

6/ Almost 80,000 connections were made at events hosted by @CentreforEA's Events team, prompting people to change jobs, start new projects and explore new ideas. EAGx conferences were held around the world — including in Berlin, Australia and Singapore.#Events

7/ The EU Commission said it will "put forward a proposal to end the ‘disturbing’ systematic practice of killing male chicks across the EU" — another huge win for animal welfare campaigners.

8/ What We Owe The Future, a book by @willmacaskill arguing that we can — and should — help build a better world for future generations, became a bestseller in both the US and UK.

9/ New evidence prompted @GiveWell to re-evaluate its views on water quality interventions. It then made a grant of up to $64.7 million for @EvidenceAction's Dispensers for Safe Water water chlorination program, which operates in Kenya, Malawi and Uganda./

10/ Lots of members of the effective altruism community were featured on @voxdotcom's inaugural Future Perfect 50 list of the people building a better future.

11/ Fish welfare was discussed in the UK Parliament for the first time ever, featuring contributions from effective-altruism-backed charities./

12/ Researchers at @iGEM published a paper looking at how we might be able to better detect whether states are complying with the Biological Weapons Convention — work which could help improve biosecurity around the world.

13/ New research from the Lead Exposure Elimination Project showed the dangerous levels of lead in paint in Zimbabwe and Sierra Leone. In response, governments in both countries are working with LEEP to try to tackle the problem and reduce lead exposure./ and/

14/ The EA Forum criticism contest sparked a bunch of interesting and technical debate. One entry prompted GiveWell to re-assess their estimates of the cost-effectiveness of deworming, and inspired a second contest of its own!#Prize_for_inspiring_the_Change_Our_Mind_Contest____20_000

15/ The welfare of crabs, lobsters and prawns was recognised in UK legislation thanks to the new Animal Welfare (Sentience) Bill

16/ Rethink Priorities, meanwhile, embarked on their ambitious Moral Weight Project to provide a better way to compare the interests of different species.

17/ At the @medialab, the Nucleic Acid Observatory project launched — working to develop systems that will help provide an early-warning system for new biological threats.

18/ Longview Philanthropy and @givingwhatwecan launched the Longtermism Fund, a new fund...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Overall Karma vs Agreement Karma, published by Simon M on December 28, 2022 on The Effective Altruism Forum.

Someone recently shared with me a comment (which will appear in this post) which had strong agreement and but a strong down vote. I thought comments like that might make interesting comments to find (turns out they'd found the best example - but I enjoyed searching for more examples and I'm sharing this to save anyone else the effort).

There is a risk that just be surfacing these comments it will change their karma, so I've stated their current numbers in this post in case the numbers change in response to this.

Since agreement karma is new this year, there isn't a huge sample size of posts to analyse. Unsurprisingly there is a strong relationship between overall karma and agreement:

I see roughly 4 different types of outlier:

"Hall of Fame" - abnormally high karma and agreement

"Hall of Shame" - abnormally low karma and agreement

"Terrible point, well made"- abnormally high karma compared to agreement

"The worst person you know has a point" - abnormally low karma compared to agreement

Hall of Fame

Owen Cotton-Barratt's explanation for buying Wytham Abbey (490, 187)

Habryka's defence of the EA leadership (359, 166)

Scott Alexander's explanation for why he denied Kathy Forth's accusations (266, 199)

Milan_Griffes asking Will MacAskill to account for his involvement in the FTX saga (265, 182)

ClaireZabel lifting the lid on where the $ came from for Wytham Abbey (288, 91)

The Worst Person You Know has a Point

emre kaplan's reaction to the Kelsey Piper / SBF interview

Sabs's take on Phil Torres

Terrible point, well made

RAB thought the FTX Future fund team resigned too early (34, -104)

MichaelPlant doesn't like the EA leadership discussing reputational issues in private (21, -81)

EliezerYudkowsky makes his case for when fucking matters (18, -81)

Honorable mention: Guy Raveh doesn't see the point in agreement karma... but people disagree

Hall of Shame

I'm a little bit disappointed with these - I wouldn't bother clicking on any of them. I was hoping some of them would be "so bad it's good", but really these are just bad. Don't waste your time. (But to save anyone searching, here they are)

michaelroberts is rude in a boring way (-118, -91)

effectenator comes from twitter to get some money back (-88, -123)

[anonymous] has a meandering essay about why EA was doomed to fail (-60, -63)

Yorad419 calls SBF & Caroline ugly (-78, -36)

michaelroberts (again) is rude (-63, -47)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: GiveDirectly $1 Million Match Campaign, published by adam galas on December 27, 2022 on The Effective Altruism Forum.

1:1 Match, up to $50K per donor. Until the end of the year or until they hit $1 million.

I gave all I could and still afford to pay my taxes and medical bills.

$2K donation, $4K effective donation, eight people lifted out of poverty, and $10,400 economic impact.

A happy new year to all the people of the world. Live long and prosper:)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How many hours is your standard workweek? Why?, published by Rockwell on December 28, 2022 on The Effective Altruism Forum.

EAs talk about having 80,000 hours in one's career, but working hours vary throughout time and geography. There are open questions about the typical "ideal" number of hours to work in a day for maximum productivity. A recent discussion here on the Forum discussed increasing one's working hours, which runs counter to the trend in recent years of 4-day workweeks or 30-hour workweeks. Sometimes disclosing how many hours one works can also elicit feelings of shame or feel like a competition, and we lose valuable insight as a result.I'm wondering:How many hours is your standard workweek?

Why do you work that many hours rather than fewer or greater?How do you stagger your working hours across a day or week?Of your working hours, how many do you feel are actually productive versus, say, time spent scrolling Twitter or getting more coffee?Does your employer have policies in place around how many hours you must work, the maximum number of hours you are permitted to work, and/or time tracking systems?How has this changed for you over time?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: The Case for Analysing Toy Language Models, published by Neel Nanda on December 28, 2022 on The AI Alignment Forum.

This is the second post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.

I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!

Disclaimer: Mechanistic Interpretability is a small and young field, and I was involved with much of the research and resources linked here. Please take this sequence as a bunch of my personal takes, and try to seek out other researcher’s opinions too!

Motivation

In A Mathematical Framework for Transformer Circuits, we got a lot of traction interpreting toy language models - that is, transformers trained in exactly the same way as larger models, but with only 1 or 2 layers. I think there’s a lot of low-hanging fruit left to pluck when studying toy language models! To accompany this piece, I’ve trained and open sourced some toy models. The models are documented here: there are 12 models, 1 to 4 layers, one attention-only, one normal (with MLPs and GELU activations) and one normal with SoLU activations.

So, why care about studying toy language models? The obvious reason is that it’s way easier to get traction. In particular, the inputs and outputs of a model are intrinsically interpretable, and in a toy model there’s just not as much space between the inputs and outputs for weird complexity to build up. But the obvious objection to the above is that, ultimately, we care about understanding real models (and ideally extremely large ones like GPT-3), and learning to interpret toy models is not the actual goal. This is a pretty valid objection, but to me, there are two natural ways that studying toy models can be valuable:

The first is by finding fundamental circuits that recur in larger models, and motifs that allow us to easily identify these circuits in larger models. A key underlying question here is that of universality: does each model learn its own weird way of completing its task, or are there some fundamental principles and algorithms that all models converge on?

A striking example of universality is induction heads, which we found in A Mathematical Framework in two layer attention-only models. Induction heads are part of a two head induction circuit which models use to detect and continue repeated sequences from earlier in the prompt, and which turn out to be such a fascinating circuit that we wrote a whole other paper on them! They’re universal in all models we’ve looked at, they all appear in a sudden phase change, they seem to be a core mechanism behind complex behaviours such as translation and few shot learning, and they seem to be the main way that transformers use text far back in the context to predict the next token better. (See my overview of induction circuits for more). And knowing about these has helped to disentangle the more complex behaviour of indirect object identification.

The second is by forming a better understanding of how to reverse engineer models - what are the right intuitions and conceptual frameworks, what tooling and techniques do and do not work, and what weird limitations. This one feels less clear to me. Our work in A Mathematical Framework significantly clarified my understanding of transformers in general, especially attention, in a way that seems to generalise - in particular, thinking of the residual stream as the central object, and the significance of the QK-Circuits and OV-Circuits. But there’s also ways it can be misleading, and some techniques that work well in toy models seem to generalise less well.

One angle...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 Concrete Open Problems in Mechanistic Interpretability: Introduction, published by Neel Nanda on December 28, 2022 on The AI Alignment Forum.

This is the first post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. If you want to learn the basics before you think about open problems, check out my post on getting started.

I’ll make another post every 1-2 days, giving a new category of open problems - the second post is out now. If you want to read ahead, check out the draft sequence here!

Disclaimer: Mechanistic Interpretability is a small and young field, and I was involved with much of the research and resources linked here. Please take this sequence as a bunch of my personal takes, and try to seek out other researcher’s opinions too!

Introduction

Mechanistic Interpretability (MI) is the study of reverse engineering neural networks. Taking an inscrutable stack of matrices where we know that it works, and trying to reverse engineer how it works. And often this inscrutable stack of matrices can be decompiled to a human interpretable algorithm! In my (highly biased) opinion, this is one of the most exciting research areas in ML.

There are a lot of reasons to care about mechanistic interpretability research happening. First and foremost, I think that mechanistic interpretability done right can both be highly relevant for alignment. In particular, can we tell whether a model is doing a task well because it’s deceiving us or because it genuinely wants to be helpful? Without being able to look at how a task is being done, these are essentially indistinguishable when facing a sufficiently capable model. But it also has a lot of fascinating scientific questions - how do models actually work? Are there fundamental principles and laws underlying them, or is it all an inscrutable mess?

It is a fact about today’s world that there exist computer programs like GPT-3 that can essentially speak English at a human level, but we have no idea how to write these programs in normal code. It offends me that this is the case, and I see part of the goal of mechanistic interpretability as solving this! And I think that this would be a profound scientific accomplishment.

Purpose

In addition to being very important, mechanistic interpretability is also a very young field, full of low-hanging fruit. There are many fascinating open research questions that might have really impactful results! The point of this sequence is to put my money where my mouth is, and make this concrete. Each post in this sequence is a different category where I think there’s room for significant progress, and a brainstorm of concrete open problems in that area.

Further, you don’t need a ton of experience to start getting traction on interesting problems! I have an accompanying post, with advice on how to build the background skills. The main audience I have in mind for the sequence is people new to the field, who want an idea for where to start. The problems span the spectrum from good, toy, intro problems, to things that could be a serious and impactful research project if well executed, and are accompanied by relevant resources and advice for doing good research. One of the great joys of mechanistic interpretability is that you can get cool results in small models or by interpreting a model that someone else trained. It’s full of rich empirical data and feedback loops, and getting your hands dirty by playing around with a model and trying to make progress is a great way to learn and build intuition!

My hope is that after reading this sequence you’ll have a clear sense of the contours of this field, where value can be added and how to get started pushing on it, and have a better sense of where the low-hanging fruit is.

Disclaimer: As a consequence of this being aimed at people getting...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on my 5-month AI alignment upskilling grant, published by Jay Bailey on December 28, 2022 on The Effective Altruism Forum.

Five months ago, I received a grant from the Long Term Future Fund to upskill in AI alignment. As of a few days ago, I was invited to Berkeley for two months of full-time alignment research under Owain Evans’s stream in the SERIMATS program. This post is about how I got there.

The post is partially a retrospective for myself, and partially a sketch of the path I took so that others can decide if it’s right for them. This post was written relatively quickly - I’m happy to answer more questions via PM or in the comments.

Summary

I was a software engineer for 3-4 years with little to no ML experience before I was accepted for my grant.

I did a bunch of stuff around fundamental ML maths, understanding RL and transformers, and improving my alignment understanding.

Having tutors, getting feedback on my plan early on, and being able to pivot as I went were all very useful for not getting stuck doing stuff that was no longer useful.

I probably wouldn’t have gotten into SERIMATS without that ability to pivot midway through.

After SERIMATS, I want to finish off the last part of the grant while I find work, then start work as a Research Engineer at an alignment organisation.

If in doubt, put in an application!

My Background

My background is more professional and less academic than most. Until I was 23, I didn’t do much of anything - then I got a Bachelor of Computer Science from a university ranked around 1,000th, with little maths and no intent to study ML at all, let alone alignment. It was known for strong graduate employment though, so I went straight into industry from there. I had 3.5 years of software engineering experience (1.5 at Amazon, 2 as a senior engineer at other jobs) before applying for the LTFF grant. I had no ML experience at the time, besides being halfway through doing the fast.ai course in my spare time.

Not going to lie, seeing how many Top-20 university PhD students I was sharing my cohort with (At least three!) was a tad intimidating - but I made it in the end, so industry experience clearly has a role to play as well.

Grant

The details of the grant are one of the main reasons I wrote this - I’ve been asked for 1:1’s and details on this at least three times in the last six months, and if you get asked something from at least three different people, it might be worth writing it up and sharing it around.

Firstly, the process. Applying for the grant is pretty painless. As long as you have a learning plan already in place, the official guidance is to take 1-2 hours on it. I took a bit longer, polishing it more than required. I later found out my plan was more detailed than it probably had to be. In retrospect, I think my level of detail was good, but I spent too much time editing. AI Safety Support helped me with administration. The main benefit that I got from it was that the tutoring and compute money was tax free (since I didn’t get the money personally, rather I used a card they provided me) and I didn’t have to worry about tax withholding throughout the year.

Secondly, the money. I agonized over how much money to ask for. This took me days. I asked myself how much I really needed, then I asked myself how much I would actually accept gladly with no regrets, then I balked at those numbers, even knowing that most people ask for too little, not too much. I still balk at the numbers, to be honest, but it would have been so much easier to write this if I had other grants to go off. So, in the interest of transparency and hopefully preventing someone else going through the same level of anguish, I’m sharing the full text of my grant request, including money requested (in Australian dollars, but you can always convert it) here....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Things I didn’t feel that guilty about before getting involved in effective altruism, published by Ada-Maaria Hyvärinen on December 28, 2022 on The Effective Altruism Forum.

Not wanting to move countries (“there would be a lot more effective work options if I lived elsewhere”)

Wanting a permanent work contract(“there would be a lot more effective work options with temporary contracts or grant-based pay”)

Not wanting to be an independent researcher(“it could potentially be an effective thing to do, and I wouldn’t have to worry about replaceability”)

Wanting to have a child(“if I didn’t want one I’d probably be much more flexible on the points above”)

Wanting to take some time off from work to take care of said child, in case I ever manage to have one(“although if I’m not having an impactful job by that time it probably won’t matter much anyway”)

Burning out(“it wastes time and sets a bad example”)

Feeling guilty about things(“I have read Replacing Guilt but I’m still having all these unproductive feelings”)

(Despite feeling guilty I’m doing ok – ultimately, a lot of this is just sadness about not having an unlimited altruism budget. I wanted to post this because I like reading about others’ experiences and thought someone else might like reading about mine. I don’t really need solutions proposals but if you have some, other readers might benefit from them.)

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.