Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: App and book recommendations for people who want to be happier and more productive, published by KatWoods on November 6, 2021 on LessWrong.

If somebody asks you the same question more than ten times, that’s probably a sign it’s time to write a blog post about it. So after being asked by far more than ten people about what books and resources I recommend, both for EAs more broadly and for people interested in charity entrepreneurship more specifically, here it is! After spending roughly the last 15 years optimizing nearly constantly, these are the systems and products I recommend.

If you know better versions of what I recommend, please share it in the comments! While I recommend the apps I use, I’m sure there’s ones out there that do the same thing but in a better way and I just haven’t had the time or energy to pay the switching costs yet.

Apps and Extensions

Game-changers. Everybody should be using these.

Video Speed Controller. This allows you to hot key change the speed of videos anywhere on Chrome. It also doesn’t limit you to 2x, which so many apps do for some reason. You'll never (involuntarily) watch things on 1x again.

Clipboard history with CopyQ (Mac) or here for Windows. Absolute game changer. It remembers everything you've copy-pasted and you can click it from a list or use shortcut keys to paste them again in the future. Saves you so much time and hassle. It's hard to describe how much this changes how you use your computer.

Switch between your two most recent tabs. Use the shortcut Alt + Q to switch between your two most recently used tabs. It's like alt-tab but for tabs instead of windows. I can't imagine navigating a computer without this. It feels crippling. I know that there are better ones that allow you cycle through multiple tabs, not just your most recent. If you know of one, recommend it in the comments! I just haven't had the spare time to optimize this more.

Google docs quick create. Shortcut key or single click to automatically create a new google document or spreadsheet. Saves a ton of time.

Very good. Will be extremely useful for a lot of, but not all, people

Quickcompose. You know how easy it is to get distracted by your inbox when you need to send an email? Quick compose makes it so that you can open up a window that’s just a compose window so you can’t get distracted by new emails.

I don't care about cookies. Makes it so that Chrome automatically accepts cookies, so you never have to click "I accept cookies" ever again.

StayFocusd. Limits your time on social media.

News Feed Eradicator. Eradicates news feeds from Twitter and Facebook and probably other social media as well. Saves you innumerable hours a year.

SmileAlways. Makes it so that it automatically always opens Amazon Smile so that all your purchases contribute to charity. AMF is an option there. Nice passive impact opportunity.

Beeminder. You only pay if you don't achieve the goals you set for yourself. Stickk is similar and allows you to make the money to go to an anti-charity of your choice and easily add an accountability buddy. I like the UI of Beeminder better since it’s better for tracking my progress, but tastes will vary.

Focusmate. Have an immediate, remote, videochat accountability buddy to help you work on things for 30 or 60 minute increments. Really good for getting done those things you don’t usually want to do or if you’re having trouble focusing (e.g. paperwork, homework, etc).

Keysmith. Allows you to make a “program” by having the computer watch what you do and try to mimic your actions (e.g. open a window, type X, press these keys, etc). Allows you to code with no coding. It’s not perfect, but it works for a lot of things. I use it to automate really repetitive tasks. (Mac only. Probably a Windows version somewhere.)

Habitica. Gamified habit formation with a so...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: App and book recommendations for people who want to be happier and more productive, published by KatWoods on November 6, 2021 on LessWrong.

If somebody asks you the same question more than ten times, that’s probably a sign it’s time to write a blog post about it. So after being asked by far more than ten people about what books and resources I recommend, both for EAs more broadly and for people interested in charity entrepreneurship more specifically, here it is! After spending roughly the last 15 years optimizing nearly constantly, these are the systems and products I recommend.

If you know better versions of what I recommend, please share it in the comments! While I recommend the apps I use, I’m sure there’s ones out there that do the same thing but in a better way and I just haven’t had the time or energy to pay the switching costs yet.

Apps and Extensions

Game-changers. Everybody should be using these.

Video Speed Controller. This allows you to hot key change the speed of videos anywhere on Chrome. It also doesn’t limit you to 2x, which so many apps do for some reason. You'll never (involuntarily) watch things on 1x again.

Clipboard history with CopyQ (Mac) or here for Windows. Absolute game changer. It remembers everything you've copy-pasted and you can click it from a list or use shortcut keys to paste them again in the future. Saves you so much time and hassle. It's hard to describe how much this changes how you use your computer.

Switch between your two most recent tabs. Use the shortcut Alt + Q to switch between your two most recently used tabs. It's like alt-tab but for tabs instead of windows. I can't imagine navigating a computer without this. It feels crippling. I know that there are better ones that allow you cycle through multiple tabs, not just your most recent. If you know of one, recommend it in the comments! I just haven't had the spare time to optimize this more.

Google docs quick create. Shortcut key or single click to automatically create a new google document or spreadsheet. Saves a ton of time.

Very good. Will be extremely useful for a lot of, but not all, people

Quickcompose. You know how easy it is to get distracted by your inbox when you need to send an email? Quick compose makes it so that you can open up a window that’s just a compose window so you can’t get distracted by new emails.

I don't care about cookies. Makes it so that Chrome automatically accepts cookies, so you never have to click "I accept cookies" ever again.

StayFocusd. Limits your time on social media.

News Feed Eradicator. Eradicates news feeds from Twitter and Facebook and probably other social media as well. Saves you innumerable hours a year.

SmileAlways. Makes it so that it automatically always opens Amazon Smile so that all your purchases contribute to charity. AMF is an option there. Nice passive impact opportunity.

Beeminder. You only pay if you don't achieve the goals you set for yourself. Stickk is similar and allows you to make the money to go to an anti-charity of your choice and easily add an accountability buddy. I like the UI of Beeminder better since it’s better for tracking my progress, but tastes will vary.

Focusmate. Have an immediate, remote, videochat accountability buddy to help you work on things for 30 or 60 minute increments. Really good for getting done those things you don’t usually want to do or if you’re having trouble focusing (e.g. paperwork, homework, etc).

Keysmith. Allows you to make a “program” by having the computer watch what you do and try to mimic your actions (e.g. open a window, type X, press these keys, etc). Allows you to code with no coding. It’s not perfect, but it works for a lot of things. I use it to automate really repetitive tasks. (Mac only. Probably a Windows version somewhere.)

Habitica. Gamified habit formation with a so...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Study Guide, published by johnswentworth on November 6, 2021 on LessWrong.

This post is for students who hope to eventually work on technical problems we don’t understand, especially agency and AI alignment, and want to know what to study or practice.

Guiding Principles

Current alignment researchers have wildly different recommendations on paths into the field, usually correlated with the wildly different paths these researchers have themselves taken into the field. This also correlates with different kinds of work on alignment. This guide largely reflects my own path, and I think it is useful if you want to do the sort of research I do. That means fairly theoretical work (for now), very technical, drawing on models and math from a lot of different areas to understand real-world agents.

Specializing in Problems We Don’t Understand lays out a general framework which guides many of the recommendations here. I’ll also briefly go over some guiding principles more specific to choosing what (and how much) to study:

Breadth over depth

Practice generalizing concepts

Be able to model anything

High volume of knowledge

Breadth Over Depth

In general, study in any particular topic has decreasing marginal returns. The first exposure or two gives you the basic frames, tells you what kinds of questions to ask and what kinds of tools are available, etc. You may not remember everything, but you can at least remember what things to look up later if you need them - which is a pretty huge improvement over not even knowing that X is a thing you can look up at all!

Another way to frame this: problems-we-don’t-understand rely heavily on bringing in frames and tools from other fields. (If the frames and tools of this field were already sufficient, it wouldn’t be a problem-we-don’t-understand in the first place.) So, you want to have a very large library of frames and tools to apply. On the other hand, you don’t necessarily need very much depth in each frame or tool - just enough to recognize problems where it might apply and maybe try it out in a quick-and-dirty way.

Practice Generalizing Concepts

Bringing in frames and tools from other fields requires the ability to recognize and adapt those frames and tools for problems very different from the field in which we first learned them. So, practice generalizing concepts from one area to another is particularly important.

Unfortunately, this is not a focus in most courses. There are exceptions - applied math classes often involve applying tools in a wide variety of ways, and low-level physics courses often provide very good practice in applying a few mathematical tools to a wide variety of problems. Ultimately, though, this is something you should probably practice on your own a lot more than it’s practiced in class.

Keeping a list of 10-20 hard problems in the back of your mind, and trying out each new frame or tool on one of those problems, is a particularly useful technique to practice generalization.

Be Able To Model Anything

One common pitfall is to be drawn into areas which advertise extreme generality, but are rarely useful in practice. (A lot of high-level math is like this.) On the other hand, we still want a lot of breadth, including things which are not obviously useful to whatever problem we’re most interested in (e.g. alignment). After all, if the obviously-relevant tools sufficed, then it wouldn’t be a problem-we-don’t-understand in the first place.

To that end, it’s useful to look for frames/tools which are at least useful for something. Keeping a list of 10-20 hard problems in the back of your mind is one useful test for this. Another useful heuristic is “be able to model anything”: if there’s some system or phenomenon which you’re not sure how to model, even in principle, and field X has good tools for modelling it, then study f...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Study Guide, published by johnswentworth on November 6, 2021 on LessWrong.

This post is for students who hope to eventually work on technical problems we don’t understand, especially agency and AI alignment, and want to know what to study or practice.

Guiding Principles

Current alignment researchers have wildly different recommendations on paths into the field, usually correlated with the wildly different paths these researchers have themselves taken into the field. This also correlates with different kinds of work on alignment. This guide largely reflects my own path, and I think it is useful if you want to do the sort of research I do. That means fairly theoretical work (for now), very technical, drawing on models and math from a lot of different areas to understand real-world agents.

Specializing in Problems We Don’t Understand lays out a general framework which guides many of the recommendations here. I’ll also briefly go over some guiding principles more specific to choosing what (and how much) to study:

Breadth over depth

Practice generalizing concepts

Be able to model anything

High volume of knowledge

Breadth Over Depth

In general, study in any particular topic has decreasing marginal returns. The first exposure or two gives you the basic frames, tells you what kinds of questions to ask and what kinds of tools are available, etc. You may not remember everything, but you can at least remember what things to look up later if you need them - which is a pretty huge improvement over not even knowing that X is a thing you can look up at all!

Another way to frame this: problems-we-don’t-understand rely heavily on bringing in frames and tools from other fields. (If the frames and tools of this field were already sufficient, it wouldn’t be a problem-we-don’t-understand in the first place.) So, you want to have a very large library of frames and tools to apply. On the other hand, you don’t necessarily need very much depth in each frame or tool - just enough to recognize problems where it might apply and maybe try it out in a quick-and-dirty way.

Practice Generalizing Concepts

Bringing in frames and tools from other fields requires the ability to recognize and adapt those frames and tools for problems very different from the field in which we first learned them. So, practice generalizing concepts from one area to another is particularly important.

Unfortunately, this is not a focus in most courses. There are exceptions - applied math classes often involve applying tools in a wide variety of ways, and low-level physics courses often provide very good practice in applying a few mathematical tools to a wide variety of problems. Ultimately, though, this is something you should probably practice on your own a lot more than it’s practiced in class.

Keeping a list of 10-20 hard problems in the back of your mind, and trying out each new frame or tool on one of those problems, is a particularly useful technique to practice generalization.

Be Able To Model Anything

One common pitfall is to be drawn into areas which advertise extreme generality, but are rarely useful in practice. (A lot of high-level math is like this.) On the other hand, we still want a lot of breadth, including things which are not obviously useful to whatever problem we’re most interested in (e.g. alignment). After all, if the obviously-relevant tools sufficed, then it wouldn’t be a problem-we-don’t-understand in the first place.

To that end, it’s useful to look for frames/tools which are at least useful for something. Keeping a list of 10-20 hard problems in the back of your mind is one useful test for this. Another useful heuristic is “be able to model anything”: if there’s some system or phenomenon which you’re not sure how to model, even in principle, and field X has good tools for modelling it, then study f...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Chu are you?, published by Adele Lopez on November 6, 2021 on The AI Alignment Forum.

Maybe you've heard about something called a Chu space around here. But what the heck is a Chu space? And whatever it is, does it really belong with all the rich mathematical structures we know and love?

Say you have some stuff. What can you do with it?

Maybe it's made of little pieces, and you can do a different thing with each little piece.

But maybe the pieces are structured in a certain way, and you aren't allowed to do anything that would break this structure.

A Chu space is a versatile way of formalizing anything like that!

To represent something in a Chu space, we'll put the names of our pieces on the left. How about the rules? For a Chu space, the rules are about allowed ways to color our pieces. To represent these rules, we can simply make columns showing all the allowed ways we can color our pieces (just get rid of any columns that break the rules). Here's what a basic 3-element set (pictured on the left) looks like as a Chu space:

It doesn't have any sort of structure, so we show that by allowing all the possible colorings (with two colors). Chu spaces that don't have any rules (i.e. all colorings are allowed) are equivalent to sets.

What about the one with the arrows from above? How can we make an arrow into a coloring rule? One way we could do it is by stipulating that if there's an arrow x→y, we'll make a rule that if x is colored black, then y has to be colored black too, where x and y can stand in for any of the pieces. Here's what that Chu space looks like:

Spend a minute looking at the picture until you're convinced that our coloring rule is obeyed for every arrow on the left side of the picture. Any Chu space that has this kind of arrow rule has the structure of a poset.

There and back again

If we have two Chu spaces, say A and B, what sort of maps should we be able to make between them?

We'd like to be able to map the pieces of A to the pieces of B. So this part of our map will just be a normal function between sets:

f:Apieces→Bpieces

But we also want our maps to respect the rules of A as it maps to B. How can we check this?

Let's think through how we would be able to tell if a potential map did break the rules. The map will take pieces of A to pieces of B, so let's look at the pieces of B that got mapped onto, i.e. f(Apieces). It will help if we have a concrete example in mind, so let's consider a potential map that we think should break the rules: one that breaks the arrow structure of a poset by breaking it up into a mere set. For simplicity, we'll just have the pieces map f take pieces to pieces with the same label.

We can see now that if this potential map breaks the rules, it must be because one of the colorings for f(Apieces) is invalid. Specifically, the colorings highlighted in red:

The problem with these colorings is that there are no colorings in the original space for them to correspond to; so they must break one of the rules of the original space. So for a map that does follow the rules, we'll want to make sure every coloring in B has a corresponding coloring in A. This will be another function between sets, this time going backwards between the sets of colorings:

r:Bcolorings→Acolorings

Now let's look at an example of what we expect should be a legit map between Chu spaces.

Again, we are just mapping pieces to pieces with the same labels. This map is ok because even though B has some additional structure, it respects all of the structure from A.

The last part we need to define a map between Chu spaces is to determine exactly what it means for colorings of B to have a corresponding coloring in A. For a given piece x, and a given coloring s, we have a function that gives us the color of x using the coloring s:

ColorA:APieces×AColorings→Palette

We want ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Speaking of Stag Hunts, published by Duncan Sabien on November 6, 2021 on LessWrong.

This is an essay about the current state of the LessWrong community, and the broader EA/rationalist/longtermist communities that it overlaps and bridges, inspired mostly by the dynamics around these three posts. The concepts and claims laid out in Concentration of Force, which was originally written as part one of this essay, are important context for the thoughts below.

(And somewhere in the back of his mind was a small, small note of confusion, a sense of something wrong about that story; and it should have been a part of Harry's art to notice that tiny note, but he was distracted. For it is a sad rule that whenever you are most in need of your art as a rationalist, that is when you are most likely to forget it.)

I claim that something has gone a little bit wrong.

And as readers of many of my/other/essays/know, I claim that things going a little bit wrong is often actually quite a big problem.

I am not alone in thinking that the small scale matters. Tiny mental flinches, itty bitty little incentives, things thrown ever so slightly off course (and then never brought back). That small things often have outsized or cumulative effects is a popular view, either explicitly stated or discernible as an underlying assumption in the writings of Eliezer Yudkowsky, Nate Soares, Logan Brienne Strohl, Scott Alexander, Anna Salamon, and Andrew Critch, just to name a few.

Yet I nevertheless feel that I encounter resistance of various forms when attempting to point at small things as if they are important. Resistance rather than cooperative disagreement—impatience, dismissal, often condescension or sneering, sometimes projection and strawmanning.

This is absolutely at least in part due to my own clumsiness and confusion. A better version of me, more skilled at communication and empathy and bridging inferential gaps, would undoubtedly run into these problems less. Would better be able to recruit people's general enthusiasm for even rather dull and tedious and unsexy work, on that split-second level.

But it seems to me that I can't locate the problem entirely within myself. That there's something out there that's Actually Broken, and that it fights back, at least a little bit, when I try to point at it and fix it.

Here's to taking another shot at it.

Below is a non-exhaustive list of things which my brain will tend to do, if I don't put forth strategic effort to stop it:

Make no attempt to distinguish between what it feels is true and what is reasonable to believe.

Make no attempt to distinguish between what it feels is good and what is actually good.

Make wildly overconfident assertions that it doesn't even believe (that it will e.g. abandon immediately if forced to make a bet).

Weaponize equivocation and maximize plausible deniability à la motte-and-bailey, squeezing the maximum amount of wiggle room out of words and phrases. Say things that it knows will be interpreted a certain way, while knowing that they can be defended as if they meant something more innocent.

Neglect the difference between what things look like and what they actually are; fail to retain any skepticism on behalf of the possibility that I might be deceived by surface resemblance.

Treat a 70% probability of innocence and a 30% probability of guilt as a 100% chance that the person is 30% guilty (i.e. kinda guilty).

Wantonly project or otherwise read into people's actions and statements; evaluate those actions and statements by asking "what would have to be true inside my head, for me to output this behavior?" and then just assume that that's what's going on for them.

Pretend that it is speaking directly to a specific person while secretly spending the majority of its attention and optimization power on playing to some imagined la...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Speaking of Stag Hunts, published by Duncan Sabien on November 6, 2021 on LessWrong.

This is an essay about the current state of the LessWrong community, and the broader EA/rationalist/longtermist communities that it overlaps and bridges, inspired mostly by the dynamics around these three posts. The concepts and claims laid out in Concentration of Force, which was originally written as part one of this essay, are important context for the thoughts below.

(And somewhere in the back of his mind was a small, small note of confusion, a sense of something wrong about that story; and it should have been a part of Harry's art to notice that tiny note, but he was distracted. For it is a sad rule that whenever you are most in need of your art as a rationalist, that is when you are most likely to forget it.)

I claim that something has gone a little bit wrong.

And as readers of many of my/other/essays/know, I claim that things going a little bit wrong is often actually quite a big problem.

I am not alone in thinking that the small scale matters. Tiny mental flinches, itty bitty little incentives, things thrown ever so slightly off course (and then never brought back). That small things often have outsized or cumulative effects is a popular view, either explicitly stated or discernible as an underlying assumption in the writings of Eliezer Yudkowsky, Nate Soares, Logan Brienne Strohl, Scott Alexander, Anna Salamon, and Andrew Critch, just to name a few.

Yet I nevertheless feel that I encounter resistance of various forms when attempting to point at small things as if they are important. Resistance rather than cooperative disagreement—impatience, dismissal, often condescension or sneering, sometimes projection and strawmanning.

This is absolutely at least in part due to my own clumsiness and confusion. A better version of me, more skilled at communication and empathy and bridging inferential gaps, would undoubtedly run into these problems less. Would better be able to recruit people's general enthusiasm for even rather dull and tedious and unsexy work, on that split-second level.

But it seems to me that I can't locate the problem entirely within myself. That there's something out there that's Actually Broken, and that it fights back, at least a little bit, when I try to point at it and fix it.

Here's to taking another shot at it.

Below is a non-exhaustive list of things which my brain will tend to do, if I don't put forth strategic effort to stop it:

Make no attempt to distinguish between what it feels is true and what is reasonable to believe.

Make no attempt to distinguish between what it feels is good and what is actually good.

Make wildly overconfident assertions that it doesn't even believe (that it will e.g. abandon immediately if forced to make a bet).

Weaponize equivocation and maximize plausible deniability à la motte-and-bailey, squeezing the maximum amount of wiggle room out of words and phrases. Say things that it knows will be interpreted a certain way, while knowing that they can be defended as if they meant something more innocent.

Neglect the difference between what things look like and what they actually are; fail to retain any skepticism on behalf of the possibility that I might be deceived by surface resemblance.

Treat a 70% probability of innocence and a 30% probability of guilt as a 100% chance that the person is 30% guilty (i.e. kinda guilty).

Wantonly project or otherwise read into people's actions and statements; evaluate those actions and statements by asking "what would have to be true inside my head, for me to output this behavior?" and then just assume that that's what's going on for them.

Pretend that it is speaking directly to a specific person while secretly spending the majority of its attention and optimization power on playing to some imagined la...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concentration of Force, published by Duncan Sabien on November 6, 2021 on LessWrong.

This essay began as part one of a two-part essay. Part one is standalone and "timeless." Part two is focused on the local dynamics of the EA/rationality/longtermist communities and LessWrong in November of 2021. Following wise advice from Zack_M_Davis, I've split them into two separate posts. Nevertheless, I recommend that people intending to read both seriously consider reading them back-to-back, so that the content of this one is fresh in the mind. It's both something of a prerequisite and also relevantly context-setting.

Introduction

Concentration of force is a military concept (sometimes referred to as "mass"). It used to be concentration of forces, until innovations like machine guns and cruise missiles made gathering all of your actual personnel together into more of a liability.

The idea is simple. Essentially, there is a difference between relevant moments and irrelevant moments. Battles and non-battles, moments of engagement and moments between engagements.

At each relevant moment, you want to project locally superior or overwhelming force. Perhaps this means having the most soldiers/guns/tanks/planes actually present, or perhaps this just means having the right missiles pointed in the right directions.

If you are good at coordination and maneuver, you can concentrate force in most or every engagement, and consistently win even against an overall larger or more powerful opponent. This is how guerrilla warfare works—you choose the time and place of conflict in order to ensure that you outnumber the enemy in each specific encounter, and you fade into the mists before their reinforcements arrive.

I claim that the Grey Tribe generally, and rationalists/longtermists/EAs more specifically, and LessWrong the website and community even more specifically, are systematically and spectacularly failing at concentration of force. That none of those groups puts anything like sufficient strategic energy into ensuring that critical mass coheres at crucial moments, and that each would benefit from optimizing their ability to do so quickly, reliably, and effectively, and from thinking in terms of concentration of force as a matter of habit.

I anticipate a (reasonable) objection along the lines of "mistake theory rather than conflict theory!" or "generous tit-for-tat rather than vengeful tit-for-tat!" and I assert a further subclaim that the above advice is every bit as relevant for nonviolent and nonconfrontational frames. Actual conflicts are a vanishingly small subset of the times when concentration of force is a relevant principle; the "force" in question could just as easily be e.g. "calm, generous, charitable, level-headed, clear-minded, skilled communicators arriving at exactly the moment when things were about to become wastefully contentious and adversarial."

(Indeed, that's a preview of the recommendation I have for LessWrong specifically.)

TAPs: A Motivating Example

There's a picture I tend to draw quite frequently when giving people crash courses in CFAR-esque rationality, and it looks like this:

The idea behind the picture is that you're trucking along, living a generally good and happy life, and then something happens, and you find yourself in the sad timeline. You ate an entire package of Oreos, despite intending to lose weight. You got in another fight with your romantic partner, despite really not wanting to. You road raged, you failed to finish the presentation before the deadline, you spent all evening on Reddit instead of thinking about your research, you somehow never called them back and now it's too awkward.

The point of showing people this picture is to draw their attention to two key facts:

For most goals and values, there actually exists a moment-of-departure from a p...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concentration of Force, published by Duncan Sabien on November 6, 2021 on LessWrong.

This essay began as part one of a two-part essay. Part one is standalone and "timeless." Part two is focused on the local dynamics of the EA/rationality/longtermist communities and LessWrong in November of 2021. Following wise advice from Zack_M_Davis, I've split them into two separate posts. Nevertheless, I recommend that people intending to read both seriously consider reading them back-to-back, so that the content of this one is fresh in the mind. It's both something of a prerequisite and also relevantly context-setting.

Introduction

Concentration of force is a military concept (sometimes referred to as "mass"). It used to be concentration of forces, until innovations like machine guns and cruise missiles made gathering all of your actual personnel together into more of a liability.

The idea is simple. Essentially, there is a difference between relevant moments and irrelevant moments. Battles and non-battles, moments of engagement and moments between engagements.

At each relevant moment, you want to project locally superior or overwhelming force. Perhaps this means having the most soldiers/guns/tanks/planes actually present, or perhaps this just means having the right missiles pointed in the right directions.

If you are good at coordination and maneuver, you can concentrate force in most or every engagement, and consistently win even against an overall larger or more powerful opponent. This is how guerrilla warfare works—you choose the time and place of conflict in order to ensure that you outnumber the enemy in each specific encounter, and you fade into the mists before their reinforcements arrive.

I claim that the Grey Tribe generally, and rationalists/longtermists/EAs more specifically, and LessWrong the website and community even more specifically, are systematically and spectacularly failing at concentration of force. That none of those groups puts anything like sufficient strategic energy into ensuring that critical mass coheres at crucial moments, and that each would benefit from optimizing their ability to do so quickly, reliably, and effectively, and from thinking in terms of concentration of force as a matter of habit.

I anticipate a (reasonable) objection along the lines of "mistake theory rather than conflict theory!" or "generous tit-for-tat rather than vengeful tit-for-tat!" and I assert a further subclaim that the above advice is every bit as relevant for nonviolent and nonconfrontational frames. Actual conflicts are a vanishingly small subset of the times when concentration of force is a relevant principle; the "force" in question could just as easily be e.g. "calm, generous, charitable, level-headed, clear-minded, skilled communicators arriving at exactly the moment when things were about to become wastefully contentious and adversarial."

(Indeed, that's a preview of the recommendation I have for LessWrong specifically.)

TAPs: A Motivating Example

There's a picture I tend to draw quite frequently when giving people crash courses in CFAR-esque rationality, and it looks like this:

The idea behind the picture is that you're trucking along, living a generally good and happy life, and then something happens, and you find yourself in the sad timeline. You ate an entire package of Oreos, despite intending to lose weight. You got in another fight with your romantic partner, despite really not wanting to. You road raged, you failed to finish the presentation before the deadline, you spent all evening on Reddit instead of thinking about your research, you somehow never called them back and now it's too awkward.

The point of showing people this picture is to draw their attention to two key facts:

For most goals and values, there actually exists a moment-of-departure from a p...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comments on OpenPhil's Interpretability RFP, published by Paul Christiano on November 5, 2021 on The AI Alignment Forum.

I'm very excited about research that tries to deeply understand how neural networks are thinking, and especially to understand tiny parts of neural networks without too much concern for scalability, as described in OpenPhil's recent RFP or the Circuits thread on Distill.

I want to give some detail on the kind of research I'm most excited about and why.

(Disclaimer: These are off-the-cuff thoughts. I'm not at all an expert in this area, just someone who cares a lot about alignment and has opinions.)

Why I'm excited about interpretability

I'm very scared of powerful AI doing something deliberately destructive that no one wanted it to do. If we had a reasonably complete story about how a neural network was thinking, then I doubt this failure mode would take us by surprise---we'd be able to see that the model was only behaving well on the training distribution because we were watching, and understand that the model was capable and inclined to cause harm when deployed. I think that even a partial understanding could provide a huge amount of clarity on open disputes about how models will generalize and when they may pose risks.

So the simplest positive case for interpretability is that it gives us a much clearer warning about an impending problem. That may or may not be enough for us to get our act together and fix the problem, but it's at least a start.

Beyond that, interpretability plays a central role in many proposals for training safe AI. For example, we might use interpretability to help identify cases where the model would behave badly, to help ensure that the model never develops deceptive intent, or to help understand what the model knows about the world so that we can better evaluate its outputs. We don't really know which of these approaches will pan out, but the fact that interpretability keeps coming up makes it seem important beyond just a warning shot.

Some of these approaches would put more "strain" on an interpretability method. If we optimize models so that they don't appear to think certain kinds of thoughts, then gradient descent will find a way to hide those thoughts. I don't know if interpretability techniques would ever be strong enough to stand up to this kind of optimization pressure, but I think interpretability is incredibly useful anyway---it still gives you a key warning shot, and many of the potential applications in the training process wouldn't create as much adversarial pressure.

It's possible for models to eventually hide their cognition even if we don't apply gradient descent. But I think this may be extremely hard; that the process of doing so is likely to itself be visible if we are careful; and that even if transformative AI is smart enough to obfuscate its thoughts we would still learn a huge amount from studying dumber models.

I'm significantly more interested and optimistic about mechanistic interpretability than I was 4 years ago. So from my perspective Chris Olah, who has championed this perspective during the interim, gets a lot of epistemic credit here.

Why I'm not worried about scalability

I think I basically totally agree with the OpenPhil RFP, and in particular think the most exciting work is getting an extremely detailed mechanical understanding of a tiny part of a neural network. If anything I'd lean slightly further in that direction than Chris does in the RFP.

Expanding on that: it seems to me that the current bottleneck for interpretability is that we almost never have a good understanding of what's going on, not that we have good methods that scale poorly. I think people should mostly not worry about scalability while we try to get really good at understanding what's up with small pieces of neural net...