Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Using Brain-Computer Interfaces to get more data for AI alignment, published by Robbo on November 7, 2021 on LessWrong.

[Epistemic status: sketchy. Sharing some ideas about Brain-Computer Interfaces and AI alignment that have come up in discussion with AI safety researchers, with the hopes of others elaborating on them.]

The purpose of this post is to sketch some ways that Brain Computer Interface (BCI) technology might help with various AI alignment techniques. Roughly, we can divide the strategic relevance of BCI technology into three broad categories.[1]

Enhancement. BCI technology could enhance human intelligence - for example by providing new sensory modalities, or augmenting cognition. [2]

Merge. BCI technology could enable a “merge” between AIs and humans. This is advocated by among others, Sam Altman and Elon Musk, and is the stated raison d'etre of Neuralink:

“I think if we can effectively merge with AI by improving the neural link between your cortex and your digital extension of yourself, which already...exists, just has a bandwidth issue. And then effectively you become an AI-human symbiote. And if that then is widespread, with anyone who wants it can have it, then we solve the control problem as well, we don't have to worry about some evil dictator AI because we are the AI collectively. That seems like the best outcome I can think of.” -Elon Musk, interview with Y Combinator (2016) [3]

On these proposals, humans are not merely enhanced - in some radical sense, humans merge with AI. It’s not entirely clear what these “merge” proposals mean, what merging would look like (Niplav: “It seems worrying that a complete company has been built on a vision that has no clearly articulated path to success.”), and "merge" as a alignment strategy seems to be quite unpopular in the AI safety community. In future work, I’d like to clarify merge proposals more.

Alignment aid. BCI allows us to get data from the brain that could improve the effectiveness of various AI alignment techniques. Whereas enhancement would indirectly help alignment by making alignment researchers smarter, alignment aid proposals are about directly improving the techniques themselves.

This post is about category 3. In conversation, several AI safety researchers have mentioned that BCI could help with AI alignment by giving us more data or better data. The purpose of this post is to sketch a few ways that this could go, and prompt further scrutiny of these ideas.

1. Types of Brain Computer Interfaces

Various technologies fall under the term “brain computer interface”. The commonality is that they record neural activity and transmit information to a computer for further use -- e.g., controlling an electronic device, moving a prosthetic arm, or playing pong. In April 2021 a prominent the Neuralink demo, showed a macaque monkey playing 'mind pong' using “a 1,024 electrode fully-implanted neural recording and [Bluetooth!] data transmission device”.

Some BCIs only “read” neural signals and use them, as in the ‘mind pong’ demo, while other BCIs also involve “writing” to the brain. While both reading and writing would be necessary for enhancement and merge, I will assume in this post that “reading” capabilities alone would be needed for the alignment aid proposals.

In assessing the state of BCI technology, we can look at three main criteria:

Invasiveness: do we have to (e.g.) drill a hole in someone’s skull, or not?

Resolution: how fine-grained is the data, both temporally and spatially?

Scale: how much of the brain can we record from?

Different kinds of BCI, each with different ways of being relevant to AI, score differently on these measures. Non-invasive techniques techniques are, as you would imagine, more common, and it is non-invasive techniques that are used in current commercial applications. From “...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Using Brain-Computer Interfaces to get more data for AI alignment, published by Robbo on November 7, 2021 on LessWrong.

[Epistemic status: sketchy. Sharing some ideas about Brain-Computer Interfaces and AI alignment that have come up in discussion with AI safety researchers, with the hopes of others elaborating on them.]

The purpose of this post is to sketch some ways that Brain Computer Interface (BCI) technology might help with various AI alignment techniques. Roughly, we can divide the strategic relevance of BCI technology into three broad categories.[1]

Enhancement. BCI technology could enhance human intelligence - for example by providing new sensory modalities, or augmenting cognition. [2]

Merge. BCI technology could enable a “merge” between AIs and humans. This is advocated by among others, Sam Altman and Elon Musk, and is the stated raison d'etre of Neuralink:

“I think if we can effectively merge with AI by improving the neural link between your cortex and your digital extension of yourself, which already...exists, just has a bandwidth issue. And then effectively you become an AI-human symbiote. And if that then is widespread, with anyone who wants it can have it, then we solve the control problem as well, we don't have to worry about some evil dictator AI because we are the AI collectively. That seems like the best outcome I can think of.” -Elon Musk, interview with Y Combinator (2016) [3]

On these proposals, humans are not merely enhanced - in some radical sense, humans merge with AI. It’s not entirely clear what these “merge” proposals mean, what merging would look like (Niplav: “It seems worrying that a complete company has been built on a vision that has no clearly articulated path to success.”), and "merge" as a alignment strategy seems to be quite unpopular in the AI safety community. In future work, I’d like to clarify merge proposals more.

Alignment aid. BCI allows us to get data from the brain that could improve the effectiveness of various AI alignment techniques. Whereas enhancement would indirectly help alignment by making alignment researchers smarter, alignment aid proposals are about directly improving the techniques themselves.

This post is about category 3. In conversation, several AI safety researchers have mentioned that BCI could help with AI alignment by giving us more data or better data. The purpose of this post is to sketch a few ways that this could go, and prompt further scrutiny of these ideas.

1. Types of Brain Computer Interfaces

Various technologies fall under the term “brain computer interface”. The commonality is that they record neural activity and transmit information to a computer for further use -- e.g., controlling an electronic device, moving a prosthetic arm, or playing pong. In April 2021 a prominent the Neuralink demo, showed a macaque monkey playing 'mind pong' using “a 1,024 electrode fully-implanted neural recording and [Bluetooth!] data transmission device”.

Some BCIs only “read” neural signals and use them, as in the ‘mind pong’ demo, while other BCIs also involve “writing” to the brain. While both reading and writing would be necessary for enhancement and merge, I will assume in this post that “reading” capabilities alone would be needed for the alignment aid proposals.

In assessing the state of BCI technology, we can look at three main criteria:

Invasiveness: do we have to (e.g.) drill a hole in someone’s skull, or not?

Resolution: how fine-grained is the data, both temporally and spatially?

Scale: how much of the brain can we record from?

Different kinds of BCI, each with different ways of being relevant to AI, score differently on these measures. Non-invasive techniques techniques are, as you would imagine, more common, and it is non-invasive techniques that are used in current commercial applications. From “...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What exactly is GPT-3's base objective?, published by Daniel Kokotajlo on November 10, 2021 on The AI Alignment Forum.

[Probably a noob question]

I'm thinking about what an inner alignment failure might look like for GPT-3. This would have to involve some deployment context in which GPT-3 performs significantly worse (by the standards of the base objective) than it did in training. (It would involve other things too, such as GPT-3 being a mesa-optimizer.)

But to say how well GPT-3 performs on some prompt not in the training dataset, we have to have a definition of the base objective that extends beyond the training dataset. If the base objective only makes sense in the context of the training dataset, then inner alignment failure is impossible by definition.

Is the base objective "Predict the next word?" Or is it "Predict the next word, supposing what you are reading is typical 2019 Internet text?" Or is it "Predict the next word, supposing what you are reading is a random-with-the-following-weights sample from dataset D? [where D is the dataset used to train GPT-3]" The third option is in some sense the best, because it most closely fits what we actually did to train GPT-3. But note that the logical extension of this line of reasoning is to prefer a fourth option: "Predict the next word, supposing what you are reading is a random-with-the-following-weights sample from dataset D' [where D' is like D except that it doesn't contain any of the bits of text that GPT-3 happened to not see in training, and the randomness weights are chosen to more accurately yield the data points that GPT-3 in fact saw]."

The problem with these last two answers is that they make it undefined how well GPT-3 performs on the base objective on any prompt that wasn't in D, which then rules out psuedo-alignment by definition.

From the Risks from Learned Optimization paper:

In such a case, we will use base objective to refer to whatever criterion the base optimizer was using to select between different possible systems and mesa-objective to refer to whatever criterion the mesa-optimizer is using to select between different possible outputs. In reinforcement learning (RL), for example, the base objective is generally the expected return. Because the mesa-objective is not specified by the programmers, mesa-optimization opens up the possibility of a mismatch between the base and mesa- objectives, wherein the mesa-objective might seem to perform well on the training environment but lead to bad performance off the training environment. We will refer to this case as pseudo-alignment below.

Expected return in a particular environment/distribution? Or not? If not, then you may be in a deployment context where you aren't updating the weights anymore and so there is no expected return, or at least it's close to 0 because there's only any return if you can convince people to start updating your weights again!

I worry I am just confused about all this. Hence why I'm asking. What is GPT-3's base objective?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What exactly is GPT-3's base objective?, published by Daniel Kokotajlo on November 10, 2021 on The AI Alignment Forum.

[Probably a noob question]

I'm thinking about what an inner alignment failure might look like for GPT-3. This would have to involve some deployment context in which GPT-3 performs significantly worse (by the standards of the base objective) than it did in training. (It would involve other things too, such as GPT-3 being a mesa-optimizer.)

But to say how well GPT-3 performs on some prompt not in the training dataset, we have to have a definition of the base objective that extends beyond the training dataset. If the base objective only makes sense in the context of the training dataset, then inner alignment failure is impossible by definition.

Is the base objective "Predict the next word?" Or is it "Predict the next word, supposing what you are reading is typical 2019 Internet text?" Or is it "Predict the next word, supposing what you are reading is a random-with-the-following-weights sample from dataset D? [where D is the dataset used to train GPT-3]" The third option is in some sense the best, because it most closely fits what we actually did to train GPT-3. But note that the logical extension of this line of reasoning is to prefer a fourth option: "Predict the next word, supposing what you are reading is a random-with-the-following-weights sample from dataset D' [where D' is like D except that it doesn't contain any of the bits of text that GPT-3 happened to not see in training, and the randomness weights are chosen to more accurately yield the data points that GPT-3 in fact saw]."

The problem with these last two answers is that they make it undefined how well GPT-3 performs on the base objective on any prompt that wasn't in D, which then rules out psuedo-alignment by definition.

From the Risks from Learned Optimization paper:

In such a case, we will use base objective to refer to whatever criterion the base optimizer was using to select between different possible systems and mesa-objective to refer to whatever criterion the mesa-optimizer is using to select between different possible outputs. In reinforcement learning (RL), for example, the base objective is generally the expected return. Because the mesa-objective is not specified by the programmers, mesa-optimization opens up the possibility of a mismatch between the base and mesa- objectives, wherein the mesa-objective might seem to perform well on the training environment but lead to bad performance off the training environment. We will refer to this case as pseudo-alignment below.

Expected return in a particular environment/distribution? Or not? If not, then you may be in a deployment context where you aren't updating the weights anymore and so there is no expected return, or at least it's close to 0 because there's only any return if you can convince people to start updating your weights again!

I worry I am just confused about all this. Hence why I'm asking. What is GPT-3's base objective?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 60-Year-Old Scientific Screwup That Helped Covid Kill [Wired.com], published by Elizabeth on November 9, 2021 on LessWrong.

This article describe's a scientist's attempt to figure out where the 5 micron number, and general belief that most respiratory diseases weren't airborne, came from. I have not fact checked it extensively, but last winter I did try to look into the general state of knowledge on airborne transmission vs fomites and found it weirdly empty, in ways that are consistent with this article.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The 60-Year-Old Scientific Screwup That Helped Covid Kill [Wired.com], published by Elizabeth on November 9, 2021 on LessWrong.

This article describe's a scientist's attempt to figure out where the 5 micron number, and general belief that most respiratory diseases weren't airborne, came from. I have not fact checked it extensively, but last winter I did try to look into the general state of knowledge on airborne transmission vs fomites and found it weirdly empty, in ways that are consistent with this article.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is there a clearly laid-out write-up of the case to drop Covid precautions?, published by AspiringRationalist on November 9, 2021 on LessWrong.

Some people are advocating dropping all or most Covid precautions within the rationalist community beyond requiring vaccines. I haven't seen the case for this explicitly laid out anywhere, though. Is there such a write-up anywhere?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Is there a clearly laid-out write-up of the case to drop Covid precautions?, published by AspiringRationalist on November 9, 2021 on LessWrong.

Some people are advocating dropping all or most Covid precautions within the rationalist community beyond requiring vaccines. I haven't seen the case for this explicitly laid out anywhere, though. Is there such a write-up anywhere?

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Steelman solitaire: how to take playing devil's advocate to the next level, published by KatWoods on November 8, 2021 on LessWrong.

This is an old post of mine from The Nonlinear blog that I thought really belonged on LessWrong.

I have a tool for thinking that I call “steelman solitaire”. I have found that it comes to much better conclusions than doing “free-style” thinking, so I thought I should share it with more people.

In summary, it consists of arguing with yourself in the program Workflowy or Roam, alternating between writing a steelman of an argument, a steelman of a counter-argument, a steelman of a counter-counter-argument, etc.

In this blog post I’ll first explain the broad steps, then list the benefits, and finally, go into more depth on how to do it.

Benefits

Structure forces you to do the thing you know you should do anyway. Most people reading this already know that it’s important to consider the best arguments on all sides instead of just considering the weakest on the other. Many already know that you can’t just consider a counter-argument then consider yourself done. However, it’s easy to forget to do so. The structure of this method makes you much more likely to follow through with your existing rational aspirations.

Clarifies thinking. I’m sure everybody has experienced a discussion that’s gone all over the place, and by the end, you’re more confused than when you started. Some points get lost and forgotten while others dominate. This approach helps to organize and clarify your thinking, revealing holes and strengths in different lines of thought.

More likely to change your mind. As much as we aspire not to, most people, even the most competent rationalists, will often become entrenched in a position due to the nature of conversations. In steelman solitaire, there’s no other person to lose face to or to hurt your feelings. This often makes it more likely to change your mind than a lot of other methods.

Makes you think much more deeply than usual. A common feature of people I would describe as “deep thinkers” is that they’ve often already thought of my counter-argument, and the counter-counter-counter-etc-argument. This method will make you really dig deep into an issue.

Dealing with steelmen that are compelling to you. A problem with a lot of debates is that what is convincing to the other person isn’t convincing to you, even though there are actually good arguments out there. This method allows you to think of those reasons instead of getting caught up with what another person thinks should convince you.

You can look back at why you came to the belief you have. Like most intellectually-oriented people, I have a lot of opinions. Sometimes so many that I forget why I came to hold them in the first place (but I vaguely remember that it was a good reason, I’m sure). Writing things down can help you refer back to them later and re-evaluate.

Better at coming to the truth than most methods. For the above reasons, I think that this method makes you more likely to come to accurate beliefs.

The broad idea

Strawmanning means presenting the opposing view in the least charitable light – often so uncharitably that it does not resemble the view that the other side actually holds. The term of steelmanning was invented as a counter to this; it means taking the opposing view and trying to present it in its strongest form. This has sometimes been criticized because often the alternative belief proposed by a steelman also isn’t what the other people actually believe. For example, there’s a steelman argument that states that the reason organic food is good is that monopolies are generally bad and Monsanto having a monopoly on food could lead to disastrous consequences. This might indeed be a belief held by some people who are pro-organic, but a huge percentage of peop...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Steelman solitaire: how to take playing devil's advocate to the next level, published by KatWoods on November 8, 2021 on LessWrong.

This is an old post of mine from The Nonlinear blog that I thought really belonged on LessWrong.

I have a tool for thinking that I call “steelman solitaire”. I have found that it comes to much better conclusions than doing “free-style” thinking, so I thought I should share it with more people.

In summary, it consists of arguing with yourself in the program Workflowy or Roam, alternating between writing a steelman of an argument, a steelman of a counter-argument, a steelman of a counter-counter-argument, etc.

In this blog post I’ll first explain the broad steps, then list the benefits, and finally, go into more depth on how to do it.

Benefits

Structure forces you to do the thing you know you should do anyway. Most people reading this already know that it’s important to consider the best arguments on all sides instead of just considering the weakest on the other. Many already know that you can’t just consider a counter-argument then consider yourself done. However, it’s easy to forget to do so. The structure of this method makes you much more likely to follow through with your existing rational aspirations.

Clarifies thinking. I’m sure everybody has experienced a discussion that’s gone all over the place, and by the end, you’re more confused than when you started. Some points get lost and forgotten while others dominate. This approach helps to organize and clarify your thinking, revealing holes and strengths in different lines of thought.

More likely to change your mind. As much as we aspire not to, most people, even the most competent rationalists, will often become entrenched in a position due to the nature of conversations. In steelman solitaire, there’s no other person to lose face to or to hurt your feelings. This often makes it more likely to change your mind than a lot of other methods.

Makes you think much more deeply than usual. A common feature of people I would describe as “deep thinkers” is that they’ve often already thought of my counter-argument, and the counter-counter-counter-etc-argument. This method will make you really dig deep into an issue.

Dealing with steelmen that are compelling to you. A problem with a lot of debates is that what is convincing to the other person isn’t convincing to you, even though there are actually good arguments out there. This method allows you to think of those reasons instead of getting caught up with what another person thinks should convince you.

You can look back at why you came to the belief you have. Like most intellectually-oriented people, I have a lot of opinions. Sometimes so many that I forget why I came to hold them in the first place (but I vaguely remember that it was a good reason, I’m sure). Writing things down can help you refer back to them later and re-evaluate.

Better at coming to the truth than most methods. For the above reasons, I think that this method makes you more likely to come to accurate beliefs.

The broad idea

Strawmanning means presenting the opposing view in the least charitable light – often so uncharitably that it does not resemble the view that the other side actually holds. The term of steelmanning was invented as a counter to this; it means taking the opposing view and trying to present it in its strongest form. This has sometimes been criticized because often the alternative belief proposed by a steelman also isn’t what the other people actually believe. For example, there’s a steelman argument that states that the reason organic food is good is that monopolies are generally bad and Monsanto having a monopoly on food could lead to disastrous consequences. This might indeed be a belief held by some people who are pro-organic, but a huge percentage of peop...