Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Teaser: Hard-coding Transformer Models, published by MadHatter on December 12, 2021 on LessWrong.

Transformer models are incredibly powerful for natural language tasks (and they are starting to find uses in many other fields of machine learning). Unfortunately, it is nigh-impossible to interpret what goes on inside them. OR IS IT???

In this post, I am trying to gauge potential community interest in a strand of research that I have been doing in my spare time off and on for the past year and a half (roughly).

I have found that I can, with a fair amount of effort, hard-code the weights of a transformer model in order to perform some very crude versions of linguistic tasks. So far I have achieved English-to-French translation (on a toy corpus of about 150 sentences), text classification (is a sentence grammatical or not? on a toy corpus of a couple hundred sentences), and sentiment analysis (again on a limited corpus). These results are obviously not impressive compared to the state of the machine learning field, but I am pretty sure that they can all be drastically scaled up with the investment of some time and energy. Unfortunately, I have a fairly demanding day job, and haven't found the time and energy yet.

All of this is done by inspection (no gradient descent!). The process is a lot like programming, although it is more difficult than programming, at least right now for me. I am fairly certain that better tools and better notation can be developed to make the process easier. It is also almost certainly possible to combine hard-coding with gradient descent approaches to be able to scale these methods up in a slightly less labor-intensive way.

I think that these ideas could prove useful in alignment research - if we understand how a language model works in excruciating detail, it seems drastically more likely that we will be able to reason about and predict various misunderstandings rooted in the ambiguity of language. Given that language is (arguably) a fully general means of interacting with an artificial intelligence, it seems plausible to me that this work is on the critical path to alignment.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Teaser: Hard-coding Transformer Models, published by MadHatter on December 12, 2021 on LessWrong.

Transformer models are incredibly powerful for natural language tasks (and they are starting to find uses in many other fields of machine learning). Unfortunately, it is nigh-impossible to interpret what goes on inside them. OR IS IT???

In this post, I am trying to gauge potential community interest in a strand of research that I have been doing in my spare time off and on for the past year and a half (roughly).

I have found that I can, with a fair amount of effort, hard-code the weights of a transformer model in order to perform some very crude versions of linguistic tasks. So far I have achieved English-to-French translation (on a toy corpus of about 150 sentences), text classification (is a sentence grammatical or not? on a toy corpus of a couple hundred sentences), and sentiment analysis (again on a limited corpus). These results are obviously not impressive compared to the state of the machine learning field, but I am pretty sure that they can all be drastically scaled up with the investment of some time and energy. Unfortunately, I have a fairly demanding day job, and haven't found the time and energy yet.

All of this is done by inspection (no gradient descent!). The process is a lot like programming, although it is more difficult than programming, at least right now for me. I am fairly certain that better tools and better notation can be developed to make the process easier. It is also almost certainly possible to combine hard-coding with gradient descent approaches to be able to scale these methods up in a slightly less labor-intensive way.

I think that these ideas could prove useful in alignment research - if we understand how a language model works in excruciating detail, it seems drastically more likely that we will be able to reason about and predict various misunderstandings rooted in the ambiguity of language. Given that language is (arguably) a fully general means of interacting with an artificial intelligence, it seems plausible to me that this work is on the critical path to alignment.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hard-Coding Neural Computation, published by MadHatter on December 13, 2021 on The AI Alignment Forum.

Previously: Teaser: Hard-coding Transformer Models

Introduction

Transformer models are incredibly powerful for natural language tasks (and they are starting to find uses in many other fields of machine learning). Unfortunately, it is nigh-impossible to interpret what goes on inside them. OR IS IT???

I have found that I can, with a fair amount of effort, hard-code the weights of a transformer model in order to perform some very crude versions of linguistic tasks. So far I have achieved English-to-French translation (on a toy corpus of about 150 sentences), text classification (is a sentence grammatical or not? on a toy corpus of a couple hundred sentences), and sentiment analysis (again on a limited corpus). These results are obviously not impressive compared to the state of the machine learning field, but I am pretty sure that they can all be drastically scaled up with the investment of some time and energy. Unfortunately, I have a fairly demanding day job, and haven't found the time and energy yet.

All of this is done by inspection (no gradient descent!). The process is a lot like programming, although it is more difficult than programming, at least right now for me. I am fairly certain that better tools and better notation can be developed to make the process easier. It is also almost certainly possible to combine hard-coding with gradient descent approaches to be able to scale these methods up in a slightly less labor-intensive way.

I think that these ideas could prove useful in alignment research - if we understand how a language model works in excruciating detail, it seems drastically more likely that we will be able to reason about and predict various misunderstandings rooted in the ambiguity of language. Given that language is (arguably) a fully general means of interacting with an artificial intelligence, it seems plausible to me that this work is on the critical path to alignment.

Doneness Status

This post is a work-in-progress. I will be editing it as I go, mostly appending more content to the end, but I will also try to fix any errors or unclear parts as I notice them or commenters point them out.

So let's hard-code some neural computation! I have a very, very messy github repository where I've done my initial experiments, if you prefer to just jump into semi-working code. Otherwise, I will do my very best to explain the ideas from scratch in this post. I'm aiming this post at anyone who's willing to put in the work to understand it, so I'll try to at least give pointers to necessary background material, of which there is a fair amount.

What Can We Do Already?

Some primitive sentiment analysis using a Vanilla RNN:

Some very simple translation with a Transformer model:

How Should We Measure Success?

Before we explain how we get results, it seems worthwhile to talk about how to measure the performance of such a system. As in traditional machine learning, this can only be measured with respect to some dataset of inputs labeled with desired outputs. For a crude metric, we can look at the fraction of inputs that receive the desired output, versus the fraction that receive some other output. We can also look at more complex metrics, such as BLEU or ROUGE for sequence-to-sequence tasks.

In traditional machine learning, performance can only be (meaningfully) measured on a holdout set that was not used to train the algorithm. This is because performance tends to be much, much higher (if you're using the right architecture and hyperparameters for your task) on the training set (the set of data that was used to train the algorithm) than it will be for the test set (the set of data that has been held out). The whole purpose and challenge of machine learning, of cours...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Hard-Coding Neural Computation, published by MadHatter on December 13, 2021 on The AI Alignment Forum.

Previously: Teaser: Hard-coding Transformer Models

Introduction

Transformer models are incredibly powerful for natural language tasks (and they are starting to find uses in many other fields of machine learning). Unfortunately, it is nigh-impossible to interpret what goes on inside them. OR IS IT???

I have found that I can, with a fair amount of effort, hard-code the weights of a transformer model in order to perform some very crude versions of linguistic tasks. So far I have achieved English-to-French translation (on a toy corpus of about 150 sentences), text classification (is a sentence grammatical or not? on a toy corpus of a couple hundred sentences), and sentiment analysis (again on a limited corpus). These results are obviously not impressive compared to the state of the machine learning field, but I am pretty sure that they can all be drastically scaled up with the investment of some time and energy. Unfortunately, I have a fairly demanding day job, and haven't found the time and energy yet.

All of this is done by inspection (no gradient descent!). The process is a lot like programming, although it is more difficult than programming, at least right now for me. I am fairly certain that better tools and better notation can be developed to make the process easier. It is also almost certainly possible to combine hard-coding with gradient descent approaches to be able to scale these methods up in a slightly less labor-intensive way.

I think that these ideas could prove useful in alignment research - if we understand how a language model works in excruciating detail, it seems drastically more likely that we will be able to reason about and predict various misunderstandings rooted in the ambiguity of language. Given that language is (arguably) a fully general means of interacting with an artificial intelligence, it seems plausible to me that this work is on the critical path to alignment.

Doneness Status

This post is a work-in-progress. I will be editing it as I go, mostly appending more content to the end, but I will also try to fix any errors or unclear parts as I notice them or commenters point them out.

So let's hard-code some neural computation! I have a very, very messy github repository where I've done my initial experiments, if you prefer to just jump into semi-working code. Otherwise, I will do my very best to explain the ideas from scratch in this post. I'm aiming this post at anyone who's willing to put in the work to understand it, so I'll try to at least give pointers to necessary background material, of which there is a fair amount.

What Can We Do Already?

Some primitive sentiment analysis using a Vanilla RNN:

Some very simple translation with a Transformer model:

How Should We Measure Success?

Before we explain how we get results, it seems worthwhile to talk about how to measure the performance of such a system. As in traditional machine learning, this can only be measured with respect to some dataset of inputs labeled with desired outputs. For a crude metric, we can look at the fraction of inputs that receive the desired output, versus the fraction that receive some other output. We can also look at more complex metrics, such as BLEU or ROUGE for sequence-to-sequence tasks.

In traditional machine learning, performance can only be (meaningfully) measured on a holdout set that was not used to train the algorithm. This is because performance tends to be much, much higher (if you're using the right architecture and hyperparameters for your task) on the training set (the set of data that was used to train the algorithm) than it will be for the test set (the set of data that has been held out). The whole purpose and challenge of machine learning, of cours...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are big brains for processing sensory input?, published by lsusr on December 10, 2021 on LessWrong.

Elephants have bigger brains than human beings. But human beings are smarter. What's all the cranial matter doing?

I used to believe that a larger organism's brain needs to generate larger electrical impulses to move its muscles. But that's not how electrical engineering works[1]. It's not like your nerves power your muscles. Your brain just sends a signal to the muscles telling your muscles to activate.

Muscular motor units aren't even voltage coded. They're frequency coded. When your brain wants a muscle to contract harder it sends nerve impulses more frequently. Trying to raise the output voltage of a motor neuron is like trying to push the number "2" through a single bit of a binary computer.

It's true that I wouldn't want to use a tiny microcontroller to control a Predator drone but that has nothing to do with the power output of its motor control pins. It's because I want more processing power, fine motor control and robustness against damage.

Blue whales have gigantic brains. I don't think the brain size is for robustness against damage. It would be more efficient for evolution to invest in a thicker skull than metabolically-expensive brain matter. I don't think the brain size is for fine motor control either. What is a blue whale going to do with fine motor control? It almost doesn't even have limbs.

Encephalization

An organism's encephalization quotient is the ratio of its brain size to its body size. Mammals of similar encephalization quotients tend to have similar intelligence levels.

Big brains don't just consume lots of energy. They have lower reaction speeds too[2] (though slow reaction speeds may less important to large organisms because large muscles take longer to move). I think the slow reaction speed of large brains is related to the time it takes a signal to travel from one end of the brain to the other.

Big brains are expensive in multiple dimensions. If big organisms have big brains then the big brains must be doing something useful. Since the encephalization baseline is orthogonal to intelligence, the encephalization baseline must be driven by something other than intelligence. Mammals with bigger bodies require larger brains just to break even. A blue whale has a large brain despite its low need for intelligence. Why?

Sensory Input

If a blue whale's bigger brain isn't driven by damage resistance, motor control or intelligence needs then the only thing I can think of is processing sensory input.

In machine learning, processing an image datastream takes lots of compute. I'd be surprised if evolution didn't operate under a similar constraint—even with all its optimizations. A larger organism afford to collect more sensory data and afford to support more brainpower to process sensory data. It's easy to overlook all the compute evolution puts into processing sensory data for us. Our conscious mind doesn't think "retina cone #4,594,047 just activated". Our occipital lobe just tells us "cute African elephant".

African elephants are the largest land animals on Earth and they have the strongest sense of smell ever identified in a single species. I don't think this is a coincidence. In human brains, the sense of smell is handled by the piriform cortex in the cerebellum. Elephants have giant cerebellums. I don't think that is a coincidence either.

Cerebellums are circled in red. The relative size of a cerebellum within a brain is accurate but the brains are not to scale relative to each other. (Elephant brains are bigger than human brains.)

[See comment.]

Reality check. Don't mice have a better sense of smell than human beings?

Not necessarily. Mice are notoriously good sniffers and yet human beings beat them on a odor sensitivity test by biologist Matthia...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are big brains for processing sensory input?, published by lsusr on December 10, 2021 on LessWrong.

Elephants have bigger brains than human beings. But human beings are smarter. What's all the cranial matter doing?

I used to believe that a larger organism's brain needs to generate larger electrical impulses to move its muscles. But that's not how electrical engineering works[1]. It's not like your nerves power your muscles. Your brain just sends a signal to the muscles telling your muscles to activate.

Muscular motor units aren't even voltage coded. They're frequency coded. When your brain wants a muscle to contract harder it sends nerve impulses more frequently. Trying to raise the output voltage of a motor neuron is like trying to push the number "2" through a single bit of a binary computer.

It's true that I wouldn't want to use a tiny microcontroller to control a Predator drone but that has nothing to do with the power output of its motor control pins. It's because I want more processing power, fine motor control and robustness against damage.

Blue whales have gigantic brains. I don't think the brain size is for robustness against damage. It would be more efficient for evolution to invest in a thicker skull than metabolically-expensive brain matter. I don't think the brain size is for fine motor control either. What is a blue whale going to do with fine motor control? It almost doesn't even have limbs.

Encephalization

An organism's encephalization quotient is the ratio of its brain size to its body size. Mammals of similar encephalization quotients tend to have similar intelligence levels.

Big brains don't just consume lots of energy. They have lower reaction speeds too[2] (though slow reaction speeds may less important to large organisms because large muscles take longer to move). I think the slow reaction speed of large brains is related to the time it takes a signal to travel from one end of the brain to the other.

Big brains are expensive in multiple dimensions. If big organisms have big brains then the big brains must be doing something useful. Since the encephalization baseline is orthogonal to intelligence, the encephalization baseline must be driven by something other than intelligence. Mammals with bigger bodies require larger brains just to break even. A blue whale has a large brain despite its low need for intelligence. Why?

Sensory Input

If a blue whale's bigger brain isn't driven by damage resistance, motor control or intelligence needs then the only thing I can think of is processing sensory input.

In machine learning, processing an image datastream takes lots of compute. I'd be surprised if evolution didn't operate under a similar constraint—even with all its optimizations. A larger organism afford to collect more sensory data and afford to support more brainpower to process sensory data. It's easy to overlook all the compute evolution puts into processing sensory data for us. Our conscious mind doesn't think "retina cone #4,594,047 just activated". Our occipital lobe just tells us "cute African elephant".

African elephants are the largest land animals on Earth and they have the strongest sense of smell ever identified in a single species. I don't think this is a coincidence. In human brains, the sense of smell is handled by the piriform cortex in the cerebellum. Elephants have giant cerebellums. I don't think that is a coincidence either.

Cerebellums are circled in red. The relative size of a cerebellum within a brain is accurate but the brains are not to scale relative to each other. (Elephant brains are bigger than human brains.)

[See comment.]

Reality check. Don't mice have a better sense of smell than human beings?

Not necessarily. Mice are notoriously good sniffers and yet human beings beat them on a odor sensitivity test by biologist Matthia...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on EA Global London, published by PabloAMC on December 12, 2021 on The Effective Altruism Forum.

Summary: In this post, I aim to give an overview of my experience in EA Global London 2021.

Deciding and preparing to go

I started considering going to the Effective Altruism Global event in London some two months before the event. At the time, my main reason to go was that I am finishing a Ph.D. and my objective is to get a postdoc (or similar) on AI Safety. In particular, I was applying for a Future of Life Institute grant, and going to London would allow me to talk to knowledgeable people about my research proposal. Furthermore, I had some flying vouchers due to covid cancellations that either I could use or I would lose altogether; so it made sense to travel almost for free.

I had already taken part in EAG London in 2019, but since my objective was so clear and "high-stakes" careerwise this time, I took preparation more seriously. In particular, as soon as I was admitted and given access to the app, I looked for people to talk to. I scheduled quite a few meetings mostly with people I did not know personally but worked at places that could be useful to reach out to.

I also started looking for where to stay and can say I am very grateful to the members from EA Yale that allowed me to take the eighth bed in their room.

How did the event go

AI Safety related meetings

I think the event went really well. I had some really valuable conversations, and I cannot be but extremely grateful for people who were generous enough to give me half an hour of their time and useful feedback on my research proposal. Something I find particularly enjoyable about EA conferences is how easy it is to reach out to people of much higher status that would be inaccessible in other circumstances. I have been in academic conferences and it is oftentimes complicated to talk to people unless you have a previous reputation, which is quite scarce. On the other hand, even if you can talk to them it is really important that you think well through what you want to talk about with them, they will find it more enjoyable and easier to provide feedback and you will find it much more useful.

I met with top researchers like Geoffrey Irving, EA long term fund managers and while most of them had been carefully scheduled, I also had serendipitous meetings with new people (I even got to ask something to Victoria Krakovna!) and encounters with old friends from AI Safety Camps, etc I had not seen for a long time. It was encouraging to see so many of them doing great research work already :) I acknowledge that in contrast to 1-1s I did not put much focus on workshops or speed meetings, perhaps because I prefer more targetted conversations.

Other important topics

Besides AI Safety, I wanted to talk to people in the community about managing personal sacrifices to improve the world. It happens to be the case that my girlfriend is in Spain, and if I want to take an impactful research position it will most certainly be relatively far. I still want to do it, but it is also true that she makes me very happy, so this is a sacrifice. Unfortunately, nobody could offer me a silver bullet, but I did not expect such a thing to exist anyway, and I think talking to people about this helps me too. I am particularly grateful to Katie Glass for walking me through different options.

Something I was also very excited to do was to talk in person with Habiba Islam. I have had career coaching with her but never met her in person before. I'm really glad to have had the opportunity to meet her in person and am thankful for their supportive work at 80000hours. I also believe we should value more those in the community listening to people's worries and challenges since doing a lot of good can be quite demanding.

Concluding remarks.

I...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on EA Global London, published by PabloAMC on December 12, 2021 on The Effective Altruism Forum.

Summary: In this post, I aim to give an overview of my experience in EA Global London 2021.

Deciding and preparing to go

I started considering going to the Effective Altruism Global event in London some two months before the event. At the time, my main reason to go was that I am finishing a Ph.D. and my objective is to get a postdoc (or similar) on AI Safety. In particular, I was applying for a Future of Life Institute grant, and going to London would allow me to talk to knowledgeable people about my research proposal. Furthermore, I had some flying vouchers due to covid cancellations that either I could use or I would lose altogether; so it made sense to travel almost for free.

I had already taken part in EAG London in 2019, but since my objective was so clear and "high-stakes" careerwise this time, I took preparation more seriously. In particular, as soon as I was admitted and given access to the app, I looked for people to talk to. I scheduled quite a few meetings mostly with people I did not know personally but worked at places that could be useful to reach out to.

I also started looking for where to stay and can say I am very grateful to the members from EA Yale that allowed me to take the eighth bed in their room.

How did the event go

AI Safety related meetings

I think the event went really well. I had some really valuable conversations, and I cannot be but extremely grateful for people who were generous enough to give me half an hour of their time and useful feedback on my research proposal. Something I find particularly enjoyable about EA conferences is how easy it is to reach out to people of much higher status that would be inaccessible in other circumstances. I have been in academic conferences and it is oftentimes complicated to talk to people unless you have a previous reputation, which is quite scarce. On the other hand, even if you can talk to them it is really important that you think well through what you want to talk about with them, they will find it more enjoyable and easier to provide feedback and you will find it much more useful.

I met with top researchers like Geoffrey Irving, EA long term fund managers and while most of them had been carefully scheduled, I also had serendipitous meetings with new people (I even got to ask something to Victoria Krakovna!) and encounters with old friends from AI Safety Camps, etc I had not seen for a long time. It was encouraging to see so many of them doing great research work already :) I acknowledge that in contrast to 1-1s I did not put much focus on workshops or speed meetings, perhaps because I prefer more targetted conversations.

Other important topics

Besides AI Safety, I wanted to talk to people in the community about managing personal sacrifices to improve the world. It happens to be the case that my girlfriend is in Spain, and if I want to take an impactful research position it will most certainly be relatively far. I still want to do it, but it is also true that she makes me very happy, so this is a sacrifice. Unfortunately, nobody could offer me a silver bullet, but I did not expect such a thing to exist anyway, and I think talking to people about this helps me too. I am particularly grateful to Katie Glass for walking me through different options.

Something I was also very excited to do was to talk in person with Habiba Islam. I have had career coaching with her but never met her in person before. I'm really glad to have had the opportunity to meet her in person and am thankful for their supportive work at 80000hours. I also believe we should value more those in the community listening to people's worries and challenges since doing a lot of good can be quite demanding.

Concluding remarks.

I...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment, published by Rob Bensinger on December 12, 2021 on LessWrong.

I basically agree with Eliezer’s picture of things in the AGI interventions post.

But I’ve seen some readers rounding off Eliezer’s ‘the situation looks very dire’-ish statements to ‘the situation is hopeless’, and ‘solving alignment still looks to me like our best shot at a good future, but so far we’ve made very little progress, we aren’t anywhere near on track to solve the problem, and it isn’t clear what the best path forward is’-ish statements to ‘let’s give up on alignment’.

It’s hard to give a technical argument for ‘alignment isn’t doomed’, because I don’t know how to do alignment (and, to the best of my knowledge in December 2021, no one else does either). But I can give some of the more abstract reasons I think that.

I feel sort of wary of sharing a ‘reasons to be less pessimistic’ list, because it’s blatantly filtered evidence, it makes it easy to overcorrect, etc. In my experience, people tend to be way too eager to classify problems as either ‘easy’ or ‘impossible’; just adding more evidence may cause people to bounce back and forth between the two rather than planting flag in the middle ground.

I did write a version of 'reasons not to be maximally pesimistic' for a few friends in 2018. I’m warily fine with sharing that below, with the caveats ‘holy shit is this ever filtered evidence!’ and ‘these my own not-MIRI-vetted personal thoughts’. And 'this is a casual thing I jotted down for friends in 2018'.

Today, I would add some points (e.g., 'AGI may be surprisingly far off; timelines are hard to predict'), and I'd remove others (e.g., 'Nate and Eliezer feel pretty good about MIRI's current research'). Also, since the list is both qualitative and one-sided, it doesn’t reflect the fact that I’m quantitatively a bit more pessimistic now than I was in 2018.

Lo:

[...S]ome of the main reasons I'm not extremely pessimistic about artificial general intelligence outcomes.

(Warning: one-sided lists of considerations can obviously be epistemically bad. I mostly mean to correct for the fact that I see a lot of rationalists who strike me as overly pessimistic about AGI outcomes. Also, I don't try to argue for most of these points in any detail; I'm just trying to share my own views for others' stack.)

1. AGI alignment is just a technical problem, and humanity actually has a remarkably good record when it comes to solving technical problems. It's historically common for crazy-seeming goals to fall to engineering ingenuity, even in the face of seemingly insuperable obstacles.

Some of the underlying causes for this are 'it's hard to predict what clever ideas are hiding in the parts of your map that aren't filled in yet', and 'it's hard to prove a universal negation'. A universal negation is what you need in order to say that there's no clever engineering solution; whereas even if you've had ten thousand failed attempts, a single existence proof — a single solution to the problem — renders those failures totally irrelevant.

2. We don't know very much yet about the alignment problem. This isn't a reason for optimism, but it's a reason not to have confident pessimism, because no confident view can be justified by a state of uncertainty. We just have to learn more and do it the hard way and see how things go.

A blank map can feel like 'it's hopeless' for various reasons, even when you don't actually have enough Bayesian evidence to assert a technical problem is hopeless. For example: you think really hard about the problem and can't come up with a solution, which to some extent feels like there just isn't a solution. And: people aren't very good at knowing which parts of their map are blank, so it may feel like there aren't mo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment, published by Rob Bensinger on December 12, 2021 on LessWrong.

I basically agree with Eliezer’s picture of things in the AGI interventions post.

But I’ve seen some readers rounding off Eliezer’s ‘the situation looks very dire’-ish statements to ‘the situation is hopeless’, and ‘solving alignment still looks to me like our best shot at a good future, but so far we’ve made very little progress, we aren’t anywhere near on track to solve the problem, and it isn’t clear what the best path forward is’-ish statements to ‘let’s give up on alignment’.

It’s hard to give a technical argument for ‘alignment isn’t doomed’, because I don’t know how to do alignment (and, to the best of my knowledge in December 2021, no one else does either). But I can give some of the more abstract reasons I think that.

I feel sort of wary of sharing a ‘reasons to be less pessimistic’ list, because it’s blatantly filtered evidence, it makes it easy to overcorrect, etc. In my experience, people tend to be way too eager to classify problems as either ‘easy’ or ‘impossible’; just adding more evidence may cause people to bounce back and forth between the two rather than planting flag in the middle ground.

I did write a version of 'reasons not to be maximally pesimistic' for a few friends in 2018. I’m warily fine with sharing that below, with the caveats ‘holy shit is this ever filtered evidence!’ and ‘these my own not-MIRI-vetted personal thoughts’. And 'this is a casual thing I jotted down for friends in 2018'.

Today, I would add some points (e.g., 'AGI may be surprisingly far off; timelines are hard to predict'), and I'd remove others (e.g., 'Nate and Eliezer feel pretty good about MIRI's current research'). Also, since the list is both qualitative and one-sided, it doesn’t reflect the fact that I’m quantitatively a bit more pessimistic now than I was in 2018.

Lo:

[...S]ome of the main reasons I'm not extremely pessimistic about artificial general intelligence outcomes.

(Warning: one-sided lists of considerations can obviously be epistemically bad. I mostly mean to correct for the fact that I see a lot of rationalists who strike me as overly pessimistic about AGI outcomes. Also, I don't try to argue for most of these points in any detail; I'm just trying to share my own views for others' stack.)

1. AGI alignment is just a technical problem, and humanity actually has a remarkably good record when it comes to solving technical problems. It's historically common for crazy-seeming goals to fall to engineering ingenuity, even in the face of seemingly insuperable obstacles.

Some of the underlying causes for this are 'it's hard to predict what clever ideas are hiding in the parts of your map that aren't filled in yet', and 'it's hard to prove a universal negation'. A universal negation is what you need in order to say that there's no clever engineering solution; whereas even if you've had ten thousand failed attempts, a single existence proof — a single solution to the problem — renders those failures totally irrelevant.

2. We don't know very much yet about the alignment problem. This isn't a reason for optimism, but it's a reason not to have confident pessimism, because no confident view can be justified by a state of uncertainty. We just have to learn more and do it the hard way and see how things go.

A blank map can feel like 'it's hopeless' for various reasons, even when you don't actually have enough Bayesian evidence to assert a technical problem is hopeless. For example: you think really hard about the problem and can't come up with a solution, which to some extent feels like there just isn't a solution. And: people aren't very good at knowing which parts of their map are blank, so it may feel like there aren't mo...