Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Declustering, reclustering, and filling in thingspace, published by Stuart Armstrong on December 6, 2021 on The AI Alignment Forum.

Eliezer has talked about thingspace - an abstract space in which objects lie, defined by their properties on some scales. In thingspace, there are some more-or-less natural categories, such as the "bird" category, that correspond to clusters in this thingspace. Drawing the boundary/carving reality at the joints, can be thought of as using definitions that efficiently separate these clusters[1].

He said about the thingspace of birds:

The central clusters of robins and sparrows glowing brightly with highly typical birdness; satellite clusters of ostriches and penguins [further from the center] glowing more dimly with atypical birdness, and Abraham Lincoln a few megaparsecs away and glowing not at all.

That, however, is how things stand in the world as it is. What if, for some reason, we had a lot of power in the world, and we wanted to break these neat-ish clusters, how would we go about it?

Remaking the clusters

Suppose first we wanted to break away some of those satellites away. What if we wanted penguins to be clearly distinct from birds? Penguins are land-waddlers and good swimmers, living in cold climates. So let's fill out this category, call it, maybe, Panguens.

Panguens all live in cold climates, waddle on land, and are good swimmers. Most are mammals, and give birth to live young. Some are more like the duck-billed platypus, and lay eggs (though they suckle their young). And all of them, including penguins, can reproduce with each other (how? genetic engineering magic). Thus there are a lot of intermediate species, including some that are almost-penguins except differing on a few details of their anatomy. All have fins of some sort, but only 10% have penguin-like wing-fins.

Panguens are clearly a cluster, quite a tight one as animals go (given they can reproduce with each other). And they are clearly not birds. And penguins clearly belong inside them, rather than with birds; at most, some pedantic classifier would label penguins as "bird-like panguens".

So, by filling in extra possibilities in thingspace, we have moved penguins to a new category. Graphically, it looks like starting with:

and moving to:

What if we wanted to make Abraham Lincoln into a bird[2]? We can connect the categories of birds and humans by creating all sort of bird-human hybrids. But this is not quite enough. Bird-humans is certainly a reasonable category in this world. But humans are very close to other mammals, especially great apes; Lincoln is clearly a bird-human, but also clearly a mammal. How can we excise him from this category?

Well, we might just go around killing all the apes and monkeys, making the bird-human category more natural. But we don't need to do that. We could introduce instead mammal-dog hybrids: all mammal species (except humans) have a smooth transition to lizards, filled with intermediate species. Now the new natural definition of "mammal" includes "must have a hybrid version with lizards". All mammals fit this definition - except for humans, who have their own thing with birds. Much more natural to divide the "bird-humans" cleanly from the "mammal-lizards", and note that Lincoln is a bird-human that shares some features with some mammal-lizards.

The more we want to separate humans from mammals, the more we increase the category "mammals" in directions away from humans, and increase the category "bird-humans" in directions away from other mammals. Graphically we start with:

and move to:

Why we should care: AIs, examples, and definitions

This is an amusing mental exercise, but what's the point? The point is that this is another variant of the standard failure mode of powerful AIs.

Let's go with that old chestnut, and assume that we hav...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More Christiano, Cotra, and Yudkowsky on AI progress, published by Eliezer Yudkowsky on December 6, 2021 on The AI Alignment Forum.

This is part one of: More Christiano, Cotra, and Yudkowsky on AI progress, by Eliezer Yudkowsky

This post is a transcript of a discussion between Paul Christiano, Ajeya Cotra, and Eliezer Yudkowsky (with some comments from Rob Bensinger, Richard Ngo, and Carl Shulman), continuing from 1, 2, and 3.

Color key:

Chat by Paul and Eliezer Other chat

10.2. Prototypes, historical perspectives, and betting

[Bensinger][4:25]

I feel confused about the role "innovations are almost always low-impact" plays in slow-takeoff-ish views.

Suppose I think that there's some reachable algorithm that's different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.

The existence or nonexistence of such an algorithm is just a fact about the physical world. If I imagine one universe where such an algorithm exists, and another where it doesn't, I don't see why I should expect that one of those worlds has more discontinuous change in GWP, ship sizes, bridge lengths, explosive yields, etc. (outside of any discontinuities caused by the advent of humans and the advent of AGI)? What do these CS facts have to do with the other facts?

But AI Impacts seems to think there's an important connection, and a large number of facts of the form 'steamships aren't like nukes' seem to undergird a lot of Paul's confidence that the scenario I described --

("there's some reachable algorithm that's different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.")

-- is crazy talk. (Unless I'm misunderstanding. As seems actually pretty likely to me!)

(E.g., Paul says "To me your model just seems crazy, and you are saying it predicts crazy stuff at the end but no crazy stuff beforehand", and one of the threads of the timelines conversation has been Paul asking stuff like "do you want to give any example other than nuclear weapons of technologies with the kind of discontinuous impact you are describing?".)

Possibilities that came to mind for me:

1. The argument is 'reality keeps surprising us with how continuous everything else is, so we seem to have a cognitive bias favoring discontinuity, so we should have a skeptical prior about our ability to think our way to 'X is discontinuous' since our brains are apparently too broken to do that well?

(But to get from 1 to 'discontinuity models are batshit' we surely need something more probability-mass-concentrating than just a bias argument?)

2. The commonality between steamship sizes, bridge sizes, etc. and AGI is something like 'how tractable is the world?'. A highly tractable world, one whose principles are easy to understand and leverage, will tend to have more world-shatteringly huge historical breakthroughs in various problems, and will tend to see a larger impact from the advent of humans and the advent of AGI.

Our world looks much less tractable, so even if there's a secret sauce to building AGI, we should expect the resultant AGI to be a lot less impactful.

[Ngo][5:06]

I endorse #2 (although I think more weakly than Paul does) and would also add #3: another commonality is something like "how competitive is innovation?"

[Shulman][8:22]

@RobBensinger It's showing us a fact about the vast space of ideas and technologies we've already explored that they are not so concentrated and lumpy that the law of large numbers doesn't work well as a first approximation in a world with thousands or millions of people contributing. And that specifically includes past computer science innovation.

So the 'we find a secret sauce algorithm that causes a massive unprecedented performance jump, without crappier predecessors' is a 'separate, additional miracle' at ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: More Christiano, Cotra, and Yudkowsky on AI progress, published by Eliezer Yudkowsky on December 6, 2021 on The AI Alignment Forum.

This is part one of: More Christiano, Cotra, and Yudkowsky on AI progress, by Eliezer Yudkowsky

This post is a transcript of a discussion between Paul Christiano, Ajeya Cotra, and Eliezer Yudkowsky (with some comments from Rob Bensinger, Richard Ngo, and Carl Shulman), continuing from 1, 2, and 3.

Color key:

Chat by Paul and Eliezer Other chat

10.2. Prototypes, historical perspectives, and betting

[Bensinger][4:25]

I feel confused about the role "innovations are almost always low-impact" plays in slow-takeoff-ish views.

Suppose I think that there's some reachable algorithm that's different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.

The existence or nonexistence of such an algorithm is just a fact about the physical world. If I imagine one universe where such an algorithm exists, and another where it doesn't, I don't see why I should expect that one of those worlds has more discontinuous change in GWP, ship sizes, bridge lengths, explosive yields, etc. (outside of any discontinuities caused by the advent of humans and the advent of AGI)? What do these CS facts have to do with the other facts?

But AI Impacts seems to think there's an important connection, and a large number of facts of the form 'steamships aren't like nukes' seem to undergird a lot of Paul's confidence that the scenario I described --

("there's some reachable algorithm that's different from current approaches, and can do par-human scientific reasoning without requiring tons of compute.")

-- is crazy talk. (Unless I'm misunderstanding. As seems actually pretty likely to me!)

(E.g., Paul says "To me your model just seems crazy, and you are saying it predicts crazy stuff at the end but no crazy stuff beforehand", and one of the threads of the timelines conversation has been Paul asking stuff like "do you want to give any example other than nuclear weapons of technologies with the kind of discontinuous impact you are describing?".)

Possibilities that came to mind for me:

1. The argument is 'reality keeps surprising us with how continuous everything else is, so we seem to have a cognitive bias favoring discontinuity, so we should have a skeptical prior about our ability to think our way to 'X is discontinuous' since our brains are apparently too broken to do that well?

(But to get from 1 to 'discontinuity models are batshit' we surely need something more probability-mass-concentrating than just a bias argument?)

2. The commonality between steamship sizes, bridge sizes, etc. and AGI is something like 'how tractable is the world?'. A highly tractable world, one whose principles are easy to understand and leverage, will tend to have more world-shatteringly huge historical breakthroughs in various problems, and will tend to see a larger impact from the advent of humans and the advent of AGI.

Our world looks much less tractable, so even if there's a secret sauce to building AGI, we should expect the resultant AGI to be a lot less impactful.

[Ngo][5:06]

I endorse #2 (although I think more weakly than Paul does) and would also add #3: another commonality is something like "how competitive is innovation?"

[Shulman][8:22]

@RobBensinger It's showing us a fact about the vast space of ideas and technologies we've already explored that they are not so concentrated and lumpy that the law of large numbers doesn't work well as a first approximation in a world with thousands or millions of people contributing. And that specifically includes past computer science innovation.

So the 'we find a secret sauce algorithm that causes a massive unprecedented performance jump, without crappier predecessors' is a 'separate, additional miracle' at ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #4, published by Zvi on December 6, 2021 on LessWrong.

Previous Omicron updates: #1, #2, #3. Last weekly non-Omicron update.

An introductory word: Thanks to Dominic Cummings, I have a lot of new readers, many from the United Kingdom, so I want to welcome all of you, and I hope at least some of you will stay when I turn to non-Covid questions. I am American so these posts focus on the United States, but I keep an eye elsewhere too, and mostly we’re all in this one together and the same conclusions apply. If you have good news sources and follows to keep a better eye on the UK or Europe for Covid purposes, or data sources anywhere I may not have noticed, I invite you to share them in the comments.

The constant refrain from all sources is ‘we will know more soon.’ Soon is continuously arriving. The new information we are learning hasn’t completely ruled out many possibilities, but it is broadly consistent with Omicron spreading fast and having a high degree of immune erosion/escape in terms of infection but not protection against severe disease.

Thus, to headline the core takeaways at the top, here’s broadly where I’m at as of now (probability estimates at the end, as before). I’d be surprised if any of these were wrong.

Omicron spreads far more rapidly than Delta and is going to take over.

This will come at us fast. Omicron will be the majority strain by the end of January. This happening by end of year is on the table.

Omicron re-infects those who have already been sick, or breaks through to those who have been vaccinated, much more than Delta.

Previous infection continues to protect against severe disease, hospitalization and death.

Omicron doesn’t cause substantially more severe disease than Delta when it infects you, but we don’t know if it causes less severe disease yet, our evidence is ambiguous.

Omicron cases on average are much more mild because there will be a much higher percentage of re-infections and breakthrough cases, which are highly protected against severe disease.

The big peak is probably within a few months, so you don’t have time to wait for an Omicron-targeted shot. The current shots will still work against severe disease, so strongly consider getting vaccinated or boosted if you can.

The best thing we can do to minimize the harm Omicron causes is to ramp up production of therapeutics, especially Paxlovid, as rapidly as possible, along with the necessary tests and other logistics to get the treatments to people in time to matter. The goal is mitigation at this point, not prevention.

There is going to be a period early in 2022 when there are quite a lot of Omicron cases, such that it will be difficult to remain uninfected and it will likely be difficult to get any kind of medical treatment at a hospital. Be ready.

Also be ready in case of lockdowns and other government restrictions, especially if you live in Europe where they’ve shown a willingness to use them. And if you’re immunocompromised or otherwise at high enough risk you need to be sure to not get Omicron, then the price of success is getting super paranoid soon and lock down hard, for at least several months.

In Search of Bayesian Evidence

How rapidly does Omicron spread? How far has it spread already?

Any given infection or reported group of infections, or even piece of data, is unlikely to be conclusive. But each one does have a likelihood ratio of how common it is to witness it in worlds where Omicron will spread extremely rapidly in the West, versus worlds where Omicron doesn’t do that. Similarly, they have ratios for how often how much of that comes from immune escape versus baseline infectiousness, how much protection remains against severe disease and death, and what the baseline severity of Omicron is.

Then we need to consider what we saw relative to what we expected to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Omicron Post #4, published by Zvi on December 6, 2021 on LessWrong.

Previous Omicron updates: #1, #2, #3. Last weekly non-Omicron update.

An introductory word: Thanks to Dominic Cummings, I have a lot of new readers, many from the United Kingdom, so I want to welcome all of you, and I hope at least some of you will stay when I turn to non-Covid questions. I am American so these posts focus on the United States, but I keep an eye elsewhere too, and mostly we’re all in this one together and the same conclusions apply. If you have good news sources and follows to keep a better eye on the UK or Europe for Covid purposes, or data sources anywhere I may not have noticed, I invite you to share them in the comments.

The constant refrain from all sources is ‘we will know more soon.’ Soon is continuously arriving. The new information we are learning hasn’t completely ruled out many possibilities, but it is broadly consistent with Omicron spreading fast and having a high degree of immune erosion/escape in terms of infection but not protection against severe disease.

Thus, to headline the core takeaways at the top, here’s broadly where I’m at as of now (probability estimates at the end, as before). I’d be surprised if any of these were wrong.

Omicron spreads far more rapidly than Delta and is going to take over.

This will come at us fast. Omicron will be the majority strain by the end of January. This happening by end of year is on the table.

Omicron re-infects those who have already been sick, or breaks through to those who have been vaccinated, much more than Delta.

Previous infection continues to protect against severe disease, hospitalization and death.

Omicron doesn’t cause substantially more severe disease than Delta when it infects you, but we don’t know if it causes less severe disease yet, our evidence is ambiguous.

Omicron cases on average are much more mild because there will be a much higher percentage of re-infections and breakthrough cases, which are highly protected against severe disease.

The big peak is probably within a few months, so you don’t have time to wait for an Omicron-targeted shot. The current shots will still work against severe disease, so strongly consider getting vaccinated or boosted if you can.

The best thing we can do to minimize the harm Omicron causes is to ramp up production of therapeutics, especially Paxlovid, as rapidly as possible, along with the necessary tests and other logistics to get the treatments to people in time to matter. The goal is mitigation at this point, not prevention.

There is going to be a period early in 2022 when there are quite a lot of Omicron cases, such that it will be difficult to remain uninfected and it will likely be difficult to get any kind of medical treatment at a hospital. Be ready.

Also be ready in case of lockdowns and other government restrictions, especially if you live in Europe where they’ve shown a willingness to use them. And if you’re immunocompromised or otherwise at high enough risk you need to be sure to not get Omicron, then the price of success is getting super paranoid soon and lock down hard, for at least several months.

In Search of Bayesian Evidence

How rapidly does Omicron spread? How far has it spread already?

Any given infection or reported group of infections, or even piece of data, is unlikely to be conclusive. But each one does have a likelihood ratio of how common it is to witness it in worlds where Omicron will spread extremely rapidly in the West, versus worlds where Omicron doesn’t do that. Similarly, they have ratios for how often how much of that comes from immune escape versus baseline infectiousness, how much protection remains against severe disease and death, and what the baseline severity of Omicron is.

Then we need to consider what we saw relative to what we expected to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are there alternative to solving value transfer and extrapolation?, published by Stuart Armstrong on December 6, 2021 on The AI Alignment Forum.

Thanks to Rebecca Gorman for the discussions that inspired these ideas.

We recently argued that full understanding of value extrapolation[1] was necessary and almost sufficient for solving the AI alignment problem, due to the existence of morally underdefined situations.

People seemed to generally agree that some knowledge of human morality was needed for a safe powerful AI. But it wasn't clear this AI needed almost-full knowledge of human morality, as well as value extrapolation. We got some high-quality alternative suggestions. In this post, we'll try and distil those ideas and analyse them.

AI seems to behave reasonably

Steven Byrnes's position, if we understand it correctly, is that the AI should learn to behave in non-dangerous seeming ways[2].

Our phrasing of the idea would be (apologies for any misinterpretations):

There are many behaviours that agents can follow. Typical humans follow typical behaviours, to achieve typical consequences. And other typical humans can assess both behaviours and consequences, and grade them as (typical) good, (typical) bad, or weird.

An AI should behave in ways that typical humans judge as good, achieving consequences that typical humans judge as good. If there is too high a risk of weirdness, the AI will output NOOP (no operation).

This seems a sensible approach. But it has a crucial flaw. The AI must behave in typical ways to achieve typical consequences - but how it achieves these consequences doesn't have to be typical.

For example, it might make an explanatory video to convince someone to follow some (reasonable) course of action. The video itself is not unusual, but the colour scheme happens to hypnotise that specific viewer into following the suggestion. At a gross level of description, everything is fine: convincing-but-typical video convinces. But this only works because the AI has inhuman levels of knowledge and predictive ability, and carefully selects the "typical behaviour" that is the most effective.

Now, we might be able to make sure the AI doesn't have secret superhuman abilities (we've had an old underdeveloped idea that we could force the AI to use human models of the world to achieve its goals), but this is a very different problem, and a much harder one.

Then, given that the AI has access to superhuman levels of ability, the "typical consequences" is no longer a safe goal. It reduces to "consequences that seem typical to an observing human", which is not a safe goal for the AI; it can now do whatever it wants, as long as its actions and consequences look ok.

Still, it might be worth developing this idea; it might work well combined with some other safety approaches.

Advanced human feedback

Rohin's argument is that we don't need the AI to solve value transfer and extrapolation. Instead, we just need a "well-motivated" AI that asks us what we prefer. It must present the options and the consequences in an informative manner. If there are morally underdefined questions where we might give multiple answers depending on question phrasing, then the AI goes to a meta level and asks us how we would want it to ask us.

This is a potentially powerful approach. It solves the value extrapolation problem indirectly, by deference to humans. It is similar to a scaled-up version of "informed consent".

Would such an approach work to contain a malevolent AI? If a superintelligent AI wished to do us harm, but was constrained to following the above feedback approach, would it be safe?

It seems clear that it wouldn't. The malevolent AI would deploy all its resourcefulness to undermine the spirit of the feedback requirement. Unless we had it perfectly defined, this would just be a speedbump ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are there alternative to solving value transfer and extrapolation?, published by Stuart Armstrong on December 6, 2021 on The AI Alignment Forum.

Thanks to Rebecca Gorman for the discussions that inspired these ideas.

We recently argued that full understanding of value extrapolation[1] was necessary and almost sufficient for solving the AI alignment problem, due to the existence of morally underdefined situations.

People seemed to generally agree that some knowledge of human morality was needed for a safe powerful AI. But it wasn't clear this AI needed almost-full knowledge of human morality, as well as value extrapolation. We got some high-quality alternative suggestions. In this post, we'll try and distil those ideas and analyse them.

AI seems to behave reasonably

Steven Byrnes's position, if we understand it correctly, is that the AI should learn to behave in non-dangerous seeming ways[2].

Our phrasing of the idea would be (apologies for any misinterpretations):

There are many behaviours that agents can follow. Typical humans follow typical behaviours, to achieve typical consequences. And other typical humans can assess both behaviours and consequences, and grade them as (typical) good, (typical) bad, or weird.

An AI should behave in ways that typical humans judge as good, achieving consequences that typical humans judge as good. If there is too high a risk of weirdness, the AI will output NOOP (no operation).

This seems a sensible approach. But it has a crucial flaw. The AI must behave in typical ways to achieve typical consequences - but how it achieves these consequences doesn't have to be typical.

For example, it might make an explanatory video to convince someone to follow some (reasonable) course of action. The video itself is not unusual, but the colour scheme happens to hypnotise that specific viewer into following the suggestion. At a gross level of description, everything is fine: convincing-but-typical video convinces. But this only works because the AI has inhuman levels of knowledge and predictive ability, and carefully selects the "typical behaviour" that is the most effective.

Now, we might be able to make sure the AI doesn't have secret superhuman abilities (we've had an old underdeveloped idea that we could force the AI to use human models of the world to achieve its goals), but this is a very different problem, and a much harder one.

Then, given that the AI has access to superhuman levels of ability, the "typical consequences" is no longer a safe goal. It reduces to "consequences that seem typical to an observing human", which is not a safe goal for the AI; it can now do whatever it wants, as long as its actions and consequences look ok.

Still, it might be worth developing this idea; it might work well combined with some other safety approaches.

Advanced human feedback

Rohin's argument is that we don't need the AI to solve value transfer and extrapolation. Instead, we just need a "well-motivated" AI that asks us what we prefer. It must present the options and the consequences in an informative manner. If there are morally underdefined questions where we might give multiple answers depending on question phrasing, then the AI goes to a meta level and asks us how we would want it to ask us.

This is a potentially powerful approach. It solves the value extrapolation problem indirectly, by deference to humans. It is similar to a scaled-up version of "informed consent".

Would such an approach work to contain a malevolent AI? If a superintelligent AI wished to do us harm, but was constrained to following the above feedback approach, would it be safe?

It seems clear that it wouldn't. The malevolent AI would deploy all its resourcefulness to undermine the spirit of the feedback requirement. Unless we had it perfectly defined, this would just be a speedbump ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing High-Impact Medicine (Hi-Med), published by High Impact Medicine on December 5, 2021 on The Effective Altruism Forum.

High Impact Medicine, Hi-Med, is a new movement supported by CEA aiming to bring together medical students and doctors into local groups to learn more about the ways medics can help as many people as possible in their careers, and engage with opportunities to have a wide-reaching positive impact within medicine.

We believe medical students and doctors have the potential to effectively tackle some of the world’s leading problems alongside, and learning from, clinical practice. Medics are a demographic of individuals who are a natural fit for EA ideas and it is a career which grants flexibility and capital to contribute meaningfully to various areas.

Our goal is to explore the core opportunities and challenges facing the world today, using an effective altruism framework, and use this as a springboard for discussion about the unique and potentially highly impactful role that medical students and doctors might play in this.

This year we are setting up local groups in Cambridge, London and Oxford (UK) to foster communities of like-minded students and doctors and explore high impact opportunities. We are planning to host several speakers and socials, both in person and online, as well as organise a fellowship in the first quarter of 2022.

We are collaborating with EA Medicine, who currently facilitate an incredible global network for EA-aligned medics to connect and network with one another and further their knowledge. We primarily hope to foster local, engaged communities of medical students and doctors who may not be as familiar with EA, or who are not currently EA aligned.

If you’re a medic interested in finding out about how you can work to effectively address the world’s most pressing problems, whether you’re based in Cambridge/ London/ Oxford or not, stay involved via social media: Facebook and Instagram.

If you’re interested in getting more involved in HI-Med, whether starting a community where you are or helping organise the new, growing movement in Cambridge, London or Oxford, fill out this form!

bit.ly/HI-Med-GetInvolved

If you have any questions or comments, please get in touch with Akhil Bansal and Abe Tolley at [email protected]

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing High-Impact Medicine (Hi-Med), published by High Impact Medicine on December 5, 2021 on The Effective Altruism Forum.

High Impact Medicine, Hi-Med, is a new movement supported by CEA aiming to bring together medical students and doctors into local groups to learn more about the ways medics can help as many people as possible in their careers, and engage with opportunities to have a wide-reaching positive impact within medicine.

We believe medical students and doctors have the potential to effectively tackle some of the world’s leading problems alongside, and learning from, clinical practice. Medics are a demographic of individuals who are a natural fit for EA ideas and it is a career which grants flexibility and capital to contribute meaningfully to various areas.

Our goal is to explore the core opportunities and challenges facing the world today, using an effective altruism framework, and use this as a springboard for discussion about the unique and potentially highly impactful role that medical students and doctors might play in this.

This year we are setting up local groups in Cambridge, London and Oxford (UK) to foster communities of like-minded students and doctors and explore high impact opportunities. We are planning to host several speakers and socials, both in person and online, as well as organise a fellowship in the first quarter of 2022.

We are collaborating with EA Medicine, who currently facilitate an incredible global network for EA-aligned medics to connect and network with one another and further their knowledge. We primarily hope to foster local, engaged communities of medical students and doctors who may not be as familiar with EA, or who are not currently EA aligned.

If you’re a medic interested in finding out about how you can work to effectively address the world’s most pressing problems, whether you’re based in Cambridge/ London/ Oxford or not, stay involved via social media: Facebook and Instagram.

If you’re interested in getting more involved in HI-Med, whether starting a community where you are or helping organise the new, growing movement in Cambridge, London or Oxford, fill out this form!

bit.ly/HI-Med-GetInvolved

If you have any questions or comments, please get in touch with Akhil Bansal and Abe Tolley at [email protected]

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Modeling Failure Modes of High-Level Machine Intelligence, published by Ben Cottier on December 6, 2021 on The AI Alignment Forum.

This post, which deals with some widely discussed failure modes of transformative AI, is part 7 in our sequence on Modeling Transformative AI Risk. In this series of posts, we are presenting a preliminary model of the relationships between key hypotheses in debates about catastrophic risks from AI. Previous posts in this sequence explained how different subtopics of this project, such as mesa-optimization and safety research agendas, are incorporated into our model.

We now come to the potential failure modes caused by High-Level Machine Intelligence (HLMI). These failure modes are the principal outputs of our model. Ultimately, we need to look at these outputs to analyze the effect that upstream parts of our model have on the risks, and in turn to guide decision-making. This post will first explain the relevant high-level components of our model before going through each component in detail. In the figure below, failure modes are represented by red-colored nodes.

We do not model catastrophe itself in detail, because there seem to be a very wide range of ways it could play out. Rather, the outcomes are states from which existential catastrophe is very likely, or "points of no return" for a range of catastrophes. Catastrophically Misaligned HLMI covers scenarios where one HLMI, or a coalition of HLMIs and possibly humans, achieves a Decisive Strategic Advantage (DSA), and the DSA enables an existential catastrophe to occur. Loss of Control covers existential scenarios where a DSA does not occur; instead HLMI systems gain influence on the world in an incremental and distributed fashion, and either humanity's control over the future gradually fades away, or a sudden change in the world causes a sufficiently large and irreversibly damaging automation failure (see What failure looks like); alternatively, the use of HLMI leads to an extreme "Moloch" scenario where most of what we value is burned down through competition.

Moving one step back, there are three major drivers of those outcomes. Firstly, Misaligned HLMI is the technical failure to align an HLMI. Second, the DSA module considers different ways DSA could be achieved by a leading HLMI project or coalition. Third, Influence-seeking Behavior considers whether an HLMI would pursue accumulation of power (e.g. political, financial) and/or manipulation of humans as an instrumental goal. This third consideration is based on whether HLMI will be agent-like and to what extent the Instrumental Convergence thesis applies.

Outcomes of HLMI Development

As discussed in the "impact of safety agendas" post, we are still in the process of understanding the theory of change behind many of the safety agendas, and similarly our attempts at translating the success/failure of these safety agendas into models of HLMI development are still only approximate. With that said, for each of the green modules shown below on the Success of [agenda] Research, we model their affect on the likelihood of an incorrigible HLMI - in particular, if the research fails to produce its intended effect. Meanwhile, the Mesa-optimisation module evaluates the likelihood of dangers from learned optimization which would tend to make HLMI incorrigible.

Besides Incorrigibility, the question of HLMI being aligned may be influenced by the expected speed and level of discontinuity during takeoff (represented by the modules Takeoff Speed and Discontinuity around HLMI without self-improvement, covered in Takeoff Speeds and Discontinuities). The expectation is that misalignment is more likely the faster progress becomes, because attempts at control will be harder if less time is available. Those factors - incorrigibility and takeoff - are more...