Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Retrospective on the Summer 2021 AGI Safety Fundamentals, published by dewierwan on December 6, 2021 on The Effective Altruism Forum.

Thanks to Jamie Bernardi, Caleb Parikh, Neel Nanda, Isaac Dunn, Michael Chen and JJ Balisanyuka-Smith for feedback on drafts of this document. All mistakes are my own.

How to read this document

If you're short on time, I recommend reading the summary, background and impact assessment (in descending order of importance).

The rest of the document is most useful for individuals who are interested in running similar programmes, either for AI safety specifically, or for other cause areas / meta EA, or for people who are interested in participating or facilitating in the next round.

Summary

Overview

Over the Summer of 2021, EA Cambridge, alongside many collaborators, ran a virtual seminar programme on AGI safety.

It had 168 participants and 28 facilitators (from around the world - the majority were not based in the UK), and provided an overview of the AI safety (AIS) landscape and a space for participants to reflect upon the readings with others and experienced facilitators.

It consisted of an introduction to machine learning (ML); 7 weeks of 1.5-hour cohort discussions in groups of ~5 with a facilitator, with ~2-hours of reading on AIS per week; followed by an optional 4-week self-directed capstone project, finishing with a capstone project discussion.

This document provides some more background on the programme, feedback from participants and facilitators, mistakes we made, and some things we’ll be doing to improve the programme for the next iteration (taking place in early 2022).

I believe the main source of value this programme provided was:

Accountability and encouragement for AI safety-minded individuals to read through an excellently-curated curriculum on the topic.

Relationships and networks created via the cohort discussions, socials and 1-1s.

Accelerated people’s AIS careers by providing them with an outline of the space and where opportunities lie.

Increased the number of people in the EA community who have a broad understanding of AI risks, potentially contributing to improving individuals’ and the community’s cause prioritisation.

Information value on running a large cause-area-specific seminar programmes.

Further engagement

If you’d like to participate in the third round of the programme (Jan-March 2022), apply here.

There are two tracks: a technical AI alignment track for people from a maths/compsci background who are considering technical AI safety research, and an AI governance track for individuals considering a career in policy/governance.

If you already have a broad background in ML and AI technical safety or AI governance, please consider applying to facilitate here.

This is compensated with an optional £800 stipend.

Facilitators have the option to accept this stipend or donate it directly to EA Cam or EA Funds.

We want to ensure all individuals with the relevant expertise and knowledge have the opportunity to contribute to this programme, regardless of their financial situation.

We’re likely to be bottlenecked by facilitators with relevant expertise, so we would strongly appreciate your application!

If you’d like to run your own version of this programme at your local group/region, please fill in this form and we’ll send you our resources, how-to guides, and offer you a 1-1 and other forms of support.

Note that we will also be helping to organise local in-person cohorts via this global programme.

Why are we writing this post?

To be transparent about our activities and how we evaluate them as an EA meta org

To provide anyone who wants to learn more about this specific AI safety field-building effort an insight into how it came about, what took place, and how it went, as well as to gather feedback and suggestions from the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: Joan Rohlfing, President and COO of the Nuclear Threat Initiative, published by Joan Rohlfing on December 6, 2021 on The Effective Altruism Forum.

Hi everyone — I’m Joan Rohlfing, president and COO of the Nuclear Threat Initiative. We’re a nonprofit, nonpartisan global security organization focused on reducing nuclear and biological threats imperiling humanity. In an era when the likelihood of use of weapons of mass destruction by individuals, terrorist organizations, and states is growing, we work to drive systemic change by galvanizing large-scale institutional adoption of innovative global security practices and programs. We’ve been doing it successfully for 20 years now.

What threats are most urgent? What can we do to stop them? I'm ready to answer those questions and more on Thursday, December 9, from 11 a.m. to 2 p.m. EST. (You can ask me something during that window or anytime before.)

This AMA is a follow-on to a talk I gave about nuclear threats in October, at the Effective Altruism Global 2021 conference in London. I discussed my strong belief that it is possible to make a difference in this arena, and I described what I believe we can do — and must do — together to build a safer world. I look forward to continuing that conversation this week!

A little more about me: Before joining NTI, I held senior positions in the U.S. Department of Energy and worked as an advisor to the U.S. Ambassador to India in the wake of nuclear tests in India and Pakistan. Earlier in my career, I oversaw nuclear weapons policy and acquisition programs at the Department of Defense and the Armed Services Committee of the U.S. House of Representatives. My thinking on nuclear security has been shaped by 35 years working in this field, and I’ve worked on this from all angles — from a military, a diplomatic, a political, a technical, and an academic perspective.

I look forward to your questions — ask me anything!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AMA: Joan Rohlfing, President and COO of the Nuclear Threat Initiative, published by Joan Rohlfing on December 6, 2021 on The Effective Altruism Forum.

Hi everyone — I’m Joan Rohlfing, president and COO of the Nuclear Threat Initiative. We’re a nonprofit, nonpartisan global security organization focused on reducing nuclear and biological threats imperiling humanity. In an era when the likelihood of use of weapons of mass destruction by individuals, terrorist organizations, and states is growing, we work to drive systemic change by galvanizing large-scale institutional adoption of innovative global security practices and programs. We’ve been doing it successfully for 20 years now.

What threats are most urgent? What can we do to stop them? I'm ready to answer those questions and more on Thursday, December 9, from 11 a.m. to 2 p.m. EST. (You can ask me something during that window or anytime before.)

This AMA is a follow-on to a talk I gave about nuclear threats in October, at the Effective Altruism Global 2021 conference in London. I discussed my strong belief that it is possible to make a difference in this arena, and I described what I believe we can do — and must do — together to build a safer world. I look forward to continuing that conversation this week!

A little more about me: Before joining NTI, I held senior positions in the U.S. Department of Energy and worked as an advisor to the U.S. Ambassador to India in the wake of nuclear tests in India and Pakistan. Earlier in my career, I oversaw nuclear weapons policy and acquisition programs at the Department of Defense and the Armed Services Committee of the U.S. House of Representatives. My thinking on nuclear security has been shaped by 35 years working in this field, and I’ve worked on this from all angles — from a military, a diplomatic, a political, a technical, and an academic perspective.

I look forward to your questions — ask me anything!

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Considerations on interaction between AI and expected value of the future, published by Beth Barnes on December 7, 2021 on The AI Alignment Forum.

Some thoughts about the ‘default’ trajectory of civilisation and how AI will affect the likelihood of different outcomes.

Is the default non-extinction outcome utopic or dystopic?

Arguments for dystopia:

The world seems dystopic in many ways now

We’re getting better at various types of manipulative tactics, i.e. persuasion, marketing, creation of addiction and dependence, harnessing of tribalism. These things cause people’s actions to depart from what would best achieve their ‘true’ values. When this happens a lot, there is no reason the world should move in a direction that people think is broadly good. The things that happen in the world will no longer reflect human values; they will reflect the results of competition for resources and influence taking place between non-human actors (i.e. corporations, political parties), who are able to effectively control human actions.

Most creatures have existed in a dystopic Malthusian struggle, where the size of the population is kept stable by large numbers dying due to lack of resources. This is the state for all wild animals, and was the state for almost all of human history. We should probably consider this the default state. We should view the current state, where there are surplus resources for people to live much more comfortably than ‘barely not dying’ and the population is controlled by our reproductive choices rather than starvation or conflict, as an exception that will probably revert to normal, rather than permanent state.

Altruistic or broadly benevolent values are not the default, they’re pretty recent and rare. The fraction of human history where our moral circles have been wide enough to include humans of other races or nationalities is extremely small. Most current people’s moral circles only patchily include animals- it depends on the exact species and circumstance whether people feel any empathy. It seems like in many traditional societies, killing or harming a member of a rival group was considered not just neutral but good, something virtuous and something to be celebrated.

Technology is making the circumstances of our lives increasingly distant from the sort of settings humans are adapted for. We’ll find ourselves in settings where the environment is too unnatural to successfully trigger happiness, fulfilment, or empathy.

There are positional goods and values that involve people benefiting from the misfortune of others, or from authority and dominion over them. Current people desire various positional goods like relative social position, relative wealth, relative dating success, e.t.c. Historically, values systems which directly value the submission or suffering of others seem common. It seems to have been pretty standard, for most of human history, to have your value system include how many people you’ve conquered or dominated, and a key way to prove (to others or for your own satisfaction) that you’ve successfully conquered someone is to force them to do something they don’t want to do

Selection favours certain kinds of values that are not the values we’d want in a utopia - those with values of dominance, conquering and proliferation will survive and spread more effectively than those with values of pacifism, altruism and benevolence

Arguments for utopia:

All else being equal, most (current, westernish?) people would prefer the world to be generally good for (at least most of) the sentient beings in it. People will generally try to shape the world in a better direction, as long as it’s not too costly to themselves to do this. Technological progress will make it increasingly cheap and easy to ensure all sentient beings have good lives

Life for most people ha...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Considerations on interaction between AI and expected value of the future, published by Beth Barnes on December 7, 2021 on The AI Alignment Forum.

Some thoughts about the ‘default’ trajectory of civilisation and how AI will affect the likelihood of different outcomes.

Is the default non-extinction outcome utopic or dystopic?

Arguments for dystopia:

The world seems dystopic in many ways now

We’re getting better at various types of manipulative tactics, i.e. persuasion, marketing, creation of addiction and dependence, harnessing of tribalism. These things cause people’s actions to depart from what would best achieve their ‘true’ values. When this happens a lot, there is no reason the world should move in a direction that people think is broadly good. The things that happen in the world will no longer reflect human values; they will reflect the results of competition for resources and influence taking place between non-human actors (i.e. corporations, political parties), who are able to effectively control human actions.

Most creatures have existed in a dystopic Malthusian struggle, where the size of the population is kept stable by large numbers dying due to lack of resources. This is the state for all wild animals, and was the state for almost all of human history. We should probably consider this the default state. We should view the current state, where there are surplus resources for people to live much more comfortably than ‘barely not dying’ and the population is controlled by our reproductive choices rather than starvation or conflict, as an exception that will probably revert to normal, rather than permanent state.

Altruistic or broadly benevolent values are not the default, they’re pretty recent and rare. The fraction of human history where our moral circles have been wide enough to include humans of other races or nationalities is extremely small. Most current people’s moral circles only patchily include animals- it depends on the exact species and circumstance whether people feel any empathy. It seems like in many traditional societies, killing or harming a member of a rival group was considered not just neutral but good, something virtuous and something to be celebrated.

Technology is making the circumstances of our lives increasingly distant from the sort of settings humans are adapted for. We’ll find ourselves in settings where the environment is too unnatural to successfully trigger happiness, fulfilment, or empathy.

There are positional goods and values that involve people benefiting from the misfortune of others, or from authority and dominion over them. Current people desire various positional goods like relative social position, relative wealth, relative dating success, e.t.c. Historically, values systems which directly value the submission or suffering of others seem common. It seems to have been pretty standard, for most of human history, to have your value system include how many people you’ve conquered or dominated, and a key way to prove (to others or for your own satisfaction) that you’ve successfully conquered someone is to force them to do something they don’t want to do

Selection favours certain kinds of values that are not the values we’d want in a utopia - those with values of dominance, conquering and proliferation will survive and spread more effectively than those with values of pacifism, altruism and benevolence

Arguments for utopia:

All else being equal, most (current, westernish?) people would prefer the world to be generally good for (at least most of) the sentient beings in it. People will generally try to shape the world in a better direction, as long as it’s not too costly to themselves to do this. Technological progress will make it increasingly cheap and easy to ensure all sentient beings have good lives

Life for most people ha...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Potential EA NYC Coworking Space, published by Rockwell Schwartz on December 6, 2021 on The Effective Altruism Forum.

I am one of the co-directors of EA NYC. We are in the very initial stages of exploring the possibility of setting up a coworking space in NYC for people focused on EA-aligned work.

We have received strong initial support for this idea. At this time, this is very early-stage and may not come to fruition or may be limited in size, meaning we do not want to get anyone's hopes up at this juncture. If you are potentially interested in utilizing an EA coworking space in NYC, please fill out this survey by December 15.

We welcome any feedback. We are particularly interested in thoughts on (1) what the best version of this could look like, (2) how to make this financially feasible and accessible, and (3) risks we may not be properly anticipating.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Potential EA NYC Coworking Space, published by Rockwell Schwartz on December 6, 2021 on The Effective Altruism Forum.

I am one of the co-directors of EA NYC. We are in the very initial stages of exploring the possibility of setting up a coworking space in NYC for people focused on EA-aligned work.

We have received strong initial support for this idea. At this time, this is very early-stage and may not come to fruition or may be limited in size, meaning we do not want to get anyone's hopes up at this juncture. If you are potentially interested in utilizing an EA coworking space in NYC, please fill out this survey by December 15.

We welcome any feedback. We are particularly interested in thoughts on (1) what the best version of this could look like, (2) how to make this financially feasible and accessible, and (3) risks we may not be properly anticipating.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective Slacktivism: why somebody should do prioritization research on slacktivism, published by Kat Woods on December 3, 2021 on The Effective Altruism Forum.

Some slacktivism is probably way more effective than other slacktivism, so somebody should do some prioritization research to find the best slacktivism techniques. (1)

For example, I almost always like EA job ads on Facebook and Twitter because if I do, this leads to more people seeing the ad, which leads to EA orgs getting more and better applicants, which leads to more impact.

This seems like an exceptionally good return on investment per effort (which is essentially zero), so more people should be doing it, which is why I’m sharing it here.

However, this is just a really quick conclusion with very little thought put into it. For example, how often should you like job ads? Is it better to do it for all of them or have a more complex strategy? Is it different on different social media platforms?

What about totally different actions? For example, are there ways to help with organization’s fundraisers? What about sharing certain activities, like going vegan or taking the Giving What We Can pledge? How about using profile frames?

I’d love to see a writeup of a more in-depth analysis about the different ways to do “effective slacktivism”, or links to existing work in the area. I bet there is a lot of low-hanging fruit here and if somebody took the time to do some prioritization work on it, that could end up increasing the impact of the community for a long time.

This could be an especially good research project for somebody who’s considering getting into prioritization research work. It could give a low-risk, low-cost way to get a taste for the work while also providing a tangible, immediate benefit to the world.

As such, we are putting out a bounty of $100 USD for anybody who writes a post doing prioritization research on slacktivism that gets over 30 upvotes, subject to our discretion.

What are your ideas for small actions that could be high ROI?

Footnotes

(1) I would like to nominate this sentence for the Most EA Opening Line contest. I would also like somebody to make that contest because I would find it amazing.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Effective Slacktivism: why somebody should do prioritization research on slacktivism, published by Kat Woods on December 3, 2021 on The Effective Altruism Forum.

Some slacktivism is probably way more effective than other slacktivism, so somebody should do some prioritization research to find the best slacktivism techniques. (1)

For example, I almost always like EA job ads on Facebook and Twitter because if I do, this leads to more people seeing the ad, which leads to EA orgs getting more and better applicants, which leads to more impact.

This seems like an exceptionally good return on investment per effort (which is essentially zero), so more people should be doing it, which is why I’m sharing it here.

However, this is just a really quick conclusion with very little thought put into it. For example, how often should you like job ads? Is it better to do it for all of them or have a more complex strategy? Is it different on different social media platforms?

What about totally different actions? For example, are there ways to help with organization’s fundraisers? What about sharing certain activities, like going vegan or taking the Giving What We Can pledge? How about using profile frames?

I’d love to see a writeup of a more in-depth analysis about the different ways to do “effective slacktivism”, or links to existing work in the area. I bet there is a lot of low-hanging fruit here and if somebody took the time to do some prioritization work on it, that could end up increasing the impact of the community for a long time.

This could be an especially good research project for somebody who’s considering getting into prioritization research work. It could give a low-risk, low-cost way to get a taste for the work while also providing a tangible, immediate benefit to the world.

As such, we are putting out a bounty of $100 USD for anybody who writes a post doing prioritization research on slacktivism that gets over 30 upvotes, subject to our discretion.

What are your ideas for small actions that could be high ROI?

Footnotes

(1) I would like to nominate this sentence for the Most EA Opening Line contest. I would also like somebody to make that contest because I would find it amazing.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Declustering, reclustering, and filling in thingspace, published by Stuart Armstrong on December 6, 2021 on The AI Alignment Forum.

Eliezer has talked about thingspace - an abstract space in which objects lie, defined by their properties on some scales. In thingspace, there are some more-or-less natural categories, such as the "bird" category, that correspond to clusters in this thingspace. Drawing the boundary/carving reality at the joints, can be thought of as using definitions that efficiently separate these clusters[1].

He said about the thingspace of birds:

The central clusters of robins and sparrows glowing brightly with highly typical birdness; satellite clusters of ostriches and penguins [further from the center] glowing more dimly with atypical birdness, and Abraham Lincoln a few megaparsecs away and glowing not at all.

That, however, is how things stand in the world as it is. What if, for some reason, we had a lot of power in the world, and we wanted to break these neat-ish clusters, how would we go about it?

Remaking the clusters

Suppose first we wanted to break away some of those satellites away. What if we wanted penguins to be clearly distinct from birds? Penguins are land-waddlers and good swimmers, living in cold climates. So let's fill out this category, call it, maybe, Panguens.

Panguens all live in cold climates, waddle on land, and are good swimmers. Most are mammals, and give birth to live young. Some are more like the duck-billed platypus, and lay eggs (though they suckle their young). And all of them, including penguins, can reproduce with each other (how? genetic engineering magic). Thus there are a lot of intermediate species, including some that are almost-penguins except differing on a few details of their anatomy. All have fins of some sort, but only 10% have penguin-like wing-fins.

Panguens are clearly a cluster, quite a tight one as animals go (given they can reproduce with each other). And they are clearly not birds. And penguins clearly belong inside them, rather than with birds; at most, some pedantic classifier would label penguins as "bird-like panguens".

So, by filling in extra possibilities in thingspace, we have moved penguins to a new category. Graphically, it looks like starting with:

and moving to:

What if we wanted to make Abraham Lincoln into a bird[2]? We can connect the categories of birds and humans by creating all sort of bird-human hybrids. But this is not quite enough. Bird-humans is certainly a reasonable category in this world. But humans are very close to other mammals, especially great apes; Lincoln is clearly a bird-human, but also clearly a mammal. How can we excise him from this category?

Well, we might just go around killing all the apes and monkeys, making the bird-human category more natural. But we don't need to do that. We could introduce instead mammal-dog hybrids: all mammal species (except humans) have a smooth transition to lizards, filled with intermediate species. Now the new natural definition of "mammal" includes "must have a hybrid version with lizards". All mammals fit this definition - except for humans, who have their own thing with birds. Much more natural to divide the "bird-humans" cleanly from the "mammal-lizards", and note that Lincoln is a bird-human that shares some features with some mammal-lizards.

The more we want to separate humans from mammals, the more we increase the category "mammals" in directions away from humans, and increase the category "bird-humans" in directions away from other mammals. Graphically we start with:

and move to:

Why we should care: AIs, examples, and definitions

This is an amusing mental exercise, but what's the point? The point is that this is another variant of the standard failure mode of powerful AIs.

Let's go with that old chestnut, and assume that we hav...