Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is [AN #167]: Concrete ML safety problems and their relevance to x-risk, published by Rohin Shah on October 20, 2021 on The AI Alignment Forum.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that, while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

1. Robustness: Create models that are resilient to adversaries, unusual situations, and Black Swan events.

2. Monitoring: Detect malicious use, monitor predictions, and discover unexpected model functionality.

3. Alignment: Build models that represent and safely optimize hard-to-specify human values.

4. External Safety: Use ML to address risks to how ML systems are handled, including cyberwarfare and global turbulence.

Throughout, the paper attempts to clarify the problems’ motivation and provide concrete project ideas.

Dan Hendrycks' opinion: My coauthors and I wrote this paper with the ML research community as our target audience. Here are some thoughts on this topic:

1. The document includes numerous problems that, if left unsolved, would imply that ML systems are unsafe. We need the effort of thousands of researchers to address all of them. This means that the main safety discussions cannot stay within the confines of the relatively small EA community. I think we should aim to have over one third of the ML research community work on safety problems. We need the broader community to treat AI safety at least as seriously as safety for nuclear power plants.

2. To grow the ML safety research community, we need to suggest problems that can progressively build the community and organically grow support for elevating safety standards within the existing research ecosystem. Research agendas that pertain to AGI exclusively will not scale sufficiently, and such research will simply not get enough market share in time. If we do not get the machine learning community on board with proactively mitigating risks that already exist, we will have a harder time getting them to mitigate less familiar and unprecedented risks. Rather than try to win over the community with alignment philosophy arguments, I'll try winning them over with interesting problems and try to make work towards safer systems rewarded with prestige.

3. The benefits of a larger ML safety community are numerous. They can decrease the cost of safety methods and increase the propensity to adopt them. Moreover, to ensure that ML systems have desirable properties, it is necessary to rapidly accumulate incremental improvements, but this requires substantial growth since such gains cannot be produced by just a few card-carrying x-risk researchers with the purest intentions.

4. The community will fail to grow if we ignore near-term concerns or actively exclude or sneer at people who work on problems that are useful for both near- and long-term safety (such as adversaries). The alignment community will need to stop engaging in textbook territorialism and welcome serious hypercompetent researchers who do not post on internet forums or who happen not to subscribe to effective altruism. (We include a community strategy in the Appendix.)

5. We focus on reinforcement learning but also deep learning. Most of the machine learning research community studies deep learning (e.g., text processing, vision) and does not use, say, Bellman equations or PPO. While existentially catastrophic failures will likely require competent sequential decision-making agents, the relevant problems and solutions can often be better studied outside of gridworlds and MuJoCo. There is much u...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Future Funding/Talent/Capacity Constraints Matter, Too, published by MichaelDickens on October 18, 2021 on The Effective Altruism Forum.

Cross-posted to my website.

People who talk about talent/funding/capacity constraints mostly talk about what's the biggest constraint right now. But it also matters what the constraints will be later.

Right now, the EA community holds a lot of wealth—more wealth than it can productively spend in the next few years, at least on smaller cause areas such as AI safety, cause prioritization research, and wild animal welfare. Those newer fields need time to scale up so they can absorb more funding.

That doesn't mean EAs should stop earning to give. Maybe most EAs could do more good this year with their direct efforts than with their donations. But perhaps 10 years from now, the smaller causes will have scaled up a lot, and they'll be able to deploy much more money. Earners-to-give can invest their money for a while, and then deploy it once top causes develop enough spending capacity.

Even if top cause areas don't currently have the capacity to spend more money, earning-to-give still looks valuable if three conditions all hold:

The capacity:funding ratio will greatly increase in the future.

The money spent in high-capacity organizations will still produce a lot of utility per dollar.

The discount rate isn't too much higher than the investment rate of return.

In general, I believe it should be easier to grow capacity than to grow funding. By comparison, it's not uncommon for startups to double the number of employees year over year. Charitable organizations usually can't grow like startups, but they can at least get within the same ballpark.

Over the past few years, funding has grown faster than capacity, not the other way around. So I could be wrong.

Once capacity grows, will orgs still be able to do a lot of good per dollar? They will use up the best giving opportunities first, so by the time capacity expands, the remaining opportunities won't be as good. How much difference does that make? It depends on how much it costs to solve the world's most important problems. If it cost a lot, we will still need lots of money. If it doesn't cost much (relatively speaking), then future dollars will be worth much less than dollars today.

Is the discount rate high enough that we should care substantially less about future funding? I don't think so. (What we actually care about is the discount rate minus the investment rate, because the investment rate determines how much money we will have in the future if we invest it now.) The altruistic discount rate minus the investment rate might be around 5% on the high end (it's probably less than that, and it could be negative). I would be surprised if we couldn't grow capacity much more quickly than 5% per year. So this factor is unlikely to be decisive.

My personal predictions on the three conditions:

The capacity:funding ratio will indeed greatly increase in the future. Confidence: Likely.

I don't know how quickly the value of money will decrease over time. My best guess is that all current EA funds combined are not enough to solve the world's biggest problems, so we will need as much additional funding as we can get. Confidence: Possible.

The discount rate is low enough that it doesn't affect the outcome of this question. Confidence: Highly likely.

Appendix: Some related but distinct arguments

The first argument:

Altruists have a low discount rate.

[some other premises]

Therefore, they should give later rather than now.

Therefore, earning to give is good.

(Phil Trammell presents an argument like this here)

This is a related argument in the sense that it considers the value of marginal dollars now vs. later. But the argument I presented does not require altruists to have a low discount rate.

The second argu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Future Funding/Talent/Capacity Constraints Matter, Too, published by MichaelDickens on October 18, 2021 on The Effective Altruism Forum.

Cross-posted to my website.

People who talk about talent/funding/capacity constraints mostly talk about what's the biggest constraint right now. But it also matters what the constraints will be later.

Right now, the EA community holds a lot of wealth—more wealth than it can productively spend in the next few years, at least on smaller cause areas such as AI safety, cause prioritization research, and wild animal welfare. Those newer fields need time to scale up so they can absorb more funding.

That doesn't mean EAs should stop earning to give. Maybe most EAs could do more good this year with their direct efforts than with their donations. But perhaps 10 years from now, the smaller causes will have scaled up a lot, and they'll be able to deploy much more money. Earners-to-give can invest their money for a while, and then deploy it once top causes develop enough spending capacity.

Even if top cause areas don't currently have the capacity to spend more money, earning-to-give still looks valuable if three conditions all hold:

The capacity:funding ratio will greatly increase in the future.

The money spent in high-capacity organizations will still produce a lot of utility per dollar.

The discount rate isn't too much higher than the investment rate of return.

In general, I believe it should be easier to grow capacity than to grow funding. By comparison, it's not uncommon for startups to double the number of employees year over year. Charitable organizations usually can't grow like startups, but they can at least get within the same ballpark.

Over the past few years, funding has grown faster than capacity, not the other way around. So I could be wrong.

Once capacity grows, will orgs still be able to do a lot of good per dollar? They will use up the best giving opportunities first, so by the time capacity expands, the remaining opportunities won't be as good. How much difference does that make? It depends on how much it costs to solve the world's most important problems. If it cost a lot, we will still need lots of money. If it doesn't cost much (relatively speaking), then future dollars will be worth much less than dollars today.

Is the discount rate high enough that we should care substantially less about future funding? I don't think so. (What we actually care about is the discount rate minus the investment rate, because the investment rate determines how much money we will have in the future if we invest it now.) The altruistic discount rate minus the investment rate might be around 5% on the high end (it's probably less than that, and it could be negative). I would be surprised if we couldn't grow capacity much more quickly than 5% per year. So this factor is unlikely to be decisive.

My personal predictions on the three conditions:

The capacity:funding ratio will indeed greatly increase in the future. Confidence: Likely.

I don't know how quickly the value of money will decrease over time. My best guess is that all current EA funds combined are not enough to solve the world's biggest problems, so we will need as much additional funding as we can get. Confidence: Possible.

The discount rate is low enough that it doesn't affect the outcome of this question. Confidence: Highly likely.

Appendix: Some related but distinct arguments

The first argument:

Altruists have a low discount rate.

[some other premises]

Therefore, they should give later rather than now.

Therefore, earning to give is good.

(Phil Trammell presents an argument like this here)

This is a related argument in the sense that it considers the value of marginal dollars now vs. later. But the argument I presented does not require altruists to have a low discount rate.

The second argu...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is In the shadow of the Great War, published by jasoncrawford on October 18, 2021 on LessWrong.

The idea of progress fell out of favor in the course of the 20th century. But when exactly, and why?

In a recent essay I alluded to the pivotal role of the World Wars. Here’s a quote that adds weight to this—from Progress and Power, by historian Carl Becker, published in 1936:

For two centuries the Western world has been sustained by a profound belief in the doctrine of progress. Although God the Father had withdrawn into the places where Absolute Being dwells, it was still possible to maintain that the Idea or the Dialectic or Natural Law, functioning through the conscious purposes or the unconscious activities of men, could be counted on to safeguard mankind against future hazards. However formulated, with whatever apparatus of philosophic or scientific terminology defended, the doctrine was in essence an emotional conviction, a species of religion—a religion which, according to Professor [J. B.] Bury, served as a substitute for the declining faith in the Christian doctrine of salvation .

Since 1918 this hope has perceptibly faded. Standing within the deep shadow of the Great War, it is difficult to recover the nineteenth-century faith either in the fact or the doctrine of progress. The suggestion casually thrown out some years ago by Santayana, that “civilization is perhaps approaching one of those long winters which overtake it from time to time,” seems less perverse now than when it was made. Current events lend credit to the prophets of disaster who predict the collapse of a civilization that seemed but yesterday a permanent conquest of human reason .

At the present moment the world seems indeed out of joint, and it is difficult to believe with any conviction that a power not ourselves—the Idea or the Dialectic or Natural Law—will ever set it right. The present moment, therefore, when the fact of progress is disputed and the doctrine discredited, seems to me a proper time to raise the question: What, if anything, may be said on behalf of the human race? May we still, in whatever different fashion, believe in the progress of mankind?

(Emphasis added.)

I find it fascinating to see that the downfall of the idea of progress began as early as this, after World War I. World War II perhaps simply reinforced an existing trend.

I also find fascinating Becker’s idea that humanity required some sort of safeguard, a “power not ourselves” to “set it right.”

There is no power outside of humanity. We are the masters of our fate, for better or for worse. If there is to be a 21st-century philosophy of progress, it needs to be based not on an Idea or a Dialectic, but on human agency.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is In the shadow of the Great War, published by jasoncrawford on October 18, 2021 on LessWrong.

The idea of progress fell out of favor in the course of the 20th century. But when exactly, and why?

In a recent essay I alluded to the pivotal role of the World Wars. Here’s a quote that adds weight to this—from Progress and Power, by historian Carl Becker, published in 1936:

For two centuries the Western world has been sustained by a profound belief in the doctrine of progress. Although God the Father had withdrawn into the places where Absolute Being dwells, it was still possible to maintain that the Idea or the Dialectic or Natural Law, functioning through the conscious purposes or the unconscious activities of men, could be counted on to safeguard mankind against future hazards. However formulated, with whatever apparatus of philosophic or scientific terminology defended, the doctrine was in essence an emotional conviction, a species of religion—a religion which, according to Professor [J. B.] Bury, served as a substitute for the declining faith in the Christian doctrine of salvation .

Since 1918 this hope has perceptibly faded. Standing within the deep shadow of the Great War, it is difficult to recover the nineteenth-century faith either in the fact or the doctrine of progress. The suggestion casually thrown out some years ago by Santayana, that “civilization is perhaps approaching one of those long winters which overtake it from time to time,” seems less perverse now than when it was made. Current events lend credit to the prophets of disaster who predict the collapse of a civilization that seemed but yesterday a permanent conquest of human reason .

At the present moment the world seems indeed out of joint, and it is difficult to believe with any conviction that a power not ourselves—the Idea or the Dialectic or Natural Law—will ever set it right. The present moment, therefore, when the fact of progress is disputed and the doctrine discredited, seems to me a proper time to raise the question: What, if anything, may be said on behalf of the human race? May we still, in whatever different fashion, believe in the progress of mankind?

(Emphasis added.)

I find it fascinating to see that the downfall of the idea of progress began as early as this, after World War I. World War II perhaps simply reinforced an existing trend.

I also find fascinating Becker’s idea that humanity required some sort of safeguard, a “power not ourselves” to “set it right.”

There is no power outside of humanity. We are the masters of our fate, for better or for worse. If there is to be a 21st-century philosophy of progress, it needs to be based not on an Idea or a Dialectic, but on human agency.

Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Listen to more EA content with The Nonlinear Library published by Cat Woods on October 19th on the EA Forum and LessWrong

We are excited to announce the launch of The Nonlinear Library, which allows you to easily listen to top EA content on your podcast player. We use text-to-speech software to create an automatically updating repository of audio content from the EA Forum, Alignment Forum, LessWrong, and other EA blogs.

In the rest of this post, we’ll explain our reasoning for the audio library, why it’s useful, why it’s potentially high impact, its limitations, and our plans. You can read it here or listen to the post in podcast form here.

Goal: increase the number of people who read EA research

An EA koan: if your research is high quality, but nobody reads it, does it have an impact?

Generally speaking, the theory of change of research is that you investigate an area, come to better conclusions, people read those conclusions, they make better decisions, all ultimately leading to a better world. So the answer is no. Barring some edge cases (1), if nobody reads your research, you usually won’t have any impact.

Research → Better conclusion → People learn about conclusion → People make better decisions → The world is better

Nonlinear is working on the third step of this pipeline: increasing the number of people engaging with the research. By increasing the total number of EA articles read, we’re increasing the impact of all of that content.

This is often relatively neglected because researchers typically prefer doing more research instead of promoting their existing output. Some EAs seem to think that if their article was promoted one time, in one location, such as the EA Forum, then surely most of the community saw it and read it. In reality, it is rare that more than a small percentage of the community will read even the top posts. This is an expected-value tragedy, when a researcher puts hundreds of hours of work into an important report which only a handful of people read, dramatically reducing its potential impact.

Here are some purely hypothetical numbers just to illustrate this way of thinking:

Imagine that you, a researcher, have spent 100 hours producing outstanding research that is relevant to 1,000 out of a total of 10,000 EAs.

Each relevant EA who reads your research will generate $1,000 of positive impact. So, if all 1,000 relevant EAs read your research, you will generate $1 million of impact.

You post it to the EA Forum, where posts receive 500 views on average. Let’s say, because your report is long, only 20% read the whole thing - that’s 100 readers. So you’ve created 1001,000 = $100,000 of impact. Since you spent 100 hours and created $100,000 of impact, that’s $1,000 per hour - pretty good!

But if you were to spend, say 1 hour, promoting your report - for example, by posting links on EA-related Facebook groups - to generate another 100 readers, that would produce another $100,000 of impact. That’s $100,000 per marginal hour or ~$2,000 per hour taking into account the fixed cost of doing the original research.

Likewise, if another 100 EAs were to listen to your report while commuting, that would generate an incremental $100,000 of impact - at virtually no cost, since it’s fully automated.

In this illustrative example, you’ve nearly tripled your cost-effectiveness and impact with one extra hour spent sharing your findings and having a public system that turns it into audio for you.

Another way the audio library is high expected value is that instead of acting as a multiplier on just one researcher or one organization, it acts as a multiplier on nearly the entire output of the EA research community. This allows for two benefits: long-tail capture and the power of large numbers and multipliers.

Long-tail capture. The value of research is extremely long ta...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.

This is Listen to more EA content with The Nonlinear Library published by Cat Woods on October 19th on the EA Forum and LessWrong

We are excited to announce the launch of The Nonlinear Library, which allows you to easily listen to top EA content on your podcast player. We use text-to-speech software to create an automatically updating repository of audio content from the EA Forum, Alignment Forum, LessWrong, and other EA blogs.

In the rest of this post, we’ll explain our reasoning for the audio library, why it’s useful, why it’s potentially high impact, its limitations, and our plans. You can read it here or listen to the post in podcast form here.

Goal: increase the number of people who read EA research

An EA koan: if your research is high quality, but nobody reads it, does it have an impact?

Generally speaking, the theory of change of research is that you investigate an area, come to better conclusions, people read those conclusions, they make better decisions, all ultimately leading to a better world. So the answer is no. Barring some edge cases (1), if nobody reads your research, you usually won’t have any impact.

Research → Better conclusion → People learn about conclusion → People make better decisions → The world is better

Nonlinear is working on the third step of this pipeline: increasing the number of people engaging with the research. By increasing the total number of EA articles read, we’re increasing the impact of all of that content.

This is often relatively neglected because researchers typically prefer doing more research instead of promoting their existing output. Some EAs seem to think that if their article was promoted one time, in one location, such as the EA Forum, then surely most of the community saw it and read it. In reality, it is rare that more than a small percentage of the community will read even the top posts. This is an expected-value tragedy, when a researcher puts hundreds of hours of work into an important report which only a handful of people read, dramatically reducing its potential impact.

Here are some purely hypothetical numbers just to illustrate this way of thinking:

Imagine that you, a researcher, have spent 100 hours producing outstanding research that is relevant to 1,000 out of a total of 10,000 EAs.

Each relevant EA who reads your research will generate $1,000 of positive impact. So, if all 1,000 relevant EAs read your research, you will generate $1 million of impact.

You post it to the EA Forum, where posts receive 500 views on average. Let’s say, because your report is long, only 20% read the whole thing - that’s 100 readers. So you’ve created 1001,000 = $100,000 of impact. Since you spent 100 hours and created $100,000 of impact, that’s $1,000 per hour - pretty good!

But if you were to spend, say 1 hour, promoting your report - for example, by posting links on EA-related Facebook groups - to generate another 100 readers, that would produce another $100,000 of impact. That’s $100,000 per marginal hour or ~$2,000 per hour taking into account the fixed cost of doing the original research.

Likewise, if another 100 EAs were to listen to your report while commuting, that would generate an incremental $100,000 of impact - at virtually no cost, since it’s fully automated.

In this illustrative example, you’ve nearly tripled your cost-effectiveness and impact with one extra hour spent sharing your findings and having a public system that turns it into audio for you.

Another way the audio library is high expected value is that instead of acting as a multiplier on just one researcher or one organization, it acts as a multiplier on nearly the entire output of the EA research community. This allows for two benefits: long-tail capture and the power of large numbers and multipliers.

Long-tail capture. The value of research is extremely long ta...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Introducing High Impact Professionals, published by DevonFritz on October 19, 2021 on The Effective Altruism Forum.

Overview

It is our pleasure to present to you our new EA organization, High Impact Professionals (HIP). Our vision is to enable EA working professionals to have the biggest positive impact possible. We imagine a community working together for the greater good, contributing not only through their money but also with their time and skills. We recently went through the 2021 Charity Entrepreneurship Incubation Program and received $100,000 in funding for our first full year of operations.

The rest of this document will outline:

The problem we are solving

How we are solving it

Our next steps

How you can get involved

Key Takeaways

We created HIP to maximize the impact of EA working professionals and solve several identified problems within the EA ecosystem.

We are going to pilot several promising interventions in year one to identify the most impactful path forward. We list them in the Pilots section of the post.

We have many ways you can get involved.

Our first pilot is workplace fundraising events. We have successfully run these in the past and are now helping EA working professionals (HIPs) host them at their organizations. If you are an EA working professional and want to increase your impact, become more engaged, and gain useful skills and EA career capital, register your interest to host a fundraising event. We will provide you with the resources and support needed to host a successful event.

We want to talk to more HIPs to get a better understanding of the needs of our community. Through our own giving of time and money, we have a sense of what might be needed but want to validate this with others in the community. If you are an EA working professional looking to have more impact please register your interest here. We encourage you to have a low bar for doing this: if you are considering it, just do it!

Please feel free to share with others you know who might be a good fit.

Thanks to Cillian Crosson, Ula Zarosa, Aaron Gertler, Jack Lewars, Jan-Willem Van Putten, and Brian Tan for their invaluable feedback on this announcement. All errors are our own.

Problem

Impact

According to the most recent EA Surveys, over half of all EAs are in the private sector and earning to give is the most common career choice amongst EAs with more than a third of votes cast. This group has resources like skills, time and money that they can contribute to high-impact organizations and through surveys have expressed a desire to do so. For example, the same survey revealed that a full 55% of EAs want to donate more, and more than one-third want to volunteer their time as a way to become more engaged. This means that the largest cohort of EAs is limited in their ultimate impact and seems to be facing a bottleneck.

EA Community Issues

In addition to impact problems, there have also been issues within the EA community that our organization can likely mitigate:

Funding diversification - a recent survey of 29 meta organizations revealed that a majority of those polled have a strong desire for more funding diversification for their organization.

Supply and demand issue with EA jobs - many EAs want to work for EA organizations but there are few positions available. This creates a lot of frustration for the many EAs that apply for an EA job but don’t get one. A recent forum post implies this could be a range of 47-124 people being rejected per role.

Insularity in EA - EA is quite insular and would likely benefit from stronger connections to the private sector, both to spread the ideas of EA more widely and to establish a more solid reputation for EA in wider circles.

Evidence

The rigorous research and selection process performed by Charity Entrepreneurship identified this idea as one of...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is Introducing High Impact Professionals, published by DevonFritz on October 19, 2021 on The Effective Altruism Forum.

Overview

It is our pleasure to present to you our new EA organization, High Impact Professionals (HIP). Our vision is to enable EA working professionals to have the biggest positive impact possible. We imagine a community working together for the greater good, contributing not only through their money but also with their time and skills. We recently went through the 2021 Charity Entrepreneurship Incubation Program and received $100,000 in funding for our first full year of operations.

The rest of this document will outline:

The problem we are solving

How we are solving it

Our next steps

How you can get involved

Key Takeaways

We created HIP to maximize the impact of EA working professionals and solve several identified problems within the EA ecosystem.

We are going to pilot several promising interventions in year one to identify the most impactful path forward. We list them in the Pilots section of the post.

We have many ways you can get involved.

Our first pilot is workplace fundraising events. We have successfully run these in the past and are now helping EA working professionals (HIPs) host them at their organizations. If you are an EA working professional and want to increase your impact, become more engaged, and gain useful skills and EA career capital, register your interest to host a fundraising event. We will provide you with the resources and support needed to host a successful event.

We want to talk to more HIPs to get a better understanding of the needs of our community. Through our own giving of time and money, we have a sense of what might be needed but want to validate this with others in the community. If you are an EA working professional looking to have more impact please register your interest here. We encourage you to have a low bar for doing this: if you are considering it, just do it!

Please feel free to share with others you know who might be a good fit.

Thanks to Cillian Crosson, Ula Zarosa, Aaron Gertler, Jack Lewars, Jan-Willem Van Putten, and Brian Tan for their invaluable feedback on this announcement. All errors are our own.

Problem

Impact

According to the most recent EA Surveys, over half of all EAs are in the private sector and earning to give is the most common career choice amongst EAs with more than a third of votes cast. This group has resources like skills, time and money that they can contribute to high-impact organizations and through surveys have expressed a desire to do so. For example, the same survey revealed that a full 55% of EAs want to donate more, and more than one-third want to volunteer their time as a way to become more engaged. This means that the largest cohort of EAs is limited in their ultimate impact and seems to be facing a bottleneck.

EA Community Issues

In addition to impact problems, there have also been issues within the EA community that our organization can likely mitigate:

Funding diversification - a recent survey of 29 meta organizations revealed that a majority of those polled have a strong desire for more funding diversification for their organization.

Supply and demand issue with EA jobs - many EAs want to work for EA organizations but there are few positions available. This creates a lot of frustration for the many EAs that apply for an EA job but don’t get one. A recent forum post implies this could be a range of 47-124 people being rejected per role.

Insularity in EA - EA is quite insular and would likely benefit from stronger connections to the private sector, both to spread the ideas of EA more widely and to establish a more solid reputation for EA in wider circles.

Evidence

The rigorous research and selection process performed by Charity Entrepreneurship identified this idea as one of...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prioritization Research for Advancing Wisdom and Intelligence, published by ozziegooen on October 18, 2021 on LessWrong.

LessWrong note: I wrote this more in a way slightly more optimized for the EA Forum than LessWrong, because the post seemed slightly more appropriate there.

Summary

I think it makes sense for Effective Altruists to pursue prioritization research to figure out how best to improve the wisdom and intelligence[1] of humanity. I describe endeavors that would optimize for longtermism, though similar research efforts could make sense for other worldviews.

The Basic Argument

For those interested in increasing humanity’s long-term wisdom and intelligence[1], several types of wildly different interventions are options on the table. For example, we could improve at teaching rationality, or we could make progress on online education. We could make forecasting systems and data platforms. We might even consider something more radical, like brain-computer interfaces or highly advanced pre-AGI AI systems.

These interventions share many of the same benefits. If we figure out ways to remove people’s cognitive biases, causing them to make better political decisions, that would be similar to the impact of forecasting systems on their political decisions. It seems natural to attempt to figure out how to compare these. We wouldn’t want to invest a lot of resources into one field, to realize 10 years later that we could have spent them better in another. This prioritization is pressing because Effective Altruists are currently scaling up work in several relevant areas (rationality, forecasting, institutional decision making) but mostly ignoring others (brain-computer interfaces, fundamental internet improvements).

In addition to caring about prioritization between cause areas, we should also care about estimating the importance of wisdom and intelligence work as a whole. Estimating the importance of wisdom and intelligence gains is crucial for multiple interventions, so it doesn’t make much sense to ask each intervention’s research base to independently tackle this question on their own. Previously I’ve done a lot of thinking about this as part of my work to estimate the value of my own work on forecasting. It felt a bit silly to have to answer this bigger question about wisdom and intelligence, like the bigger question was far outside actual forecasting research.

I think we should consider doing serious prioritization research around wisdom and intelligence for longtermist reasons.[2] This work could both inform us of the cost-effectiveness of all of the available options as a whole, and help us compare directly between different options.

Strong prioritization research between different interventions around wisdom and intelligence might at first seem daunting. There are so clearly many uncertainties and required judgment calls. We don’t even have any good ways of measuring wisdom and intelligence at this point.

However, I think the Effective Altruist and Rationalist communities would prove up to the challenge. GiveWell’s early work drew skepticism for similar reasons. It took a long time for Quality-Adjusted Life Years to be accepted and adopted, but there’s since been a lot of innovative and educational progress. Now our communities have the experience of hundreds of research person-years of prioritization work. We have at least a dozen domain-specific prioritization projects[3]. Maybe prioritization work in wisdom and intelligence isn’t far off.

List of Potential Interventions

I brainstormed an early list of potential interventions with examples of existing work. I think all of these could be viable candidates for substantial investment.

Human/organizational

Rationality-related research, marketing, and community building (CFAR, Astral Codex Ten, LessWrong, Julia Galef...