Published on June 2, 2021 5:25 PM GMT

EA funders have funded various organisations working on biosecurity and pandemic preparedness, including:

John Hopkins Center for Health Security

Georgetown Center for Global Health Science and Security

Center for International Security and Cooperation Biosecurity Initiative

Nuclear Threat Initiative

Blue Ribbon Study Panel on Biodefense

It seems to be widely accepted that many mainstream institutions got important things about COVID wrong, such as masks, travel bans, and lockdowns. Have there been any reviews of how these and other EA-funded things performed on COVID-related matters, with the benefit of hindsight?

Published on June 2, 2021 5:25 PM GMT

EA funders have funded various organisations working on biosecurity and pandemic preparedness, including:

John Hopkins Center for Health Security

Georgetown Center for Global Health Science and Security

Center for International Security and Cooperation Biosecurity Initiative

Nuclear Threat Initiative

Blue Ribbon Study Panel on Biodefense

It seems to be widely accepted that many mainstream institutions got important things about COVID wrong, such as masks, travel bans, and lockdowns. Have there been any reviews of how these and other EA-funded things performed on COVID-related matters, with the benefit of hindsight?

Published on June 2, 2021 6:47 PM GMT

I'm trying to get to the crux of the differences between the progress studies (PS) and the EA / existential risk (XR) communities. I'd love input from you all on my questions below.

The road trip metaphor

Let me set up a metaphor to frame the issue:

Picture all of humanity in a car, traveling down the highway of progress. Both PS and EA/XR agree that the trip is good, and that as long as we don't crash, faster would be better. But:

XR thinks that the car is out of control and that we need a better grip on the steering wheel. We should not accelerate until we can steer better, and maybe we should even slow down in order to avoid crashing.

PS thinks we're already slowing down, and so wants to put significant attention into re-accelerating. Sure, we probably need better steering too, but that's secondary.

(See also @Max_Daniel's recent post)

My questions

Here are some things I don't really understand about the XR position (granted that I haven't read the literature on it extensively yet, but I have read a number of the foundational papers).

(Edit for clarity: these questions are not proposed as cruxes. They are just questions I am unclear on, related to my attempt to find the crux)

How does XR weigh costs and benefits?

Is there any cost that is too high to pay, for any level of XR reduction? Are they willing to significantly increase global catastrophic risk—one notch down from XR in Bostrom's hierarchy—in order to decrease XR? I do get that impression. They seem to talk about any catastrophe less than full human extinction as, well, not that big a deal.

For instance, suppose that if we accelerate progress, we can end poverty (by whatever standard) one century earlier than otherwise. In that case, failing to do so, in itself, should be considered a global catastrophic risk, or close to it. If you're willing to accept GCR in order to slightly reduce XR, then OK—but it feels to me that you've fallen for a Pascal's Mugging.

Eliezer has specifically said that he doesn't accept Pascal's Mugging arguments in the x-risk context, and Holden Karnofsky has indicated the same. The only counterarguments I've seen conclude “so AI safety (or other specific x-risk) is still a worthy cause”—which I'm fine with. I don't see how you get to “so we shouldn't try to speed up technological progress.”

Does XR consider tech progress default-good or default-bad?

My take is that tech progress is default good, but we should be watchful for bad consequences and address specific risks. I think it makes sense to pursue specific projects that might increase AI safety, gene safety, etc. I even think there are times when it makes sense to put a short-term moratorium on progress in an area in order to work out some safety issues—this has been done once or twice already in gene safety.

When I talk to XR folks, I sometimes get the impression that they want to flip it around, and consider all tech progress to be bad unless we can make an XR-based case that it should go forward. That takes me back to point (1).

What would moral/social progress actually look like?

This idea that it's more important to make progress in non-tech areas: epistemics, morality, coordination, insight, governance, whatever. I actually sort of agree with that, but I'm not sure at all that what I have in mind there corresponds to what EA/XR folks are thinking. Maybe this has been written up somewhere, and I haven't found it yet?

Without understanding this, it comes across as if tech progress is on indefinite hold until we somehow become better people and thus have sufficiently reduced XR—although it's unclear how we could ever reduce it enough, because of (1).

What does XR think about the large numbers of people who don't appreciate progress, or actively oppose it?

Returning to the road trip metaphor: while PS and EA/XR debate the ideal balance of resources towards steering vs. acceleration, and which is more neglected, there are other passengers in the ...

Published on June 2, 2021 6:47 PM GMT

I'm trying to get to the crux of the differences between the progress studies (PS) and the EA / existential risk (XR) communities. I'd love input from you all on my questions below.

The road trip metaphor

Let me set up a metaphor to frame the issue:

Picture all of humanity in a car, traveling down the highway of progress. Both PS and EA/XR agree that the trip is good, and that as long as we don't crash, faster would be better. But:

XR thinks that the car is out of control and that we need a better grip on the steering wheel. We should not accelerate until we can steer better, and maybe we should even slow down in order to avoid crashing.

PS thinks we're already slowing down, and so wants to put significant attention into re-accelerating. Sure, we probably need better steering too, but that's secondary.

(See also @Max_Daniel's recent post)

My questions

Here are some things I don't really understand about the XR position (granted that I haven't read the literature on it extensively yet, but I have read a number of the foundational papers).

(Edit for clarity: these questions are not proposed as cruxes. They are just questions I am unclear on, related to my attempt to find the crux)

How does XR weigh costs and benefits?

Is there any cost that is too high to pay, for any level of XR reduction? Are they willing to significantly increase global catastrophic risk—one notch down from XR in Bostrom's hierarchy—in order to decrease XR? I do get that impression. They seem to talk about any catastrophe less than full human extinction as, well, not that big a deal.

For instance, suppose that if we accelerate progress, we can end poverty (by whatever standard) one century earlier than otherwise. In that case, failing to do so, in itself, should be considered a global catastrophic risk, or close to it. If you're willing to accept GCR in order to slightly reduce XR, then OK—but it feels to me that you've fallen for a Pascal's Mugging.

Eliezer has specifically said that he doesn't accept Pascal's Mugging arguments in the x-risk context, and Holden Karnofsky has indicated the same. The only counterarguments I've seen conclude “so AI safety (or other specific x-risk) is still a worthy cause”—which I'm fine with. I don't see how you get to “so we shouldn't try to speed up technological progress.”

Does XR consider tech progress default-good or default-bad?

My take is that tech progress is default good, but we should be watchful for bad consequences and address specific risks. I think it makes sense to pursue specific projects that might increase AI safety, gene safety, etc. I even think there are times when it makes sense to put a short-term moratorium on progress in an area in order to work out some safety issues—this has been done once or twice already in gene safety.

When I talk to XR folks, I sometimes get the impression that they want to flip it around, and consider all tech progress to be bad unless we can make an XR-based case that it should go forward. That takes me back to point (1).

What would moral/social progress actually look like?

This idea that it's more important to make progress in non-tech areas: epistemics, morality, coordination, insight, governance, whatever. I actually sort of agree with that, but I'm not sure at all that what I have in mind there corresponds to what EA/XR folks are thinking. Maybe this has been written up somewhere, and I haven't found it yet?

Without understanding this, it comes across as if tech progress is on indefinite hold until we somehow become better people and thus have sufficiently reduced XR—although it's unclear how we could ever reduce it enough, because of (1).

What does XR think about the large numbers of people who don't appreciate progress, or actively oppose it?

Returning to the road trip metaphor: while PS and EA/XR debate the ideal balance of resources towards steering vs. acceleration, and which is more neglected, there are other passengers in the ...

Published on June 1, 2021 8:19 PM GMT

I sent a two-question survey to ~117 people working on long-term AI risk, asking about the level of existential risk from "humanity not doing enough technical AI safety research" and from "AI systems not doing/optimizing what the people deploying them wanted/intended".

44 people responded (~38% response rate). In all cases, these represent the views of specific individuals, not an official view of any organization. Since some people's views may have made them more/less likely to respond, I suggest caution in drawing strong conclusions from the results below. Another reason for caution is that respondents added a lot of caveats to their responses (see the anonymized spreadsheet), which the aggregate numbers don't capture.

I don’t plan to do any analysis on this data, just share it; anyone who wants to analyze it is of course welcome to.

If you'd like to make your own predictions before seeing the data, I made a separate spoiler-free post for that.

Methods

You can find a copy of the survey here. The main questions (including clarifying notes) were:

1. How likely do you think it is that the overall value of the future will be drastically less than it could have been, as a result of humanity not doing enough technical AI safety research?

2. How likely do you think it is that the overall value of the future will be drastically less than it could have been, as a result of AI systems not doing/optimizing what the people deploying them wanted/intended?

Note A: "Technical AI safety research" here means good-quality technical research aimed at figuring out how to get highly capable AI systems to produce long-term outcomes that are reliably beneficial.

Note B: The intent of question 1 is something like "How likely is it that our future will be drastically worse than the future of an (otherwise maximally similar) world where we put a huge civilizational effort into technical AI safety?" (For concreteness, we might imagine that human whole-brain emulation tech lets you gather ten thousand well-managed/coordinated top researchers to collaborate on technical AI safety for 200 subjective years well before the advent of AGI; and somehow this tech doesn't cause any other changes to the world.)

The intent of question 1 isn't "How likely is it that our future will be astronomically worse than the future of a world where God suddenly handed us the Optimal, Inhumanly Perfect Program?". (Though it's fine if you think the former has the same practical upshot as the latter.)

Note C: We're asking both 1 and 2 in case they end up getting very different answers. E.g., someone might give a lower answer to 1 than to 2 if they think there's significant existential risk from AI misalignment even in worlds where humanity put a major civilizational effort (like the thousands-of-emulations scenario) into technical safety research.

I also included optional fields for "Comments / questions / objections to the framing / etc." and "Your affiliation", and asked respondents to

Check all that apply:

☐ I'm doing (or have done) a lot of technical AI safety research.

☐ I'm doing (or have done) a lot of governance research or strategy analysis related to AGI or transformative AI.

I sent the survey out to two groups directly: MIRI's research team, and people who recently left OpenAI (mostly people suggested by Beth Barnes of OpenAI). I sent it to five other groups through org representatives (who I asked to send it to everyone at the org "who researches long-term AI topics, or who has done a lot of past work on such topics"): OpenAI, the Future of Humanity Institute (FHI), DeepMind, the Center for Human-Compatible AI (CHAI), and Open Philanthropy.,

The survey ran for 23 days (May 3–26), though it took time to circulate and some people didn't receive it until May 17.

Results

Each point is a response to Q1 (on the horizontal axis) and Q2 (on the vertical axis). Circles denote (pure) technical safety researchers, squares (pure) strateg...

Published on June 1, 2021 8:19 PM GMT

I sent a two-question survey to ~117 people working on long-term AI risk, asking about the level of existential risk from "humanity not doing enough technical AI safety research" and from "AI systems not doing/optimizing what the people deploying them wanted/intended".

44 people responded (~38% response rate). In all cases, these represent the views of specific individuals, not an official view of any organization. Since some people's views may have made them more/less likely to respond, I suggest caution in drawing strong conclusions from the results below. Another reason for caution is that respondents added a lot of caveats to their responses (see the anonymized spreadsheet), which the aggregate numbers don't capture.

I don’t plan to do any analysis on this data, just share it; anyone who wants to analyze it is of course welcome to.

If you'd like to make your own predictions before seeing the data, I made a separate spoiler-free post for that.

Methods

You can find a copy of the survey here. The main questions (including clarifying notes) were:

1. How likely do you think it is that the overall value of the future will be drastically less than it could have been, as a result of humanity not doing enough technical AI safety research?

2. How likely do you think it is that the overall value of the future will be drastically less than it could have been, as a result of AI systems not doing/optimizing what the people deploying them wanted/intended?

Note A: "Technical AI safety research" here means good-quality technical research aimed at figuring out how to get highly capable AI systems to produce long-term outcomes that are reliably beneficial.

Note B: The intent of question 1 is something like "How likely is it that our future will be drastically worse than the future of an (otherwise maximally similar) world where we put a huge civilizational effort into technical AI safety?" (For concreteness, we might imagine that human whole-brain emulation tech lets you gather ten thousand well-managed/coordinated top researchers to collaborate on technical AI safety for 200 subjective years well before the advent of AGI; and somehow this tech doesn't cause any other changes to the world.)

The intent of question 1 isn't "How likely is it that our future will be astronomically worse than the future of a world where God suddenly handed us the Optimal, Inhumanly Perfect Program?". (Though it's fine if you think the former has the same practical upshot as the latter.)

Note C: We're asking both 1 and 2 in case they end up getting very different answers. E.g., someone might give a lower answer to 1 than to 2 if they think there's significant existential risk from AI misalignment even in worlds where humanity put a major civilizational effort (like the thousands-of-emulations scenario) into technical safety research.

I also included optional fields for "Comments / questions / objections to the framing / etc." and "Your affiliation", and asked respondents to

Check all that apply:

☐ I'm doing (or have done) a lot of technical AI safety research.

☐ I'm doing (or have done) a lot of governance research or strategy analysis related to AGI or transformative AI.

I sent the survey out to two groups directly: MIRI's research team, and people who recently left OpenAI (mostly people suggested by Beth Barnes of OpenAI). I sent it to five other groups through org representatives (who I asked to send it to everyone at the org "who researches long-term AI topics, or who has done a lot of past work on such topics"): OpenAI, the Future of Humanity Institute (FHI), DeepMind, the Center for Human-Compatible AI (CHAI), and Open Philanthropy.,

The survey ran for 23 days (May 3–26), though it took time to circulate and some people didn't receive it until May 17.

Results

Each point is a response to Q1 (on the horizontal axis) and Q2 (on the vertical axis). Circles denote (pure) technical safety researchers, squares (pure) strateg...

Published on May 29, 2021 11:16 PM GMT

The EA Forum Editing Festival is over! (And has been, for a bit.)

I’ll use this post to look at how well the festival went, and to award prizes to people who were especially good contributors. (I’ve selected these people in consultation with Pablo Stafforini, who’s been spending more time on the wiki than I have.)

Overall

I’d consider the festival a qualified success.

The good

Traffic to the wiki increased substantially

Dozens of people applied at least one tag to a post they didn’t write; it feels like we succeeded in getting people to try this out at least once

We gave many good older posts their first tags, making them more accessible

The average wiki article is much more detailed and polished than it used to be

The bad

Overall tagging rates spiked during the festival, but now seem to have gone back to pre-festival numbers

Almost all our wiki edits came from three people, leaving us without a broad community of editors (for now)

I wish we’d seen more participation from more people, but I still wound up with a list of a dozen users whose work I’m happy to reward. (And I appreciate everyone who applied so much as one tag — see the “Honoring all contributors” section at the end.)

Traffic

Between 1 January and 24 February, our tag/wiki pages got ~4,400 total views, with an average view duration of 56 seconds.

Between 1 April and 24 May, those pages got ~16,700 views, with an average duration of 81 seconds.

To give a specific example: “Longtermism” is our most-visited tag, and went from 118 views (1/1 - 2/24) to 253 views (4/1 - 5/24).

One factor behind this is a test we ran in April to reroute clicks on certain EA Concepts pages to their wiki equivalents (as we might eventually sunset EA Concepts in favor of a set of wiki links). But this contributes only a small fraction of the change; between these two periods, EA Concepts lost ~4,000 pageviews, while the wiki gained ~12,000.

Tagging

As we requested, people tagged every post with 25 karma or more, then almost all posts with 20-24 karma (there are only three untagged posts in that category as I write this).

We also got a lot of strong tag suggestions from users.

Finally, the overall tag rate went up quite a bit during the festival; in the first week of May, we averaged around 100 tags per day, roughly tripling our number from the first week of April. Since the end of the festival, the rate has dropped back to around where it was at the beginning, which isn’t what I’d hoped for — but we did tag a lot of archival material in the meantime. If only new posts are getting tags at this point, each of them should get 3-4 tags on average at current rates, which seems very reasonable.

Wiki Edits

I’ve been happy to see the overall speed of progress on the wiki, but disappointed to see almost all of that progress come from three users so far (Pablo, his assistant Leonardo, and Michael Aird).

I think the initial collection of wiki content is shaping up well, but I still want us to have a broad set of editors so that new resources, papers, etc., routinely find their way into the appropriate articles. I’ll also note that Pablo, in the course of researching other community wikis, has found that “lack of contributors” appears to be the main reason these wikis fail to thrive.

Over the next few months, Pablo and I will be looking for ways to incentivize wiki edits from the wider community. A few things that come to mind:

A feature allowing people to see their wiki edits on their own profiles

An addition to the Forum Prize. awarding one or more wiki edits each month

More targeted solicitation toward Forum users we expect will be strong editors for specific articles

We also have a feature in the pipeline that will allow users to vote on edits, which should be helpful.

Prizes

As promised, we’ll be awarding at least $5000 in prizes to at least a dozen users (in the form of donations to charities listed on EA Funds).

Michael Aird: $2000

Michael has contribute...

Published on May 29, 2021 11:16 PM GMT

The EA Forum Editing Festival is over! (And has been, for a bit.)

I’ll use this post to look at how well the festival went, and to award prizes to people who were especially good contributors. (I’ve selected these people in consultation with Pablo Stafforini, who’s been spending more time on the wiki than I have.)

Overall

I’d consider the festival a qualified success.

The good

Traffic to the wiki increased substantially

Dozens of people applied at least one tag to a post they didn’t write; it feels like we succeeded in getting people to try this out at least once

We gave many good older posts their first tags, making them more accessible

The average wiki article is much more detailed and polished than it used to be

The bad

Overall tagging rates spiked during the festival, but now seem to have gone back to pre-festival numbers

Almost all our wiki edits came from three people, leaving us without a broad community of editors (for now)

I wish we’d seen more participation from more people, but I still wound up with a list of a dozen users whose work I’m happy to reward. (And I appreciate everyone who applied so much as one tag — see the “Honoring all contributors” section at the end.)

Traffic

Between 1 January and 24 February, our tag/wiki pages got ~4,400 total views, with an average view duration of 56 seconds.

Between 1 April and 24 May, those pages got ~16,700 views, with an average duration of 81 seconds.

To give a specific example: “Longtermism” is our most-visited tag, and went from 118 views (1/1 - 2/24) to 253 views (4/1 - 5/24).

One factor behind this is a test we ran in April to reroute clicks on certain EA Concepts pages to their wiki equivalents (as we might eventually sunset EA Concepts in favor of a set of wiki links). But this contributes only a small fraction of the change; between these two periods, EA Concepts lost ~4,000 pageviews, while the wiki gained ~12,000.

Tagging

As we requested, people tagged every post with 25 karma or more, then almost all posts with 20-24 karma (there are only three untagged posts in that category as I write this).

We also got a lot of strong tag suggestions from users.

Finally, the overall tag rate went up quite a bit during the festival; in the first week of May, we averaged around 100 tags per day, roughly tripling our number from the first week of April. Since the end of the festival, the rate has dropped back to around where it was at the beginning, which isn’t what I’d hoped for — but we did tag a lot of archival material in the meantime. If only new posts are getting tags at this point, each of them should get 3-4 tags on average at current rates, which seems very reasonable.

Wiki Edits

I’ve been happy to see the overall speed of progress on the wiki, but disappointed to see almost all of that progress come from three users so far (Pablo, his assistant Leonardo, and Michael Aird).

I think the initial collection of wiki content is shaping up well, but I still want us to have a broad set of editors so that new resources, papers, etc., routinely find their way into the appropriate articles. I’ll also note that Pablo, in the course of researching other community wikis, has found that “lack of contributors” appears to be the main reason these wikis fail to thrive.

Over the next few months, Pablo and I will be looking for ways to incentivize wiki edits from the wider community. A few things that come to mind:

A feature allowing people to see their wiki edits on their own profiles

An addition to the Forum Prize. awarding one or more wiki edits each month

More targeted solicitation toward Forum users we expect will be strong editors for specific articles

We also have a feature in the pipeline that will allow users to vote on edits, which should be helpful.

Prizes

As promised, we’ll be awarding at least $5000 in prizes to at least a dozen users (in the form of donations to charities listed on EA Funds).

Michael Aird: $2000

Michael has contribute...