Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Summary: Mistakes in the Moral Mathematics of Existential Risk (David Thorstad), published by Nicholas Kruus on April 10, 2024 on The Effective Altruism Forum.

This post summarizes "Three Mistakes in the Moral Mathematics of Existential Risk," a Global Priorities Institute Working Paper by David Thorstad. This post is part of my sequence of GPI Working Paper summaries. For more, Thorstad's blog, Reflective Altruism, has a five-part series on this paper.

Introduction

Many prominent figures in the effective altruism community argue existential risk mitigation offers astronomical value. Thorstad believes there are many philosophical ways to push back on this conclusion[1] and even mathematical ones.

Thorstad argues leading models of existential risk mitigation neglect morally relevant parameters, mislocating debates and inflating existential risk reduction's value by many orders of magnitude.

He broadly assumes we aren't in the

time of perils (which he justifies in

this paper) and treats extinction risks as only those that kill all humans.[2]

Mistake 1: Cumulative Risk

Existential risks recur many times throughout history, meaning they can be presented as a per-century risk repeated each century or as a cumulative risk of occurring during a total time interval (e.g., the cumulative risk of extinction before Earth becomes less habitable).

Mistake 1: Expected value calculations of existential risk mitigation reduce the cumulative risk, not the per-century risk.

Thorstad identifies two problems with this choice.

If humans live a long time, small reductions in cumulative risk require astronomical reductions in per-century risk. This is because the chance we survive for the total time interval in question depends on cumulative risk, and our cumulative survival chance must exceed our reduction in cumulative risk.

Reducing cumulative risks with our actions today requires changing the risk for many, many centuries to come. So, even if we can substantially shift the risk of extinction this century or even nearby ones, we'll likely have a hard time doing so for existential risk a thousand or million centuries from now.

For instance, if we want to create a meager one-in-a-hundred-million absolute reduction[3] in existential risk before Earth becomes less habitable,[4] the per-century risk must be nearly one-in-a-million or lower.[5] Many longtermists estimate this century's existential risk to be ~15-20% or higher,[6] in which case we'd need to drive the per-century risk down a hundred thousand times.

Hence, many expected value calculations of existential risk mitigation demand vastly greater reductions in per-century risk than they initially seem to.

Mistake 2: Background Risk

Millett and Snyder-Beattie (MSB) offer

one of the most cited papers discussing biorisk - biological extinction risk - featuring a favorable cost-effectiveness estimate. While Thorstad believes many complaints about MSB's model exist, he raises two.

Mistake 2: Existential risk mitigation calculations (including MSB's model) ignore background risk.

In MSB's model, the background risk is the risk of extinction from all non-biological sources. But, modifying this model to include background risk changes the estimated cost-effectiveness considerably.

Without background risk, a 1% relative reduction in biorisk has a meaningful impact on per-century risk: it discounts per-century risk by 1%. But, when you include non-biological background risk, the same reduction in biorisk changes the per-century risk far less: per-century risk becomes the discounted biorisk plus the full background risk.

Since many longtermists believe the per-century risk is very high (~15-20% or higher)[6] and thus much greater than biorisk, this substantially reduces biorisk mitigation's estimated cost-effectiveness:

For reference, GiveWell e...