Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Multiple stages of fallacy - justifications and non-justifications for the multiple stage fallacy, published by AronT on June 13, 2023 on LessWrong.

[Epistemic status: I am reasonably confident (75%) that these claims are largely correct.]

To estimate the probability of something happening, e.g. AI takeover, one approach is to break this down into stages and multiply the (conditional) probabilities together. I claim this approach yields systematically low answers. This is often referred to as the multiple stage fallacy - however - the term ‘fallacy’ in its label shouldn’t be viewed as an indisputable verdict. Smart people disagree with whether it truly is a fallacy, and it is the default method for even very good forecasters - so what’s going on? I haven’t been able to find a source that does this justice, and my impression is that consensus opinion on this overestimates the importance of disjunctive alternatives whilst underestimating the importance of correlated conditional probabilities. On a meta/ Bayesian level, I’ve seen it claimed several times that this is a fallacy because, empirically, it produces systematically low probabilities, but haven’t seen good evidence of this.

Disjunctive alternatives

What I see cited most often is that multiplying conditional probabilities fails to account for disjunctive routes. If x1,x2,x3 are events, and you are trying to estimate P(x3), then P(x3|x2)P(x2|x1)P(x1) doesn’t give the probability P(x3), it gives the probability P(x1∧x2∧x3).

If you want to compute the probability P(x3), you have to sum over all 4 possible combinations of x2 and x3 being true/ false. As the number of events grows, the number of combinations grows exponentially. In theory, this sounds like a problem, but in practice I believe the effect is smaller than claimed. In most situations where this fallacy appears to apply, I think it’s often reasonable to claim that the conclusion does actually imply the premises with reasonably high probability.

To take a concrete example, consider Carlsmith’s report on the risk of AI takeover. He lists 5 premises:

Timelines are short enough to build AGI by 2070.

There will be sufficient incentives to build AGI.

Alignment will be hard.

Systems will be deployed in ways that expose them to inputs that lead to them trying to seek power.

This will lead to AI takeover.

He computes the conditional probabilities of each premise, and then finally a conditional probability of an existential catastrophe conditional on these premises occurring.

In theory, there are 25=32 disjunctive routes to existential catastrophe here, but in practice I believe it’s reasonable to place fairly small probability mass on routes to catastrophe where the premises do not all occur.

Conditional probabilities are more correlated than you expect

Where the real fallacy lies is in incorrectly estimating conditional probabilities. The central claim is that in almost all cases, you should expect to underestimate how correlated the conditional probabilities you assign to each stage are.

As an example, consider the sample correlation matrix of the logits assigned to each stage by each of the Carlsmith report reviewers:

[Of the 15 nontrivial coefficients, 13 are positive. What is the probability of this occurring by chance? If X∼Bin(15,0.5), then P(X≤2)=0.4% which is significant at the 5% significance level. How much should you trust this? Obviously, almost not at all, but it does illustrate my point.]

There are two ways through which this happens, which roughly correspond to a failure of inside and outside views respectively:

Inside view failures

When we make predictions about the world, our predictions stem from an interplay of heuristics, biases, instincts, and in some cases formal models. These heuristics will often influence predictions of various seemingly ...