Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: When is correlation transitive?, published by Ege Erdil on June 23, 2023 on LessWrong.

It's a well-known property of correlation that it's not transitive in general. If X,Y,Z are three real-valued random variables such that ρ(X,Y)>0 and ρ(Y,Z)>0, it doesn't have to be the case that ρ(X,Z)>0.

Nevertheless, there are some circumstances under which correlation is transitive. I will focus on two such cases in this post.

Primer: correlation as an inner product

For what follows, some background knowledge is necessary that we can regard correlations of real-valued random variables with finite second moments as inner products in an appropriate Hilbert space.

Specifically, if X,Y are two such random variables with zero mean and unit standard deviation, which is a simplification we can always make as correlation is invariant under translation and scalar multiplication, then we can compute

ρ(X,Y)=cov(X,Y)σXσY=E[XY]−E[X]E[Y]=E[XY]

The pairing (A,B)E[AB] defines an inner product on the space of random variables with finite second moments where two random variables are considered equivalent if they are equal with probability 1 (almost surely). The properties that we expect out of an inner product are easy to check: the pairing is obviously bilinear and positive definite.

Furthermore, it turns out this inner product turns the space of random variables with finite second moments into a Hilbert space: the vector space turns out to be complete under the induced norm ∥X∥=√E[X2]. Roughly speaking, this means that we can take orthogonal projections onto closed subspaces with impunity.

Now that we have this framework, we can move on to the main results of this post.

Correlation is transitive when the correlations are sufficiently strong

I'll first prove the following:

Claim 1: If ρ(X,Y)=a and ρ(Y,Z)=b, then

ab−√(1−a2)(1−b2)≤ρ(X,Z)≤ab+√(1−a2)(1−b2)

Moreover, these bounds are tight: for any a,b, there is a combination X,Y,Z for which we can make either the right or the left inequality into an equality.

Proof

We can assume X,Y,Z have mean zero and unit variance without loss of generality. Taking orthogonal projections of X,Z onto the one-dimensional subspace spanned by Y, we can write

X=aY+√1−a2HXYZ=bY+√1−b2HZY

where E[YHXY]=E[YHZY]=0 and the random variables HXY,HZY have mean zero and variance 1. Taking inner products gives

E[XZ]=ab+√(1−a2)(1−b2)E[HXYHZY]

Using the Cauchy-Schwarz inequality for our inner product finishes the proof: |E[HXYHZY]|≤∥HXY∥∥HZY∥=1. For the existence proof, let Y be an arbitrary random variable with mean zero and unit variance and pick HXY,HYZ to be perfectly correlated or perfectly anti-correlated standard Gaussians that are uncorrelated with Y.

Interpretation

When a,b are large and positive, the lower bound ab−√(1−a2)(1−b2) is also positive, and so we have a guaranteed positive correlation between X and Z.

One way to simplify this is to make it single-dimensional by assuming a=b. In this case, the lower bound is 2a2−1. If we want a guaranteed positive correlation between X and Z, this means the correlations ρ(X,Y)=ρ(Y,Z)=a have to satisfy a>1/√2≈0.7.

This condition is quite strict, and we might wonder if some transitivity of correlation can be recovered in the absence of such strong correlations between X,Y and Y,Z. It turns out the answer is yes, at least if we assume the random variables are in some sense "generic".

Correlation is transitive on the average

It turns out that in a suitable sense, when X,Y and Y,Z are positively correlated, there is a tendency for X,Z to also be positively correlated, even though per Claim 1 we can't deduce that they must be positively correlated.

The precise version of this claim is as follows:

Claim 2: Let X,Y,Z be vectors independently and uniformly distributed on the n-dimensional unit sphere Sn⊂Rn+1, and let −1≤a,b≤1 be two re...