Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Use Normal Predictions, published by Jan Christian Refsgaard on January 9, 2022 on LessWrong.
Making predictions is a good practice, writing them down is even better.
However, we often make binary predictions when it is not necessary, such as
Biden win popular vote: 91%
Danish COVID deaths above 10.000 by January 1. 2022: 84%
Alternatively, we could make predictions from a normal distribution, such as ('~' means ‘comes from’):
Biden’s popular vote ~ N(0.54, 0.03)
Danish COVID deaths by January 1. 2022 ~ N(15,000, 5,000)
While making "Normal" predictions seems complicated, this post should be enough to get you started, and more importantly to get you a method for tracking your calibration, which is much harder with dichotomous predictions.
The key points are these:
Predicting from a normal is surprisingly easy.
Getting an actionable number for how over/under confident you are requires only simple math!
The normal distribution carries more information than the Bernoulli (binary outcome such as coins) and will therefore give you more information to act on!
Things this post will answer:
How do I make a normal prediction?
Why do I want to do this?
How do I track my calibration?
Quick recap about the normal distribution
The normal distribution is usually written as N(μ,σ) has 2 parameters:
a location parameter μ (pronounced mu) which is both the most likely and the average value
a scale parameter σ (pronounced sigma) which captures uncertainty, high σ implying high uncertainty
the 68-95-99.7 rule states that:
68% of your predictions should fall in μ±1σ
95% of your predictions should fall in μ±2σ
99.7% of your predictions should fall in μ±3σ
Finally 50% of the predictions should fall within 0.674≈23σ, which can be used as a quick spot check.
How to make predictions
To make a prediction, there are two steps. Step 1 is predicting μ. Step 2 is using the 68-95-99.7 rule to capture your uncertainty in μ.
I tried to predict Biden’s national vote share in the 2020 election. From the polls, I got 54% as a point estimate, so that seemed like a good guess for μ. For σ I used the 68-95-99.7 rule and tried to see what that would imply for different values of σ. Here is a table for 2-5%
Intervals. 68% 95% 99.7% σ=2. 52-56% 50-58% 48-60% σ=3. 51-57% 48-60% 45-63% σ=4. 50-58% 46-62% 42-66% σ=5. 49-59% 44-64% 39-69%
σ=2 implies a 97.5% (95% interval + half a tail) chance that Biden would get more than 50% of the votes; I was not that confident. σ=4 implies a 84% chance that Biden would get more than 50% of the votes (68% + 32%/2), and a 16% chance Trump wins, I likewise found this too high, so I settled on σ=3.
Why do I want to do this
Biden Got 52% of the vote share, which was within 1 sigma of my prediction. There are two weak lessons that I drew from this ONE data point.
The pollsters screwed up, so I should have regressed μ towards the mean (50%), such as predicting 53% instead of 54%
The prediction was exactly 23σ from μ, so the σ was on the 50%/50% boundary just as expected. This was lucky, but it's weak evidence that the σ was well chosen.
Imagine I instead had predicted Biden wins (the popular vote) 91%, well guess what he won, so I was right... and that is it. Thinking I should have predicted 80% because the pollsters screwed up seems weird, as that is a weaker prediction and the bold one was right! I would need to predict a lot of other elections to see whether I am over or under confident.
How to track your calibration
Note: In the previous section we used μ and σ for predictions. In this section we will use μi and σi where i is the index (prediction 1, prediction 2... prediction N). We will use ^σz for the calibration point estimate; this means that ^σz is a number such as 1.73. In the next post in this series, we will use σz for the calibration distribution, this means that σz...