Making predictions is a good practice, writing them down is even better.

However, we often make binary predictions when it is not necessary, such as

• Biden win popular vote: 91%
• Danish COVID deaths above 10.000 by January 1. 2022: 84%

Alternatively, we could make predictions from a normal distribution, such as ('~' means ‘comes from’):

• Biden’s popular vote ~ N(0.54, 0.03)
• Danish COVID deaths by January 1. 2022 ~ N(15,000, 5,000)

While making "Normal" predictions seems complicated, this post should be enough to get you started, and more importantly to get you a method for tracking your calibration, which is much harder with dichotomous predictions.

The key points are these:

1. Predicting from a normal is surprisingly easy.
2. Getting an actionable number for how over/under confident you are requires only simple math!
3. The normal distribution carries more information than the Bernoulli (binary outcome such as coins) and will therefore give you more information to act on!

Things this post will answer:

1. How do I make a normal prediction?
2. Why do I want to do this?
3. How do I track my calibration?

Quick recap about the normal distribution

The normal distribution is usually written as N($\mu$,$\sigma$) has 2 parameters:

• a location parameter $\mu$ (pronounced mu) which is both the most likely and the average value
• a scale parameter $\sigma$ (pronounced sigma) which captures uncertainty, high $\sigma$ implying high uncertainty

the 68-95-99.7 rule states that:

• 68% of your predictions should fall in $\mu \pm 1\sigma$

• 95% of your predictions should fall in $\mu \pm 2\sigma$

• 99.7% of your predictions should fall in $\mu \pm 3\sigma$

  

Finally 50% of the predictions should fall within $0.674\approx \frac{2}{3}\sigma$, which can be used as a quick spot check.

How to make predictions

To make a prediction, there are two steps. Step 1 is predicting $\mu$. Step 2 is using the 68-95-99.7 rule to capture your uncertainty in $\mu$.

I tried to predict Biden’s national vote share in the 2020 election. From the polls, I got 54% as a point estimate, so that seemed like a good guess for $\mu$. For $\sigma$ I used the 68-95-99.7 rule and tried to see what that would imply for different values of $\sigma$. Here is a table for 2-5%

$\sigma =2$ implies a 97.5% (95% interval + tail half tail) chance that Biden would get more than 50% of the votes ); I was not that confident. $\sigma =4$ implies a 84% chance that Biden would get more than 50% of the votes (68% + 32%/2), so there is 16% chance Trump wins, I likewise found this too high, so I settled on $\sigma =3$.

Why do I want to do this

Biden Got 52% of the vote share, which was within 1 sigma of my prediction. There are two weak lessons that I drew from this ONE data point.

1. The pollsters screwed up, so I should have regressed $\mu$ towards the mean (50%), such as predicting 53% instead of 54%
2. The prediction was exactly $\frac{2}{3}\sigma$ from $\mu$, so the $\sigma$ was on the 50%/50% boundary just as expected. This was lucky, but it's weak evidence that the $\sigma$ was well chosen.

Imagine I instead had predicted Biden wins (the popular vote) 91%, well guess what he won, so I was right... and that is it. Thinking I should have predicted 80% because the pollsters screwed up seems weird, as that is a weaker prediction and the bold one was right! I would need to predict a lot of other elections to see whether I am over or under confident.

How to track your calibration

Note: In the previous section we used $\mu$ and $\sigma$ for predictions. In this section we will use ${\mu }_i$ and ${\sigma }_i$ where i is the index (prediction 1, prediction 2... prediction N). We will use ${\hat{\sigma }}_z$ for the calibration point estimate; this means that ${\hat{\sigma }}_z$ is a number such as 1.73. In the next post in this series, we will use ${\sigma }_z$ for the calibration distribution, this means that ${\sigma }_z$ is a distribution like your predictions and thus has an uncertainty.

I also made a terrible prediction, during the early lock down in 2020. I predicted N(15,000, 5,000) COVID deaths by 2022 in Denmark. It turned out to be 3,200, which is $\frac{15000-3200}{5000}=2.36$ standard deviations away, so outside the 95% interval!

In this section we will transform your predictions to the Unit normal. This is called z-scoring, because if all predictions are on the same scale, then they are comparable:

$$z_{ideal}\sim N(0,1)$$

Normally when you convert to z-scores you use the data itself to calculate $\mu$ and $\sigma$, which guarantee a N(0,1). Here, we will use our predicted $\mu$ and $\sigma$. This means there will be a discrepancy between $z_{ideal}$ and our $z$. This discrepancy describes how under/over confident your intervals are, and thus describes your calibration, such that if ${\hat{\sigma }}_z$ = 2 then all your intervals should be twice as wide to achieve $z_{recalibrated}\sim N(0,1)$

First we z-score our data by calculating how many $\sigma$ they are away from the observed data $x$, using this formula:

$$z_i=\frac{{\mu }_i-x_i}{{\sigma }_i}$$

Second we calculate ${\hat{\sigma }}_z$ as the RMSE (root mean squared error) of all predictions:

$$Var\left(z\right)=\sum _{i=0}^{N}Var\left(z_i\right)=\sum _{i=0}^N{z}_i^2{\hat{\sigma }}_z=\sqrt{\frac{Var\left(z\right)}{N}}$$

And that is, let's calculate ${\hat{\sigma }}_z$ for my two predictions, first we calculate the variances:

• $Var\left(z_1\right)=(\frac{54-52}{3}{)}^2=\frac{4}{9}$
• $Var\left(z_2\right)=(\frac{15000-3200}{5000}{)}^2=5.57$

Then we calculate ${\hat{\sigma }}_z$

$${\sigma }_z=\sqrt{\frac{Var\left(z\right)}{N}}=\sqrt{\frac{\frac{4}{9}+5.57}{2}}=1.73$$

So if these were my only two predictions, then I should widen my future intervals by 73%. In other words, because ${\hat{\sigma }}_z$ is 1.73 and not 1, thus my intervals are too small by a factor of 1.73.

Still not convinced?

Here are some bonus arguments:

1. Weak 50/50: Sometimes you are actually 50/50, such as Scott's prediction that Bitcoin had a 50-50 shot of going over 3000 in 2019; that could be reformulated as "Bitcoin ~ N(3000, 1500)" such that a price of 10000 counts against the prediction. Now a weak prediction still gives evidence of calibration!
2. Overshooting and Undershooting: If Biden had gotten 20 or 80% of the votes, both things would be strong evidence of my prediction being wrong, where the binary predictions can only be 'wrong in one direction'
3. High Confidence Predictions are easier to calibrate: In Binary land a 99% prediction is very hard to calibrate because you need to make hundreds of them to get enough data (unless many turn out wrong of course). A Corresponding Normal prediction would have a small σ and thus give as much evidence of calibration as a 60% prediction.
4. Right for the Wrong Reason: All of N(50.67, 0.5), N(54, 3), N(58, 6), give Biden a 91% win chance, but for very different reasons, and will thus lead people to update differently after observing $x=52$.

Advanced Techniques

Sometimes your beliefs do not follow a Normal distribution. For example, the Bitcoin prediction N(3000, 1500) implies I believe there is a 2.5% chance the price will become negative, which is impossible. There are 3 solutions in increasing order of fanciness to deal with this:

1. Have different $\sigma$ for each direction such as (HN = Half Normal):

$$Bitcoin\sim 0.5HN(3000,{\sigma }_{up}=3000)+0.5HN(3000,{\sigma }_{down}=1500)$$

This means if it's above then $\sigma =3000$, while if it's below then $\sigma =1500$. If you do this, then you can use "the relevant $\sigma$" when calibrating and ignore the other one, so if the price of bitcoin ended up being $x=10.000$ then z becomes $\frac{7}{3}$:

$$z=\frac{3000-10000}{3000}=-\frac{7}{3}$$

2. Often you believe something goes up or down by a factor, such as Bitcoin dropping to half or doubling. For ease of example let imagine that Scott thought there was a 68% chance that Bitcoin’s value would change by less than a factor of 2.

$$\mu =lo{g}_{10}\left(3000\right)\approx 3.477\sigma =lo{g}_{10}\left(2\right)\approx 0.301lo{g}_{10}\left(Bitcoin\right)\sim N(3.477,0.301)$$

z-scoring works the same way, so if the Bitcoin price was 10.000 then:

$$lo{g}_{10}\left(x\right)=lo{g}_{10}\left(10.000\right)=4z\approx \frac{3.477-4}{0.301}\approx -1.74$$

3. (If this makes no sense, then ignore it): Using an arbitrary distribution for predictions, then use its CDF (Universality of the Uniform) to convert to $U(0,1)$, and then transform to z-score using the inverse CDF (percentile point function) of the Unit Normal. Finally use this as $z_i$ in when calculating your calibration.

Final Remarks

I want you to stop and appreciate that we can get a specific actionable number after 2 predictions, which is basically impossible with binary predictions! So start making normal predictions, rather than dichotomous ones!

As a final note, keep this distinction in mind:

1. If the data $x$ and the prediction $\mu$ are close, then you are a good predictor
2. If the mean prediction error on the z-scale is close to 1, then you are a well calibrated predictor.

Getting good at 1 requires domain knowledge for each specific prediction, while getting good at 2 is a general skill that applies to all predictions.

This post we calculated the point estimate ${\hat{\sigma }}_z=1.73$ based on 2 data points. There is a lot of uncertainty in a point estimate based on two data points, so we should expect the calibration distribution over ${\sigma }_z$ to be quite wide. The next post in this series will tackle this by calculating a Frequentest confidence interval for ${\hat{\sigma }}_z$ and a Bayesian posterior over ${\sigma }_z$. This allows us to make statements such as: I am 90% confident that $1<{\sigma }_z$, so it's much more likely that I am badly calibrated than unlucky. With only two data points it is however hard to tell the difference with much confidence.

Finally I would like to thank my editors Justis Mills and eric135 for making this readable.