Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Case for Overconfidence is Overstated, published by Kevin Dorst on June 28, 2023 on LessWrong.

(Written with Matthew Mandelkern.)

TLDR: When asked to make interval estimates, people appear radically overconfident—far more so than when their estimates are elicited in other ways. But an accuracy-informativity tradeoff can explain this: asking for intervals incentivizes precision in a way that the other methods don’t.

Pencils ready! For each of the following quantities, name the narrowest interval that you’re 90%-confident contains the true value:

The population of the United Kingdom in 2020.

The distance from San Francisco to Boston.

The proportion of Americans who believe in God.

The height of an aircraft carrier.

Your 90%-confidence intervals are calibrated if 90% of them contain the true value. They are overprecise if less than 90% contain it (they are too narrow), and underprecise if more than 90% do.

We bet that at least one of your intervals failed to contain the true value. If so, then at most 75% of your intervals contained the correct answer, making you overprecise on this test.

You’re in good company. Overprecision is one of the most robust findings in judgment and decision-making: asked to give confidence intervals for unknown quantities, people are almost always overprecise.

The standard interpretation? People are systematically overconfident: more confident than they rationally should be. Overconfidence is blamed for many societal ills—from market failures to polarization to wars. Daniel Kahneman summed it up bluntly: “What would I eliminate if I had a magic wand? Overconfidence.”

But this is too quick. There are good reasons to think that when people answer questions under uncertainty, they form guesses that are sensitive to an accuracy-informativity tradeoff: they want their guess to be broad enough to be accurate, but narrow enough to be informative.

When asked who’s going to win the Republican nomination, “Trump” is quite informative, but “Trump or DeSantis” is more likely to be accurate. Which you guess depends on how you trade off accuracy and informativity: if you just want to say something true, you’ll guess “Trump or Desantis”, while if you want to say some thing informative, you’ll guess “Trump”.

So accuracy and informativity compete. This competition is especially stark if you’re giving an interval estimate. A wide interval (“the population of the UK is between 1 million and 1 billion”) is certain to contain the correct answer, but isn’t very informative. A narrow interval (“between 60 and 80 million”) is much more informative, but much less likely to contain the correct answer.

We think this simple observation explains many of the puzzling empirical findings on overprecision.

The Empirical Puzzle

The basic empirical finding is that people are overprecise, usually to a stark degree. Standardly, their 90%-confidence intervals contain the true value only 40–60% of the time. They are also almost never under-precise.

In itself, that might simply signal irrationality or overconfidence. But digging into the details, the findings get much more puzzling:

1) Intervals are insensitive to confidence level

You can ask for intervals at various levels of confidence. A 90%-confidence interval is the narrowest interval that you’re 90%-confident contains the true value; a 50%-confidence interval is the narrowest band that you’re 50%-confident contains it, etc.

A standard finding is that people give intervals of very similar width regardless of whether they’re asked for 50%-, 90%-, or even 99%-confidence intervals. For example, this study found that the widths of 99%-confidence intervals were statistically indistinguishable from those of 75%-confidence intervals.

This is puzzling. If people are giving genuine confidence intervals, this implies an extremely im...