Confidence Calibration

Redefining confidence

Not much is ever certain. Samuel Arbesman’s The Half-Life of Facts is a great read about how practically every fact we’ve ever known has been subject to revision or reversal. We are in a perpetual state of learning, and that can make any prior fact obsolete. One of many examples he provides is about the presumed extinction of the coelacanth lineage, known from Late Cretaceous fossils. A mass-extinction event (such as a large meteor striking the Earth, a series of volcanic eruptions, or a permanent climate shift) ended the Cretaceous period. Non-avian dinosaurs died out, and coelacanths were also thought to have disappeared. In the late 1930s and independently in the mid-1950s, however, coelacanths were found alive and well. A species becoming “unextinct” is pretty common. Arbesman cites the work of a pair of biologists at the University of Queensland who made a list of all 187 species of mammals declared extinct in the last five hundred years. More than a third of those species have subsequently been rediscovered.

Given that even scientific facts can have an expiration date, we would all be well-advised to take a good hard look at our beliefs, which are formed and updated in a much more haphazard way than those in science. We don’t need someone challenging us to an actual bet to do this. We can think like a bettor, purposefully and on our own, like it’s a game even if we’re just doing it ourselves.

We would be better served as communicators and decision-makers if we thought less about whether we are confident in our beliefs and more about how confident we are. Instead of thinking of confidence as all-or-nothing (“I’m confident” or “I’m not confident”), our expression of our confidence would then capture all the shades of grey in between.

When we express our beliefs (to others or just to ourselves as part of our internal decision-making dialogue), they don’t generally come with qualifications. What if, in addition to expressing what we believe, we also rated our level of confidence about the accuracy of our belief on a scale of zero to ten? Zero would mean we are certain a belief is not true. Ten would mean we are certain that our belief is true. A zero-to-ten scale translates directly to percentages. If you think the belief rates a three, that means you are 30% sure the belief is accurate. A nine means you are 90% sure. So instead of saying to ourselves, “Citizen Kane won the Oscar for best picture,” we would say, “I think Citizen Kane won the Oscar for best picture but I’m only a six on that.” Or “I’m 60% that Citizen Kane won the Oscar for best picture.” For a well-calibrated set of comparable beliefs, about 40% of those assigned that confidence would turn out false; the historical outcome itself is fixed. Forcing ourselves to express how sure we are of our beliefs brings to plain sight the probabilistic nature of those beliefs, that what we believe is almost never 100% or 0% accurate but, rather, somewhere in between.

In a similar vein, the number can reflect several different kinds of uncertainty. “I’m 60% confident that Citizen Kane won best picture” reflects that our knowledge of this past event is incomplete. “I’m 60% confident the flight from Chicago will be late” incorporates a mix of our incomplete knowledge and the inherent uncertainty in predicting the future (e.g., the weather might intervene or there might be an unforeseen mechanical issue).

We can also express how confident we are by thinking about the number of plausible alternatives and declaring that range. For example, if I am stating my belief about what age Elvis died, I might say, “Somewhere between age forty and forty-seven.” I know he died in his forties and I remember that it was his earlier forties, so for me this is the range of plausible alternatives. The more we know about a topic, the better the quality of information we have, the tighter the range of plausible alternatives. (When it comes to predictions, the plausible range of outcomes would also be tighter when there is less luck involved.) The less we know about a topic or the more luck involved, the wider our range.

We can declare how sure we are whether we are thinking about a particular fact or set of facts (“dinosaurs were herd animals”), a prediction (“I think there is life on other planets”), or how the future will turn out given some decision we might make (“I think I will be happier if I move to Des Moines than I am where I live now” or “I think the company will be better off if we fire the president”). These are all beliefs of differing sorts.

Incorporating uncertainty into the way we think about our beliefs comes with many benefits. By expressing our level of confidence in what we believe, we are shifting our approach to how we view the world. Acknowledging uncertainty is the first step in measuring and narrowing it. Incorporating uncertainty in the way we think about what we believe creates open-mindedness, moving us closer to a more objective stance toward information that disagrees with us. We are less likely to succumb to motivated reasoning since it feels better to make small adjustments in degrees of certainty instead of having to grossly downgrade from “right” to “wrong.” When confronted with new evidence, it is a very different narrative to say, “I was 58% but now I’m 46%.” That doesn’t feel nearly as bad as “I thought I was right but now I’m wrong.” Our narrative of being a knowledgeable, educated, intelligent person who holds quality opinions isn’t compromised when we use new information to calibrate our beliefs, compared with having to make a full-on reversal. This shifts us away from treating information that disagrees with us as a threat, as something we have to defend against, making us better able to truthseek.

When we work toward belief calibration, we become less judgmental of ourselves. Incorporating percentages or ranges of alternatives into the expression of our beliefs means that our personal narrative no longer hinges on whether we were wrong or right but on how well we incorporate new information to adjust the estimate of how accurate our beliefs are. There is no sin in finding out there is evidence that contradicts what we believe. The only sin is in not using that evidence as objectively as possible to refine that belief going forward.

Declaring our uncertainty in our beliefs to others makes us more credible communicators. We assume that if we don’t come off as 100% confident, others will value our opinions less. The opposite is usually true. If one person expresses a belief as absolutely true, and someone else expresses a belief by saying, “I believe this to be true, and I’m 80% on it,” who are you more likely to believe? The fact that the person is expressing their confidence as less than 100% signals that they are trying to get at the truth, that they have considered the quantity and quality of their information with thoughtfulness and self-awareness. And thoughtful and self-aware people are more believable.

Expressing our level of confidence also invites people to be our collaborators. As I said, most of us don’t live our lives in poker rooms, where it is more socially acceptable to challenge a peer who expresses an opinion we believe to be inaccurate to a wager. Outside of the poker room, when we declare something as 100% fact, others might be reluctant to offer up new and relevant information that would inform our beliefs for two reasons. First, they might be afraid they are wrong and so won’t speak up, worried they will be judged for that, by us or themselves. Second, even if they are very confident their information is high quality, they might be afraid of making us feel bad or judged. By saying, “I’m 80%” and thereby communicating we aren’t sure, we open the door for others to tell us what they know. They realize they can contribute without having to confront us by saying or implying, “You’re wrong.” Admitting we are not sure is an invitation for help in refining our beliefs, and that will make our beliefs much more accurate over time as we are more likely to gather relevant information.

Expressing our beliefs this way also serves our listeners. We know that our default is to believe what we hear, without vetting the information too carefully. If we communicate to our listeners that we are not 100% on what we are saying, they are less likely to walk away having been infected by our beliefs. Expressing the belief as uncertain signals to our listeners that the belief needs further vetting, that further checking is still needed.

When scientists publish results of experiments, they share with the rest of their community their methods of gathering and analyzing the data, the data itself, and their confidence in that data. That makes it possible for others to assess the quality of the information being presented, systematized through peer review before publication. Scientists quantify uncertainty, but these statistical measures differ from personal confidence ratings. A p-value describes how unusual a result at least as extreme as the observed one would be under a specified null hypothesis and model; it does not give the probability that a belief is true. A confidence interval comes from a procedure designed to cover the true parameter at a stated rate over repeated samples. Scientists, by institutionalizing the expression of uncertainty, invite their community to share relevant information and to test and challenge the results and explanations. The information that gets shared back might confirm, disconfirm, or refine published hypotheses. The goal is to advance knowledge rather than affirm what we already believe. This is why science advances at a fast clip.*

By communicating our own uncertainty when sharing beliefs with others, we are inviting the people in our lives to act like scientists with us. This advances our beliefs at a faster clip because we miss out on fewer opportunities to get new information, information that would help us to calibrate the beliefs we have.

To assess calibration, record predictions and compare their outcomes with the confidence levels you assigned. Start by recording predictions—anything from “the project will ship on time” to “the restaurant will be crowded tonight”—alongside the percentage you’d assign to your certainty (say, 65% or 80%). After you know the outcome, tally how many times you were right and wrong for each confidence bracket. For example, you might create a simple table:

Confidence Level	Times Right	Times Wrong	% Correct
55% sure
65% sure
75% sure
85% sure
95% sure

For each row, divide Times Right by the sum of Times Right and Times Wrong, then convert the result to a percentage. If you said you were 75% sure and were correct on 75% of those predictions, confidence matches observed accuracy in that sample. Plot stated confidence on the horizontal axis and observed accuracy on the vertical axis. The diagonal represents agreement between them. A small sample can fall away from that line by chance; a persistent pattern of lower accuracy suggests overconfidence, and higher accuracy suggests underconfidence.

With enough comparable predictions, events assigned 90% confidence should occur about nine times out of ten. Review persistent gaps between confidence and observed frequency, including whether they occur mainly in unfamiliar subjects or under particular conditions. Use that feedback when assigning probabilities to future predictions.

Thinking in bets means treating decisions as choices under uncertainty and revising beliefs as evidence arrives. These habits need practice: naming a bias or assigning a percentage does not by itself improve judgment.