Discriminability and Confidence in Probability Judgments
In the probability-judgment studies described here, participants tended to be overconfident when competing hypotheses were hard to distinguish and underconfident when the evidence distinguished them strongly.
THE IDEA
Discriminability and Confidence
Table 13.3 summarizes sample data, posterior probabilities, and subjects' median confidence judgments. The comparison of confidence judgments to Bayesian posterior probabilities indicates that subjects focused primarily on the degree to which the data fit the favored hypothesis with insufficient regard for how well they fit the alternate hypothesis (Fischhoff & Beyth-Marom, 1983). Figure 13.4 plots subjects' median confidence judgments against Bayesian posterior probabilities both for low and high discriminability comparisons. When discriminability between hypotheses was low (e.g., coin bias of .6 vs. .5), subjects were slightly overconfident; when discriminability was high (e.g., bias of .6 vs. .25), subjects were grossly underconfident.
In early experimental literature, most studies examined symmetric hypotheses that were highly discriminable (e.g., 3:2 vs. 2:3) and found consistent underconfidence. In accord with the hypothesis, studies that included pairs of hypotheses of low discriminability found overconfidence. For example, Peterson and Miller (1965) found overconfidence in posterior probability judgments when the respective ratios were 3:2 and 3:4, and Phillips and Edwards (1966) found overconfidence when the ratios were 11:9 and 9:11.
When to use it
Examining whether confidence in a diagnosis reflects how well the evidence distinguishes possible causes
Quality control: Comparing confidence judgments with statistical evidence when distinguishing defect rates such as 1% and 2%
Investment research: Comparing confidence in competing financial explanations with how well the same evidence distinguishes them
Election forecasting: Assessing how well a polling result distinguishes competing hypotheses about vote shares
FROM THEORY TO EVERYDAY LIFE
In practice
Low Discriminability Overconfidence
In a study by Peterson and Miller (1965), subjects were overconfident when comparing hypotheses with low discriminability (ratios 3:2 vs. 3:4). For example, if the evidence slightly favors a 3:2 ratio over a 3:4 ratio, people wrongly express high confidence in the 3:2 hypothesis.
High Discriminability Underconfidence
In early studies using highly discriminable hypotheses (e.g., 3:2 vs. 2:3), subjects consistently showed underconfidence. Even when data strongly supported one hypothesis, they were less confident than Bayesian analysis would dictate.
A MOMENT TO REFLECT
Put it to work.
Think of a recent decision where you had to choose between two similar options. Did your confidence in the choice reflect the actual strength of the evidence? How might discriminability have affected your judgment?
When comparing two very different hypotheses, people often underweight strong evidence. Can you recall a situation where you were less certain than the data warranted? Why might this happen?
How could you adjust your judgment framework to account for the discriminability of the hypotheses you are comparing?
References
- Fischhoff & Beyth-Marom (1983) - Hypothesis evaluation from a Bayesian perspective — Fischhoff, B. & Beyth-Marom, R. (1983)
- Peterson, Schneider, & Miller (1965) - Sample size and the revision of subjective probabilities — Peterson, C. R., Schneider, R. J., & Miller, A. J. (1965)
- Peterson & Miller (1965) - Confidence and the revision of subjective probabilities — Peterson, C. R. & Miller, A. J. (1965)
- Phillips & Edwards (1966) - Conservatism in a simple probability inference task — Phillips, L. D. & Edwards, W. (1966)
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)