Overall Accuracy in Probability Judgments
Overall accuracy in probability judgments is a composite measure capturing calibration, base rate, and discrimination, often indexed by the Brier score (mean probability score). Cross-cultural studies show mixed results, with some showing similar overall accuracy despite differences in overconfidence.
THE IDEA
Overall Accuracy
Overall accuracy in probability judgments can be conceptualized as follows (Murphy, 1973; see also Yates, 1990, 1994):
| image | (2)
In this conceptual functional equation, Base Rate refers to the overall relative frequency of the focal, target event in a judgment situation (e.g., proportion correct in a general-knowledge study). (We discuss discrimination later.) For the moment, the key point highlighted by Eq. 2 is that calibration is only one of several contributors to overall judgment accuracy. This then directs our attention to a now-obvious question: What, if any, are the cross-cultural variations in overall accuracy for probability judgments? The mean probability score, denoted 
(sometimes called the Brier (1950) score), is the most common means of indexing overall accuracy, and thus provides a means of addressing this question. (See Yates, 1990, Chap. 3, 1984, 1998, for discussions of the details of accuracy measures like .)
Recall that the mainland Chinese respondents in Yates et al. (1989) exhibited much greater general-knowledge overconfidence than their American and Japanese counterparts. Yet overall accuracy, as indexed by , was essentially the same across all three groups. In the general-knowledge study by Yates et al. (1999), the mean values of for judgments by respondents from Taiwan, Japan, and the United States were, .2375, .1971, and .2286 respectively. ( ranges between 0 and 1, with 0 being ideal.) The Japanese versus Chinese and Japanese versus American comparisons were statistically significant by the usual standards, whereas the Chinese versus American comparison was not. Yates et al. (1998) also examined overall accuracy, but in the context of judgments about a homogeneous class of externally defined focal events (diseases in a fictional medical ecology) rather than general knowledge. In that context, the overall accuracy of the Chinese participants was significantly worse than that of the Japanese and American participants. We are thus left with something of a muddled picture. Sometimes cross-cultural differences in overall probability judgment accuracy are observed, but sometimes not.
When to use it
Evaluating the performance of a weather forecaster who makes probability of rain predictions over many days.
Comparing the judgment accuracy of expert and novice investors when predicting stock price movements.
Assessing the impact of a training program designed to improve probability judgments in medical diagnosis.
Investigating cultural differences in decision-making under uncertainty in multinational organizations.
FROM THEORY TO EVERYDAY LIFE
In practice
General-Knowledge Overconfidence vs. Overall Accuracy
In Yates et al. (1989), mainland Chinese respondents showed much greater overconfidence in general knowledge compared to American and Japanese respondents, yet overall accuracy (as measured by the Brier score) was similar across all three groups. Overconfidence in one component (calibration) does not necessarily degrade overall accuracy.
Medical Judgments and Cross-Cultural Accuracy Differences
In a study by Yates et al. (1998) using a fictional medical ecology, Chinese participants had significantly worse overall accuracy than Japanese and American participants. Unlike the general knowledge task, this homogeneous externally defined event set revealed a clear cross-cultural difference in overall probability judgment accuracy.
A MOMENT TO REFLECT
Put it to work.
How might focusing solely on calibration, rather than overall accuracy, lead to an incomplete picture of judgment performance?
Why might cross-cultural differences in overconfidence not always translate into differences in overall accuracy?
In what situations would you choose to use the Brier score over a measure of calibration or discrimination alone?
Often confused with
References
- Murphy (1973) — Murphy (1973)
- Yates (1990) — Yates (1990)
- Yates (1994) — Yates (1994)
- Brier (1950) Score — Brier (1950)
- Yates et al. (1989) — Yates et al. (1989)
- Yates et al. (1999) — Yates et al. (1999)
- Yates et al. (1998) — Yates et al. (1998)
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)