Expert Judges and Expert Judgments: Two Textbook Examples

Expert Judges and Expert Judgments: Two Textbook Examples

Two often-cited examples of excellent calibration come from bridge players and weather forecasters. Few textbook discussions of calibration and prediction fail to mention the essentially perfectly calibrated predictions of expert bridge players (Keren, 1987) and meteorologists predicting rain in Chicago (Murphy & Winkler, 1977). These two striking case studies have been enough to convince many commentators of the intrinsic superiority of expert calibration in the field. Comparing the cases with theoretical models helps distinguish calibration from the processes that produce it.

Perfect calibration is theoretically possible with various possible combinations of outcome base rate and discriminability or difficulty. Indeed, ecological models predict perfect calibration in any naturalistic or representative environment. Case-based or direct support models predict excellent calibration when the base rate is moderate (near 50%) and when evidence quality is moderate. These conditions are well met in the predictions of the world-class bridge players studied by Keren (1987). The probability of successfully obtaining a given contract (the focal outcome) was moderate (55%) and so no response bias was needed or found (observed β = .05). Furthermore, the players were moderately good at discriminating between successful and unsuccessful contracts (α = .96, where α = 0 represents no ability to discriminate, and α values above 1.3 represent excellent discrimination); and this level of discriminability was almost exactly matched by the moderate extremity of predictions (σ = .90). Thus, excellent calibration can be entirely consistent with a heuristic case-based account such as direct support theory.

The government meteorologists studied by Murphy and Winkler (1977) were also moderately good at discriminating between days with and without rain (α = 1.09), and also showed moderate extremity of judgment (σ = .94). However, rain fell on only 26% of the days, so a substantially negative response bias was required (and obtained) to maintain calibration (β = −.71). Here, excellent calibration marks a distinct departure from the predictions of the heuristic case-based account. It is difficult to know if this sensitivity is a tribute to clinical or actuarial judgment, as meteorologists combine statistical summary data with intuitive adjustments.

Not all data sets considered in the source provided enough information to fit the full RST model. However, all provided a calibration curve or table relating subjective probabilities to actual outcomes, the base rate of the target or focal outcome, and some measure of discriminability (usually Az, the area under the Receiver Operating Characteristics curve in SDT, which is monotonically related to d in SDT and α in RST; the source also discusses the ordinal measure of discrimination recommended by Liberman & Tversky, 1993). In the absence of complete data, we provide a graphical and qualitative comparison of the direct support predictions with the expert data. We create “direct support” calibration curves by setting the extremity parameter σ to 1, indicating a moderate level of judgmental extremity that is insensitive to evidence quality, and by setting the response bias parameter β to 0, indicating no sensitivity to outcome base rate, and allowing discriminability and base rate to vary from one data set to the next. We then examine the fit of the direct support curves to the observed judgments.

These direct support curves serve as “neglect models,” indicating the pattern of judgments expected if base rate and discriminability are completely neglected. Observed curves that fall close to these predicted curves provide evidence consistent with the case-based neglect account. Observed curves that fall closer to the identity line than the predicted curves imply some sensitivity to base rate and discriminability.