Calibration and Miscalibration

Calibration and Miscalibration

Miscalibration presents itself in a number of forms. Figure 39.1 displays four typical patterns of miscalibrated probability judgments. The solid diagonal line, identity line, or line of perfect calibration, indicates the set of points at which judged probability and relative frequency coincide. The solid line marked A, where all judgments are higher than the corresponding relative frequency, represents overprediction bias. The solid line marked B, where all judgments are lower than the corresponding relative frequency, represents underprediction bias. The dotted line marked C, where judgments lower than 50% are too low and judgments higher than 50% are too high, represents overextremity bias. The dotted line marked D, where judgments lower than 50% are too high and judgments higher than 50% are too low, represents underextremity bias. Note that overextremity entails overly radical judgments (too close to 0 and 100) and underextremity entails overly conservative judgments (too far from 0 and 100). Combinations of under- or overprediction and either of the extremity biases are also possible, and result in lines that cross the diagonal at points other than 50%. Overconfidence, the poster child of judgmental biases, is a simple summary term (average subjective probability minus overall outcome relative frequency) that does not uniquely identify any one of these patterns (Wallsten & Budescu, 1983). Nonetheless, when referring to previous research, we use the term overconfidence, as this simple bias measure is often the primary statistic reported.

Figure 39.1. Calibration curves, generated by RST simulations, illustrating distinction between miscalibration due to overprediction (or underprediction) of target event versus that due to judgments which are overly (or insufficiently) extreme.

Note: Neglect model predictions based on RST simulation assuming no focal bias (β=0) and fixed judgmental extremity (σ=1). Over- and underprediction curves constructed assuming fixed discriminability (α=1) with varying target event base rate (BR=20% for overestimation; BR=80% for underestimation). Over- and underextremity curves constructed assuming fixed base rate (BR=50%) with varying discriminability (α=2.0 for overextremity; α=0.5 for underextremity).

Each of the four patterns illustrated in Fig. 39.1 is consistent with the use of judgmental heuristics (e.g., Kahneman & Tversky, 1973), and the pattern found depends on specific qualities of the judgmental task or situation and how they interact with the relevant heuristics (Griffin & Tversky, 1992; Griffin, Gonzalez, & Varey, 2000). The neglect of base rates that follows from reliance on judgmental heuristics leads to overprediction bias (A) when the outcome base rate is low and underprediction bias (B) when the outcome base rate is high. The neglect of evidence quality that follows from reliance on judgmental heuristics leads to overextremity bias (C) when the evidence quality is low and underextremity bias (D) when the evidence quality is high. The dependence of miscalibration upon the difficulty of the task and the knowledge of the judge (the difficulty or hard–easy effect) was established as the fundamental bias in general knowledge calibration by Lichtenstein, Fischhoff, and Philips (1982) in their comprehensive review. Nonetheless, general claims about the ubiquity of "overconfidence" continue to dominate managerial and academic textbooks.