Error Models

A number of researchers have considered the consequences of unreliability in confidence judgments for the assessment of calibration. Erev, Wallsten, and Budescu (1994; also Wallsten & Budescu, 1983) offered a Thurstonian “true score plus error” model in which an underlying feeling of confidence is subjected to some degree of stochastic variation prior to being translated into an overt judgment. They assumed, for demonstration purposes, that the underlying “true” judgment equals the actuarial probability of the event being judged, and investigated how error due to unreliability affected the resulting calibration analysis. To illustrate, suppose that a weather forecaster’s true judgment based on a set of forecasting cues is equal to the actual probability of precipitation given those cues, but that his or her overt judgment is subject to some error. How does this error influence calibration? Erev et al. (1994; also Budescu, Erev, and Wallsten, 1997), assuming a particular instantiation of their general model, demonstrated using simulation data that calibration curves can indicate overextremity even when the confidence judgments themselves are unbiased estimates of the true probability subject to symmetric error. This result arises primarily from effects of regression, but may also be enhanced by boundary effects due to rounding at scale endpoints. Pfeifer (1994) independently developed a very similar thesis to that of Erev et al. Such error models generally produce greater overextremity as the degree of error increases, and in this sense can account for the difficulty effect.

The results of Erev et al. can be interpreted in two ways, as a methodological prescription or as a descriptive model of probability judgment. The methodological prescription highlights the perils of diagnosing overconfidence on the basis of the calibration curve because “error alone” can produce the appearance of overconfidence in such a curve even when underlying “true” judgments are unbiased. Brenner, Koehler, Liberman, and Tversky (1996) argued that the standard measure of overconfidence in 2 AFC tasks, namely the difference between mean confidence and overall accuracy, provides an unbiased estimate of aggregate overconfidence which is not subject to the same kind of regression effect apparent in the calibration curve. In response to the more general methodological recommendations of Erev et al., Brenner (2000) raised questions about the appropriateness of evaluating calibration on hard-to-define true scores rather than well-defined actual judgments.

Budescu, Wallsten, and Au (1997) assess the relative contributions of measurement error and systematic bias (i.e., over- or underconfidence) to overall miscalibration, using Wallsten and González-Vallejo’s (1994) statement verification model. The reliability of subjective probability judgments is assessed from replicate judgments and used to estimate the degree of miscalibration expected on the basis of error alone (i.e., in the absence of systematic bias), which is then used to construct a less strict standard of “ideal” performance than that which is usually employed (see also Klayman, Soll, González-Vallejo, & Barlas, 1999). Using this method, Budescu, Wallsten, and Au (1997) found substantial overextremity, even after correcting for the unreliability of the assessments, in a full-range task involving the relative populations of pairs of cities. As a descriptive model, then, the assumption of an unbiased “true score” subject to error is not a sufficient account of the miscalibration found in this and other laboratory tasks.

Several researchers (Björkman, 1994; Juslin et al., 1997; Juslin, Wennerholm, & Olsson, 1999; Soll, 1996) offered modified ecological models in which stochastic error components have been introduced as a way of improving the fit of such models to actual calibration data. In such models, the “internal” probability is only an estimate of the corresponding ecological probability, unbiased but subject to sampling error. Soll (1996), Juslin et al. (1997), and Budescu, Erev, and Wallsten (1997) have shown, using simulations, that a modified ecological model incorporating sampling error can produce overconfidence that increases with task difficulty (as determined by the uncertainty of the environment and the sample size, i.e., number of available previous cases matching the present one). Soll (1996) further demonstrated, in a cue-based task somewhat different from the typical calibration experiment, that subjects appeared to be able to produce unbiased estimates of the ecological conditional probabilities associated with a set of cues, even though in terms of calibration these subjects were substantially overconfident.

Sampling error alone, it should be noted, cannot produce underextremity (Juslin et al., 1997, p. 197). Given the number of empirical studies which have reported underconfidence for relatively easy tasks (e.g., Baranski & Petrusic, 1994; Björkman, Juslin, & Winman, 1993; Brake, Doherty, & Kleiter, 1997; Griffin & Tversky, 1992; Juslin et al., 1999; Suantak et al., 1996), it is clear that sampling error, by itself, cannot account for the full range of results from calibration experiments. Moreover, at a conceptual level, it is not entirely clear what it means to argue that a judge’s confidence assessments are not systematically biased, but instead merely fail to account for the uncertainty associated with prediction based on the available evidence. Arguably, it is precisely such uncertainty which the judge is expected to convey in his or her confidence assessment. This issue highlights an apparent conflict of goals between proponents of error models, who are concerned with whether the judge says what she means (i.e., gives a confidence assessment that properly reflects the corresponding “internal” probability), and the typical “consumer” of such assessments in everyday life, who is more concerned with whether the judge means what she says (i.e., gives a confidence assessment that properly reflects the actual probability of being correct). The consumer of judgments by Soll’s (1996) subjects, for example, is unlikely to take much comfort in the purported accuracy of these subjects’ underlying judgments upon discovering that when they indicated confidence in the range of 90% to 100%, they were accurate approximately 60% of the time.