Sources of Overconfidence

In several sets of carefully analyzed measurements of physical constants, we found consistent replication of a robust finding of laboratory studies of human judgement: Reported uncertainties are too small. How could this apparent overconfidence arise? Experimental studies of human judgment have shown that such biases can arise quite unintentionally from cognitive strategies used in processing uncertain information. However, there are two possible ways in which such effects might be caused by deliberate decisions by the scientist.

One concerns the procedures chosen to assess the uncertainty. The recommended practice in physics is to consider all possible sources of systematic uncertainty when reporting results. However, without specific guidelines regarding what to consider and explicit recognition of the subjective elements in uncertainty assessments, one cannot be sure how comprehensively individual scientists have examined the uncertainty surrounding their own experiments. Conceivably, some of the apparent overconfidence reflects a deliberate decision to ignore the harder-to-assess sources of uncertainty.

A second possible source of bias is that, unlike laboratory experiments on judgment, which can take great care to ensure that subjects are motivated to express their uncertainty candidly, real-world settings create other pressures. Taylor et al. (1969) suggest that variations in attitude leads some investigators to cautiously assign unreasonably large errors so that a later measurement will not prove their work to have been “incorrect.” Others tend to underestimate the sources of systematic error in their experiments, perhaps because of an unconscious (or conscious) desire “to have done the best experiments.” Such variation in attitude, although out of keeping with scientific objectivity, is nevertheless unavoidable so long as scientists are also human beings. (p. 20)

In principle, none of these problems should affect the compilations of recommended values. These analyses are intended to be comprehensive, to consider reporting practices in the field, and to capture the state of knowledge (not just the precision of particular studies).

The factors that seem to have encouraged overconfidence in laboratory studies of judgement also seem to be likely candidates for having encouraged overconfidence in the estimation of physical constants. One such factor is the difficulty of thinking of reasons why one’s best guess might be wrong (Koriat, Lichtenstein, & Fischhoff, 1980). Supporting reasons typically come to mind more readily than contradicting ones. If ease of recollection is taken as an indicator of frequency (Tversky & Kahneman, 1973), the preponderance of reasons supporting the best guess will tend to be overestimated.

A second factor contributing to overconfidence is the unequal treatment of such confirming and disconfirming evidence as is discovered. When discrepant results are encountered, it could mean that either the new data or the old database contain undetected systematic errors. Unfortunately, people have a considerable ability to “explain away” events that are inconsistent with their prior beliefs (Nisbett & Ross, 1980). The data elimination and adjustment procedures that precede estimation of recommended values are natural places for disproportionate skepticism regarding unexpected results to emerge. One documented example of “trimming” (i.e., excessive zeal in eliminating outliers) emerged in an examination of Millikan’s laboratory notebooks for his oil-drop experiment. Of the 107 oil drops he observed, he excluded from publication 49 that seemed discrepant, despite his claim that he reported all his observations (Franklin, 1981). This had little effect on the best estimate of e, but increased the apparent precision of the experiment.

Having a preexisting recommended value may particularly encourage investigators to discard or adjust unexpected results, and so induce correlated errors in apparently independent experiments. The result would be initially small Birge ratios, unduly tight confidence intervals, and the exclusion of discrepant data that later prove to be more accurate than included ones. This can explain what Franklin (in press) has termed the bandwagon effect [e.g., the tendency for the measurements of the speed of light (in Fig. 37.1) to cluster around particular values at different periods]. The large Birge ratio (2.95) for high accuracy measurements of α−1 (Table 37.1) may be due to the fact that those measurements were actually derived from experiments designed to measure other quantities, so that discrepancies in the implied value of α−1 would have been less obvious to the original investigators. However, measurements of the ratio of the absolute ohm to the as-maintained ohm () have a natural anchor at unity (i.e., assuming they are equal), which could explain the unusual degree of agreement, evidenced by the Birge ratio of 0.40.