Improving Uncertainty Assessment

IMPROVING UNCERTAINTY ASSESSMENT

In order to reduce the need for judgment in selecting data for producing recommended values, several attempts have been made to develop mathematical procedures for selecting and averaging measurements. Statisticians have long sought more robust estimators, such as trimmed means, that use all useful information without being unduly sensitive to outliers. However, in tests with real data, more sophisticated estimators provide only marginal improvements over simple averages (Stigler, 1977). Roos et al. (1975) describe a maximum likelihood technique based on their initial observation that the adjusted residual errors for particle properties had a broad tailed Student distribution. This was used in the Review of Particle Properties from 1976 to 1980, but was dropped after subsequent distributions of residuals differed in form (Trippe, 1983). There have also been attempts to develop algorithms for estimating the fundamental physical constants that deal better with apparently discrepant data. However, after tests with historical data, Taylor concluded that thoughtful and conscientious judgment about which data to exclude is still much more important than the choice of algorithm (Taylor, 1982).

It is no easy matter to eliminate judgmental biases (Lichtenstein et al., 1982). A categorical recommendation to treat all results equally would, on average, give undue credence to inaccurate observations and bad research. Nor is simple exhortation to “think harder” likely to prove any more useful here than it has in laboratory experiments on judgment. What is needed are ways to think more effectively. If the judgmental aspect of assessing uncertainty in physical measurements is explicitly acknowledged, then several techniques based on insights from psychological research may be helpful (Fischhoff & Whipple, 1981; Morgan et al., 1981, 1984; Spetzler & Staël von Holstein, 1975).

One approach is to encourage a broader search for relevant considerations, both supportive and disconfirming, and an unbiased evaluation of those considerations. For example, referees might be asked to scrutinize the account of an experiment blind before knowing its outcome and whether it affirmed prior expectations (Slovic & Fischhoff, 1977). To overcome the problem of anchoring on an initial best guess, investigators can be pressed to search for reasons why they might be wrong (Koriat, Lichtenstein, & Fischhoff, 1980). One device to encourage that search is to focus attention on extreme possibilities, for example, asking, “Imagine that, ten years hence, today’s best estimate proves to be off by four standard deviations; how could you explain it?” Additional prompts may come from reading “horror stories”: case studies describing major unsuspected errors in past experiments.

A second approach is to decompose the holistic judgement of overall uncertainty into its component sources of potential systematic errors, each to be estimated separately with the best available elicitation procedure (Fischhoff & Whipple, 1981; Morgan et al., 1981, 1984; Spetzler & Staël von Holstein, 1975). This can be aided by checklists covering each component of the measurement apparatus, each auxiliary quantity, and each theoretical assumption and approximation employed in the calculations.

Wherever analysis of reported uncertainties reveals a systematic bias, users of the measurements may use this information to readjust the original reports so as to improve calibration. Along these lines, the Particle Data Group expands the standard deviation whenever the Birge ratio is greater than unity among measurements of the same quantity (Rosenfeld, 1975; Particle Data Group, 1982). Their procedure usually produces stable recommended values for particle properties; however, it somewhat exaggerates the uncertainty by ignoring the possibility that some fluctuations of R**B above 1 are purely by chance. Roos et al. (1975) describe another procedure for readjusting the standard errors, based on the empirical distribution of the residuals they had observed for the particle properties.

The basis for such readjustments must be secondary analyses, such as those given here, showing probabilistic miscalibration in the estimates of uncertainty for previous similar measurements. In using such data, it is important to bear in mind that miscalibration reflects, not properties of the physical quantities themselves, but the nature of the procedures and judgmental processes used in selecting data and assessing uncertainty. Thus they are likely to vary with the training of the experimenter, the familiarity of the experimental techniques, and the maturity of the field, among other things. Compiling such data for wider classes of measurements can permit analysis of the importance of these factors. Even if uncertainty estimates are not explicitly readjusted, empirical calibration information could be helpful for anyone who uses measurements and recommended values for physical constants in their work. Even a rough estimate of overconfidence could help one interpret the significance of apparent discrepancies between measurements.

Another approach to improving uncertainty assessment is education designed to get better estimates in the first place. For example, in teaching experimental methods, greater exposure to the kind of results presented here should be helpful, together with attention to the role of judgment in the assessment process. Although one might hope that instructions in the processes and pitfalls of judgement would, by themselves, improve performance, the evidence to date suggests that people have difficulty in integrating an understanding of general principles with their own cognitive processes (see, e.g., Gaeth & Shanteau, 1984). What has proved more effective in other domains is task-specific training with personal feedback (Lichtenstein et al., 1982). For example, in laboratory classes, where students are required to measure the same quantities, it should be instructive to compile distributions of normalized residuals and measures of calibration for the class results and discuss together the reasons both for individual errors and for systematic miscalibration in reported uncertainties.