Superiority of Actuarial Methods

FACTORS UNDERLYING THE SUPERIORITY OF ACTUARIAL METHODS

Contrasts between the properties of actuarial procedures and clinical judgment help to explain their differing success. First, actuarial procedures, unlike the human judge, always lead to the same conclusion for a given data set. In one study, rheumatologists and radiologists often reached different opinions when reassessing cases they had previously evaluated (Fries et al., 1986). Such factors as fatigue, recent experience, or seemingly minor changes in the ordering of information or in the conceptualization of the case or task can produce random fluctuations in judgment (Hammond & Summers, 1965; Kahneman & Tversky, 1984). Random fluctuation decreases judgmental reliability and hence accuracy. For example, if the same data lead to the correct decision in one case but to a different, incorrect decision in the second case, overall accuracy will obviously suffer.

Actuarial methods also make it possible to test which variables predict the outcome and how to combine them. Multiple regression estimates their contributions within a fitted model. It does not automatically eliminate every unhelpful variable or ensure that the fitted relationships will hold in new data. Those questions require evaluation; explicit rules make that evaluation easier than when a judge’s weights are unstated or change from case to case.

Research shows that individuals have considerable difficulty distinguishing valid and invalid variables and commonly develop false beliefs in associations between variables (Chapman & Chapman, 1967, 1969). In psychology and psychiatry, clinicians often obtain little or no information about the accuracy of their diagnoses and predictions. Consultants asked to predict violence may never learn whether their predictions were correct. Furthermore, clinicians rarely receive immediate feedback about criterion judgments (for example, diagnoses) of comparable validity to that physicians obtain when the pathologist reports at the end of a clinicopathological conference (Meehl, 1973). Lacking sufficient or clear information about judgmental accuracy, it is problematic to determine the actual validity, if any, of the variables on which one relies. The same problem may occur if actuarial methods are applied blindly to new situations or settings without any performance checks.

In other circumstances, clinical judgments produce “self-fulfilling prophecies.” Prediction of an outcome often leads to decisions that influence or bias that outcome (Einhorn & Hogarth, 1978). An anecdote illustrates this problem. A psychiatrist in a murder trial predicted future dangerousness, and the defendant was sentenced to death. While on death row the defendant acted violently, which appeared to support the psychiatrist’s predictive powers. However, once sentenced to death this individual had little to lose; he may have acted differently had the psychiatrist’s appraisal, and in turn the sentence, been different.

Additionally, known outcomes seem more predictable than they are in advance (Fischhoff, 1975), and past predictions are mistakenly recalled as overly consistent with actual outcomes (Fischhoff, 1980; Fischhoff & Beyth-Marom, 1975). For example, Arkes et al. (1981) presented the same case materials to groups of physicians and asked them to assign probabilities to alternate diagnoses. When probabilities were assigned in foresight, each diagnosis was considered about equally likely. However, when the physicians were informed that one or another diagnosis had been established previously and they were then asked to state what initial diagnosis they likely would have made, they assigned the highest probability to whatever diagnosis they were told had been established. If one’s view or recall of initial judgments is inadvertently shaped to fit whatever happens to occur, outcome information will have little or no corrective value.

The clinician is also exposed to a skewed sample of humanity and, short of exposure to truly representative samples, it may be difficult, if not impossible, to determine relations among variables. For example, suppose that about half of the adolescents appraised for a history of juvenile delinquency show subtle electroencephalographic (EEG) abnormalities. Based on these co-occurrences the clinician may come to consider EEG abnormality a sign of delinquency or may conclude that delinquency is associated with brain dysfunction. In fact, clinicians have often postulated these relations (Spreen, 1981).

One cannot determine, however, whether a relation exists unless one also knows whether the sign occurs more frequently among those with, versus those without, the condition. For example, to determine whether EEG abnormality is associated with delinquency, one must also know the frequency with which delinquents do not obtain EEG abnormalities and the frequencies with which nondelinquents do and do not obtain EEG abnormalities. Further, even should a valid relation exist, one cannot determine the sign’s actual utility unless one knows: (1) how much more frequently it occurs when the condition is present than when it is absent and (2) the frequency of the condition. For example, a sign that is slightly more common among those with the condition may be of little diagnostic utility. If the condition is infrequent, then positive identifications based on the sign’s presence can even be wrong in most cases, for most individuals who display the sign will not have the condition. If 10% of brain-damaged individuals make a particular response on a psychological test and only 5% of normals, but 9 of 10 clinic patients are not brain-damaged, most patients who show the feature will not be brain-damaged.

In practice, the clinician is far more likely to evaluate individuals with significant problems than those without them, and this skewed exposure hinders attempts to make all of the needed comparisons. In fact, empirical study shows that EEG “abnormalities” are common among normal children and further suggests that the incidence of delinquency is no greater among those with than without neurological disorder (Capute, Neidermeyer, & Richardson, 1968; Spreen, 1981). The formation of such false beliefs is further compounded by a decided human tendency to overattend to information consistent with one’s hypotheses and to underattend to contradictory information (Greenwald, Pratkanis, & Leippe, 1986). The result is that mistaken beliefs or conclusions, once formed, resist counterevidence. . . .

The difficulty in separating valid and invalid variables on the basis of clinical experience or judgment is demonstrated in many studies examining diagnostic or predictive accuracy (Faust, 1984). The review cites studies in which clinical judgments based on interviews achieved, at best, negligible accuracy or validity (Carroll et al., 1982). Other studies show that clinical judgments based on psychological test results may be of low absolute validity (Leli & Filskov, 1981, 1984; Wedding, 1983; Werner, Rose, & Yesavage, 1983). Although clinical interviews or psychological tests can produce useful information, the clinical judge often cannot distinguish what is useful from what is useless. In all studies cited immediately above, statistical analysis of the same data uncovered useful variables or enhanced predictive accuracy.

The optimal weighting of variables is a less important advantage of the statistical method than is commonly assumed. In fact, unit (equal) weights yield predictions that correlate highly with those derived from optimally weighted composites, the only provisos being that the direction in which each predictor is related to the criterion can be specified beforehand and the predictors not be negatively correlated with each other (Dawes & Corrigan, 1974; Einhorn & Hogarth, 1975; Wainer, 1978; Wilks, 1938). Further, optimal weights are specific to the population in which they were derived, and any advantage gained in one setting may be lost when the same method is applied in another setting. However, when optimal weighting adds meaningfully to predictive accuracy, the human judge is at a decided disadvantage. As Meehl (1986, p. 372) has stated:

Surely we all know that the human brain is poor at weighting and computing. When you check out at a supermarket you don’t eyeball the heap of purchases and say to the clerk, “Well it looks to me as if it’s about $17.00 worth; what do you think?” The clerk adds it up. There are no strong arguments . . . from empirical studies . . . for believing that human beings can assign optimal weights in equations subjectively or that they apply their own weights consistently.

It might be objected that this analogy, offered not probatively but pedagogically, presupposes an additive model that a proponent of configural judgment will not accept. Suppose instead that the supermarket pricing rule were, “Whenever both beef and fresh vegetables are involved, multiply the logarithm of 0.78 of the meat price by the square root of twice the vegetable price”; would the clerk and customer eyeball that any better? Worse, almost certainly. When human judges perform poorly at estimating and applying the parameters of a simple or component mathematical function, they should not be expected to do better when required to weight a complex composite of these variables.