The Determinants of Confidence

The Determinants of Confidence

The assessment of confidence or degree of belief in a given hypothesis typically requires the integration of different kinds of evidence. In many problems, it is possible to distinguish between the strength, or extremeness, of the evidence and its weight, or predictive validity.

When we evaluate a letter of recommendation for a graduate student written by a former teacher, we may wish to consider two separate aspects of the evidence: (1) How positive or warm is the letter? and (2) How credible or knowledgeable is the writer? The first question refers to the strength or extremeness of the evidence, whereas the second question refers to its weight or credence. Similarly, suppose we wish to evaluate the evidence for the hypothesis that a coin is biased in favor of heads rather than tails. In this case, the proportion of heads in a sample reflects the strength of evidence for the hypothesis in question, and the size of the sample reflects the credence of these data. The distinction between the strength of evidence and its weight is closely related to the distinction between the size of an effect (e.g., a difference between two means) and its reliability (e.g., the standard error of the difference).

Although it is not always possible to decompose the impact of evidence into the separate contributions of strength and weight, there are many contexts in which they can be varied independently. A strong or a weak recommendation may come from a reliable or unreliable source, and the same proportion of heads can be observed in a small or large sample.

Statistical theory and the calculus of chance prescribe rules for combining strength and weight. For example, probability theory specifies how sample proportion and sample size combine to determine posterior probability. The extensive experimental literature on judgment under uncertainty indicates that people do not combine strength and weight in accord with the rules of probability and statistics. Rather, intuitive judgments are overly influenced by the degree to which the available evidence is representative of the hypothesis in question (Dawes, 1988; Kahneman, Slovic, & Tversky, 1982; Nisbett & Ross, 1980). If people were to rely on representativeness alone, their judgments (e.g., that a person being interviewed will be a successful manager) would depend only on the strength of their impression (e.g., the degree to which the individual in question “looks like” a successful manager) with no regard for other factors that control predictive validity. In many situations, however, it appears that people do not neglect these factors altogether. We propose instead that people focus on the strength of the evidence – at least, as they perceive it – and then make some adjustment in response to its weight.

In evaluating a letter of recommendation, we suggest, people first attend to the warmth of the recommendation and then make allowance for the writer’s limited knowledge. Similarly, when judging whether a coin is biased in favor of heads or tails, people focus on the proportion of heads in the sample and then adjust their judgment according to the number of tosses. Because such an adjustment is generally insufficient (Slovic & Lichtenstein, 1971; Tversky & Kahneman, 1974), the strength of the evidence tends to dominate its weight in comparison to an appropriate statistical model. Furthermore, the tendency to focus on the strength of the evidence leads people to underutilize other variables that control predictive validity, such as base rate and discriminability. This treatment combines judgment by representativeness, which is based entirely on the strength of an impression, with an anchoring and adjustment process that takes the weight of the evidence into account, albeit insufficiently.

This hypothesis implies a distinctive pattern of overconfidence and underconfidence. If people are highly sensitive to variations in the extremeness of evidence and not sufficiently sensitive to variations in its credence or predictive validity, then judgments will be overconfident when strength is high and weight is low, and they will be underconfident when weight is high and strength is low. As shown later, this hypothesis serves to organize and summarize much experimental evidence on judgment under uncertainty.

Consider the prediction of success in graduate school on the basis of a letter of recommendation. If people focus primarily on the warmth of the recommendation with insufficient regard for the credibility of the writer, or the correlation between the predictor and the criterion, they will be overconfident when they encounter a glowing letter based on casual contact, and they will be underconfident when they encounter a moderately positive letter from a highly knowledgeable source. Similarly, if people’s judgments regarding the bias of a coin are determined primarily by the proportion of heads and tails in the sample with insufficient regard for sample size, then they will be overconfident when they observe an extreme proportion in a small sample, and underconfident when they observe a moderate proportion in a large sample.