Pattern Noise

Figure 15: Decomposing Pattern Noise

For a simple case of stable pattern noise, consider recruiters who predict the future performance of executives on the basis of a set of ratings. In chapter 9 we spoke of a “model of the judge.” The model of an individual recruiter assigns a weight to each rating, which corresponds to its importance in that recruiter’s judgments. The weights vary among recruiters: leadership may count more for one recruiter, communication skills for another. Such differences produce variability in the recruiters’ ranking of candidates—an instance of what we call stable pattern noise.

Personal reactions to individual cases can also produce patterns that are stable but highly specific. Imagine assessing Julie’s prospects after a troubled adolescence. What might lead you to pay more attention to some parts of her story than to others? Some details of the case may resonate with your life experience. Perhaps something about Julie reminds you of a close relative who kept almost succeeding but ultimately failing, because of what you believe are deep character flaws that were evident since that relative’s teenage years. Conversely, Julie’s story may evoke memories of a close friend who, after a troubled adolescence, did make it to medical school and is now a successful specialist. The associations that Julie evokes in different people are idiosyncratic and unpredictable, but they are likely to be stable: if you had read Julie’s story last week, you would have been reminded of the same people and would have seen her story in the same distinctively personal light.

Individual differences in the quality of judgments are another source of pattern noise. Imagine a single forecaster with crystal-ball powers that no one knows about (including herself). Her accuracy would make her deviate in many cases from the average forecast. In the absence of outcome data, these deviations would be regarded as pattern errors. When judgments are unverifiable, superior accuracy will look like pattern noise.

Pattern noise also arises from systematic differences in the ability to make valid judgments about different dimensions of a case. Consider the process of selection for professional sports teams. Coaches may focus on skills in various aspects of the game, physicians on susceptibility to injuries, and psychologists on motivation and resilience. When these different specialists evaluate the same players, we can expect a considerable amount of pattern noise. Similarly, professionals in the same generalist role may be more skilled at some aspects of the judgment task than at others. In such cases, pattern noise is better described as variability in what people know than as error.

When professionals make decisions on their own, variability in skills is simply noise. However, when management has the opportunity to construct teams that will reach judgments together, diversity of skills becomes a potential asset, because different professionals will cover different aspects of the judgment and complement one another. The discussion in chapter 21 examines how teams can benefit from this diversity.

A client of an insurance company or a defendant assigned a judge faces two lotteries: who makes the judgment, and when. The first selects more than the professional’s average level of strictness or leniency (the level error). The lottery also selects a kaleidoscopic assemblage of values, preferences, beliefs, memories, experiences, and associations that are unique to this particular professional. Whenever you make a judgment, you carry your own baggage, too. You come with the habits of mind you formed on the job and the wisdom you gained from your mentors. You bring along the successes that built your confidence and the mistakes that you are careful not to repeat. And somewhere in your brain are the formal rules you remember, those you forgot, and those you learned that it is okay to ignore. No one is exactly like you in all these respects; your stable pattern errors are unique to you.

The second lottery is the one that picks the moment when you make your judgment, the mood you are in, and other extraneous circumstances that should not affect your judgment but do. This lottery creates occasion noise. Imagine, for instance, that shortly before you read Julie’s case, you read a newspaper article about drug use on college campuses. The piece featured the story of a gifted student who was determined to go to law school and worked hard—but was unable to make up for the deficit he had accumulated while using drugs in his early years of college. Because it is fresh in your mind, this story will lead you to pay more attention to Julie’s pot-smoking habit in your assessment of her overall chances. However, you probably would not remember the article if you encountered the question about Julie in a couple of weeks (and you would obviously not have known about it if you had read the case yesterday). The effect of reading the newspaper article is transient; it is occasion noise.

As this example illustrates, there is no sharp discontinuity between stable pattern noise and the unstable variant that we call occasion noise. The main difference is whether a person’s unique sensitivity to some aspects of the case is itself permanent or transient. When the triggers of pattern noise are rooted in our personal experiences and values, we can expect the pattern to be stable, a reflection of our uniqueness.

The Personality Analogy

The idea of uniqueness in the responses of particular people to certain features or combinations of features is not immediately intuitive. To understand it, we might consider another complex combination of features that we all know well: the personalities of people around us. In fact, the event of a judge making a judgment about a case should be seen as a special case of a broader topic that is the domain of personality research: how a person acts in a situation. There is something to be learned about judgment from the decades of intensive study of the broader topic.

Psychologists have long sought to understand and measure individual differences in personality. People differ from one another in many ways; an early attempt to scan the dictionary for terms that may describe a person identified eighteen thousand words. Today, the dominant model of personality, the Big Five model, combines traits into five groupings (extraversion, agreeableness, conscientiousness, openness to experience, neuroticism), with each of the Big Five covering a range of distinguishable traits. A personality trait is understood as a predictor of actual behaviors. If someone is described as conscientious, we expect to observe some corresponding behaviors (arriving on time, keeping commitments, and so on). And if Andrew scores higher than Brad on a measure of aggressiveness, we should observe that, in most situations, Andrew behaves more aggressively than Brad does. In fact, however, the validity of broad traits for predicting specific behaviors is quite limited; a correlation of .30 (PC = 60%) would be considered high.

Common sense suggests that while behavior may be driven by personality, it is also strongly affected by situations. In some situations no one is aggressive, and in other situations everyone is. When consoling a bereaved friend, neither Andrew nor Brad will act aggressively; at a football game, however, both will display some aggression. Behavior reflects both personality and situation.

What makes people unique and endlessly interesting is that this joining of personality and situation is not a mechanical, additive function. For example, the situations that trigger more or less aggression are not the same for all people. Even if Andrew and Brad are equally aggressive on average, they do not necessarily display equal aggressiveness in every context. Perhaps Andrew is aggressive toward his peers but docile with superiors, whereas Brad’s level of aggressiveness is not sensitive to hierarchical level. Perhaps Brad is particularly prone to aggression when criticized and unusually restrained when physically threatened.

These signature patterns of response to situations are likely to be fairly stable over time. They constitute much of what we consider someone’s personality, although they do not lend themselves to a description by a broad trait. Andrew and Brad may share the same score on a test of aggression, but they are unique in their pattern of response to aggression triggers and contexts. Two people who share a trait level—if, for example, they are equally obstinate or equally generous—should be described by two distributions of behaviors that have the same average but not necessarily the same pattern of responses to different situations.

You can now see the parallel between this discussion of personality and the model of judgment we have presented. Level differences between judges correspond to the differences among scores on personality traits, which represent an average of behaviors in multiple situations. Cases are analogous to situations. A person’s judgment of a particular problem is only moderately predictable from that person’s average level, just as specific behaviors are only moderately predictable from personality traits. The ranking of individuals by their judgment varies substantially from one case to another because people differ in their reaction to the features and combinations of features that they find in each case. The signature of an individual who makes judgments and decisions is a unique pattern of sensitivity to features and a correspondingly unique pattern in the judgment of cases.

The analogy concerns professional judgments that are meant to follow a shared standard, where unwanted variation can be an error. It does not imply that all differences in knowledge or perspective are harmful. The point of the analogy is that pattern noise in judgment is not random—even if we have little hope of explaining it and even if the individuals who make distinctive judgments could not explain them.

Speaking of Pattern Noise

“You seem confident in your conclusion, but this is not an easy problem: there are cues pointing in different directions. Have you overlooked alternative interpretations of the evidence?”

“You and I have interviewed the same candidate, and usually we are equally demanding interviewers. Yet we have completely different judgments. Where does this pattern noise come from?”

“The uniqueness of people’s personalities is what makes them capable of innovation and creativity, and simply interesting and exciting to be around. When it comes to judgment, however, that uniqueness is not an asset.”

CHAPTER 17

The Sources of Noise

The Components of Noise

Figure 16 offers a combined graphical representation of the three equations we introduced in chapters 5, 6, and 16. The figure illustrates three successive breakdowns of error:

• error into bias and system noise,

• system noise into level noise and pattern noise,

• pattern noise into stable pattern noise and occasion noise.

The book’s decomposition expresses mean squared error (MSE) as the sum of squared bias and the variances associated with level noise, stable pattern noise, and occasion noise.