Bias and Noise: Components of Error in Human Judgment

Figure 1: Four teams

Imagine four teams shooting at targets. The figures referenced below are not included, but their patterns can be described in words. Team A’s shots are tightly clustered around the bull’s-eye, close to a perfect pattern.

We call Team B biased because its shots are systematically off target. As the figure illustrates, the consistency of the bias supports a prediction. If one of the team’s members were to take another shot, we would bet on its landing in the same area as the first five. The consistency of the bias also invites a causal explanation: perhaps the gunsight on the team’s rifle was bent.

We call Team C noisy because its shots are widely scattered. There is no obvious bias, because the impacts are roughly centered on the bull’s-eye. If one of the team’s members took another shot, we would know very little about where it is likely to hit. Furthermore, no interesting hypothesis comes to mind to explain the results of Team C. We know that its members are poor shots. We do not know why they are so noisy.

Team D is both biased and noisy. Like Team B, its shots are systematically off target; like Team C, its shots are widely scattered.

The targets illustrate two components of error in human judgment: bias, or systematic deviation, and noise, or random scatter.

The shooting range is a metaphor for what can go wrong in human judgment, especially in the diverse decisions that people make on behalf of organizations. In these situations, we will find the two types of error illustrated in Figure 1. Some judgments are biased; they are systematically off target. Other judgments are noisy, as people who are expected to agree end up at very different points around the target. Many organizations, unfortunately, are afflicted by both bias and noise.

Figure 2 illustrates an important difference between bias and noise. It shows what you would see at the shooting range if you were shown only the backs of the targets at which the teams were shooting, without any indication of the bull’s-eye they were aiming at.

From the back of the target, you cannot tell whether Team A or Team B is closer to the bull’s-eye. But you can tell at a glance that Teams C and D are noisy and that Teams A and B are not. Indeed, you know just as much about scatter as you did in Figure 1. A general property of noise is that you can recognize and measure it while knowing nothing about the target or bias.