Noise in Human Judgment
Noise is unwanted variability in judgments that should ideally be identical. It is a major, often overlooked component of error, distinct from bias, and arises from differences between judges, inconsistent reactions to cases, and transient factors.
THE IDEA
Understanding Noise
Noise refers to the variability in judgments that should ideally be identical. When different professionals make different judgments about the same case, or the same professional makes inconsistent judgments on different occasions, noise is present. Noise is a statistical concept that becomes visible only when we examine an ensemble of similar judgments.
Components of System Noise
System noise includes differences in average judgments and differences in reactions to particular cases:
- Level noise: Consistent differences between judges in their average judgments (e.g., some judges are consistently harsher than others).
- Pattern noise: Idiosyncratic reactions of particular judges to specific cases. This is further divided into:
- Stable pattern noise: Persistent personal reactions that are consistent over time.
- Occasion noise: Random variability due to transient circumstances (mood, fatigue, time of day).
Research shows that stable pattern noise is often the largest component of system noise, larger than level noise or occasion noise.
Why Noise Is Invisible
Noise can be difficult to notice when we focus on causes of individual errors. We naturally explain errors by pointing to biases or incompetence, but noise is inherently statistical. As a result, noise is often ignored until a noise audit reveals its magnitude.
Reducing Noise: Decision Hygiene
Decision hygiene refers to techniques that reduce noise without targeting specific biases. Key strategies include:
- Aggregation: Averaging independent judgments with comparable error variance reduces the standard deviation of their errors in proportion to the square root of their number.
- Selection: Choosing better judges (e.g., superforecasters) who are less noisy and less biased.
- Guidelines: Using structured criteria to standardize judgments (e.g., Apgar score in medicine).
- Sequencing information: Controlling the flow of information to prevent premature intuitions and confirmation bias.
- Structuring judgments: Decomposing complex judgments into simpler, independent sub-judgments.
When to use it
A company uses forced ranking to reduce level noise in performance evaluations, but must guard against demotivation and pattern noise.
A hospital implements diagnostic guidelines (e.g., Centor score for strep throat) to reduce variability in diagnoses and improve patient outcomes.
A forecasting team collects independent estimates, averages them, and compares the result with individual forecasts to assess whether aggregation improves accuracy.
A forensic lab adopts linear sequential unmasking to prevent confirmation bias, ensuring examiners see only relevant information in the proper order.
FROM THEORY TO EVERYDAY LIFE
In practice
Fingerprint Misidentification
In the Brandon Mayfield case, several FBI experts incorrectly identified a fingerprint. The account illustrates why independent verification and control over contextual information matter; agreement among examiners is less informative when their judgments influence one another.
Superforecasters vs. Intelligence Analysts
In the Good Judgment Project, a small group of 'superforecasters' outperformed intelligence analysts with access to classified data. Their success came from active open-mindedness, constant updating of forecasts (perpetual beta), and use of base rates—reducing both bias and noise.
A MOMENT TO REFLECT
Put it to work.
Think of a recent decision you made with others. Were there differences of opinion? How much of that variability was due to noise rather than genuine disagreement about facts?
In your organization, how could you conduct a simple noise audit to quantify the variability in judgments (e.g., performance ratings, hiring decisions)?
When you receive a second opinion, do you consider it a sign of noise or a valuable correction? How can you tell the difference?
Often confused with
References
- Noise: A Flaw in Human Judgment — Daniel Kahneman, Olivier Sibony, Cass R. Sunstein (2021)
- Principles of Forecasting — J. Scott Armstrong (2001)
- The Good Judgment Project — Philip E. Tetlock, Barbara Mellers, Don Moore (2011)
- The Forensic Confirmation Bias — Itiel Dror (2012)
CONTINUE THE CONVERSATION
Source texts
- Noise A Flaw in Human Judgment (Daniel Kahneman, Olivier Sibony etc.)