System Noise and Its Components: Level Noise, Pattern Noise, and Occasion Noise
System noise is unwanted variation in judgments of the same case. Level noise reflects differences in judges’ average responses; pattern noise reflects case-specific differences, including stable personal patterns and transient occasion effects. Separating them helps an organization decide what to measure and change.
THE IDEA
System Noise and Its Components
System noise is the unwanted variability in judgments made by different people or by the same person at different times when evaluating the same case. The following distinctions separate average differences, case-specific responses and transient effects:
Level Noise
Level noise refers to differences in the average level of judgments across different judges. For example, in a judicial sentencing study, some judges were consistently more severe than others, regardless of the case details.
Pattern Noise
Pattern noise is the variability in how judges respond to specific cases. Two judges might agree on average but disagree on which defendants deserve harsher or more lenient treatment. Stable pattern noise captures a judge’s consistent reactions to particular combinations of case features. Occasion noise adds transient variation to those responses.
Occasion Noise
Occasion noise is within-person variability due to transient factors like mood, fatigue, weather, or the order of cases. Even the same judge can give different verdicts on different days because of these irrelevant influences. For instance, judges are more likely to grant asylum if they are in a good mood, and physicians prescribe more opioids at the end of a long day.
How Groups Amplify Noise
Groups can amplify noise through social influence, informational cascades, and group polarization. The music download experiment by Salganik showed that songs' success depended heavily on early popularity, creating wide disparities across identical groups. Similarly, informational cascades occur when people follow early speakers, and group polarization pushes groups to more extreme positions.
Judgments vs. Models
Clinical judgments (human intuition) are often outperformed by simple mechanical models. Paul Meehl's 1954 book compared clinical predictions with statistical formulas and documented the formulas’ advantages. Lewis Goldberg later showed that a linear model fitted to a judge’s assessments could outperform the judge by applying the same weights consistently. Even some models with randomly selected weights performed well in the comparisons discussed in the book.
The Crowd Within
Occasion noise can be reduced by averaging two estimates from the same person (the "crowd within"). Dialectical bootstrapping involves making a second estimate after considering different assumptions. Averaging can help when the estimates contain partly independent errors; repeating the same reasoning may add little.
Practical Implications
A noise audit compares independent judgments of the same cases. Its findings can guide changes such as structured assessments, averaging independent judgments or testing a consistent decision rule against current practice.
When to use it
A hiring committee wants to reduce noise in candidate evaluations. They can implement structured interviews with scoring rubrics and average independent assessments before discussion.
A doctor diagnosing patients should be aware that time of day and fatigue may affect accuracy. Using decision support tools can help reduce occasion noise.
A business team making strategic decisions should avoid group polarization by encouraging diverse opinions and using a 'devil's advocate' role.
A judge or parole board can use statistical risk assessment tools to complement their judgment and reduce disparity.
FROM THEORY TO EVERYDAY LIFE
In practice
Judicial Sentencing Disparity
In a noise audit of judges, level noise caused different judges to give systematically different average sentences, while pattern noise led judges to disagree on which defendants deserved harsher punishment. Occasion noise meant that the same judge might give different sentences on different days due to mood or fatigue.
Music Download Experiment
In Salganik's study, social influence created noise across groups: songs that became popular early in one group succeeded, while identical songs failed in other groups due to random early popularity.
Crowd Within Application
Vul and Pashler found that averaging two guesses from the same person (e.g., estimating the percentage of US airports) improves accuracy. Waiting weeks between guesses or using dialectical bootstrapping further enhances the effect.
A MOMENT TO REFLECT
Put it to work.
Think of a recent professional decision you made. How might your mood or fatigue have influenced it? What could you do to reduce occasion noise?
Consider a group decision you were part of. Did early speakers influence the outcome? How could the group have made a more independent judgment?
If you were to replace your own judgment with a simple rule (like a checklist), would the rule likely be more accurate? Why or why not?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Noise A Flaw in Human Judgment (Daniel Kahneman, Olivier Sibony etc.)