Level Noise and Pattern Noise
Noise in judgments can be decomposed into level noise (systematic differences in average severity across judges) and pattern noise (idiosyncratic reactions to specific cases), both contributing to system noise.
THE IDEA
Decomposition of System Noise
System noise (the variability in judgments of the same case by different judges) can be broken down into two components: level noise and pattern noise.
Level Noise
Level noise refers to differences in the average severity of judgments across judges. In the sentencing study, judges varied widely in their average sentence length (standard deviation of 2.4 years). This reflects their personal tendencies—some are consistently harsh ("hanging judges") and others consistently lenient ("bleeding-heart judges"). Level noise is like a personality trait of the judge, unrelated to the specific case.
Pattern Noise
Pattern noise (also called judge × case interaction) captures the remaining variability after accounting for level noise. It reflects how judges respond differently to specific cases—a judge may be harsher on white-collar criminals but more lenient on accomplices, for example. In the sentencing study, pattern noise contributed approximately equally to system noise as level noise did. Pattern noise is pervasive in many judgment tasks (e.g., doctors deciding hospital admissions, hiring decisions, television show production).
The relationship is: [ \text{System Noise}^2 = \text{Level Noise}^2 + \text{Pattern Noise}^2 ]
The Sentencing Study
A study of 208 federal judges and 16 cases found that the average sentence was 7.0 years, with a standard deviation of 3.4 years across judgments of the same case. The mean absolute difference between two randomly chosen judges on the same case was 3.8 years—a large and consequential lottery for defendants.
Distinction from Bias
The text emphasizes that noise is distinct from bias. Bias refers to shared systematic errors (e.g., racial discrimination), while noise is the unwanted variability among judges. Both are sources of injustice.
When to use it
Judicial sentencing: Understanding and reducing unwanted variability among judges.
Performance evaluations: Level noise can arise from some managers being generous and others strict; pattern noise from differing reactions to specific employee traits.
Medical diagnoses: Different doctors may have different thresholds for recommending surgery (level noise) and different responses to patient demographics (pattern noise).
Hiring decisions: Interviewers may differ in overall stringency and in their preferences for certain candidate profiles.
FROM THEORY TO EVERYDAY LIFE
In practice
Sentencing Study Example
In a study with 208 federal judges and 16 cases (A–P), the average sentence across all cases was 7.0 years. For the average case, the standard deviation of sentences was 3.4 years. This means that if two judges were randomly picked, the average difference between their sentences for the same case would be 3.8 years—a huge disparity. Level noise (differences in judges' average severity) had a standard deviation of 2.4 years, and pattern noise (idiosyncratic reactions to cases) contributed the rest.
Real-World Analogy
Consider medical decisions: some doctors are more likely to hospitalize patients (level noise), and each doctor may have idiosyncratic reactions to specific symptoms (pattern noise). Both create unwanted variability in care.
A MOMENT TO REFLECT
Put it to work.
In what domains of your life have you observed level noise (consistent differences in judgment among people)? How might you measure it?
Can you think of a situation where pattern noise (idiosyncratic preferences) led to unfairness? How could it be reduced?
How might you distinguish between noise and bias in a decision-making process you are familiar with?
Often confused with
References
- Noise: A Flaw in Human Judgment — Daniel Kahneman, Olivier Sibony, Cass R. Sunstein (2021)
CONTINUE THE CONVERSATION
Source texts
- Noise A Flaw in Human Judgment (Daniel Kahneman, Olivier Sibony etc.)