Noise (Unwanted Variability in Judgment)
Noise is unwanted variation in judgments of the same case. It can arise between professionals or when the same person assesses a case on different occasions. Comparing independent judgments makes that variation visible.
THE IDEA
Noise: Unwanted Variability in Judgment
Noise is unwanted variation in judgments that should agree. Bias is a systematic departure from the right answer; noise is the scatter around an average judgment. A group can have both.
System noise describes variation across a judgment system. Some differences are stable: one judge may generally be more lenient, or respond differently to particular kinds of cases. Occasion noise reflects changes in the same person’s judgment at different times.
Doctors, underwriters, interviewers, forecasters, and forensic examiners can disagree even when assessing the same information. The source reports asylum grant rates ranging from 5% for one judge to 88% for another. Differences in outcomes are a reason to investigate, while an audit using shared cases helps separate judgment variation from differences in the cases assigned.
A noise audit asks professionals to assess the same cases independently. At an insurance company, the reported median difference between underwriters’ quotes was 55%, much larger than leaders expected. People had mistaken familiarity with a shared profession for agreement on particular cases.
For repeated numerical judgments, standard deviation measures scatter. Under a squared-error measure, bias and noise both contribute to average error. Reducing either can improve accuracy, but enforcing agreement around a wrong answer leaves bias in place.
Structured interviews, separate ratings of relevant factors, shared criteria, and aggregation of independent judgments can reduce unwanted variation. A noise audit helps identify where such changes are needed. These methods can also inform a one-off decision, even when the exact choice will not recur.
When to use it
Criminal sentencing: Use guidelines to reduce disparities among judges.
Insurance pricing: Conduct noise audits and implement structured decision processes to reduce variability in premium quotes.
Medical diagnosis: Use second opinions and standardized diagnostic criteria to reduce noise.
Hiring decisions: Use structured interviews and scoring rubrics to reduce variability among interviewers.
FROM THEORY TO EVERYDAY LIFE
In practice
Medical Diagnosis
Different doctors give different diagnoses for the same patient, leading to inconsistent treatment. For example, in reading X-rays, noise is high, and in psychiatry, subjective judgment leads to even more variability.
Criminal Sentencing
In the 1970s, Judge Frankel highlighted that similar defendants received vastly different sentences depending on the judge. Studies showed sentences for identical cases ranged from probation to years in prison.
Insurance Underwriting
A noise audit at an insurance company found that the median difference between underwriters' premium quotes for the same risk was 55%, costing the company hundreds of millions annually.
Asylum Decisions
A study of immigration courts found that one judge admitted 5% of applicants while another admitted 88%, leading to the title 'Refugee Roulette.' The decision often depends on the judge assigned.
Forecasting
Professional forecasters disagree widely on sales projections, unemployment rates, and bankruptcy risks. Even the same software developers, estimating completion time on different days, differed by 71% on average.
A MOMENT TO REFLECT
Put it to work.
Think of a recent decision in your organization. How much of the variability was due to noise rather than bias? How could you measure it?
Consider a time when two experts disagreed on a complex issue. What factors other than knowledge might have caused the disagreement?
If you were to conduct a noise audit in your field, what judgments would you test and how would you design the study?
How does noise affect fairness in systems like hiring, grading, or medical diagnosis? Can you identify an example of unfairness caused by noise?
Often confused with
References
- Noise: A Flaw in Human Judgment — Daniel Kahneman, Olivier Sibony, Cass R. Sunstein (2021)
- Refugee Roulette: Disparities in Asylum Adjudication — Jaya Ramji-Nogales, Andrew I. Schoenholtz, Philip G. Schrag (2007)
- A Study of Sentencing Disparity — Marvin Frankel (1973)
- Sentencing Disparity and the Federal Sentencing Guidelines — William Austin, Thomas Williams (1977)
- The Effect of Mandatory vs. Advisory Guidelines on Inter-Judge Sentencing Disparity — Crystal Yang (2015)
CONTINUE THE CONVERSATION
Source texts
- Noise A Flaw in Human Judgment (Daniel Kahneman, Olivier Sibony etc.)