Why Don’t Statisticians Detect Rare Events?
Under suitable sampling assumptions, estimation error often shrinks with the square root of sample size. Taleb emphasizes a different practical difficulty: a long record without a rare event may reveal little about its risk, while a single observation establishes that it can occur.
THE IDEA
Why Don’t Statisticians Detect Rare Events?
Statistics to the layman can appear rather complex, but the concept behind what is used today is so simple that my French mathematician friends call it deprecatorily “cuisine.” It is all based on one simple notion: the more information you have, the more you are confident about the outcome. Now the problem: by how much? A common statistical rule concerns estimation precision: under suitable sampling assumptions, an n times increase in sample size reduces standard error by a factor equal to the square root of n. Suppose I am drawing from an urn containing red and black balls. The precision of an estimate of the relative proportion after 20 drawings is not twice that after 10 drawings; under this rule, the improvement factor is the square root of 2 (that is, 1.41).
Rare events make short samples difficult to interpret. If an urn is dominated by black balls, many draws without a red ball do not establish the absence of red balls. Finding one, however, immediately establishes their presence. Skewness alone does not invalidate statistics, but familiar large-sample approximations may be poor when the event of interest has scarcely been observed. This asymmetry connects the example to the philosophical problem of induction discussed by David Hume and Karl Popper.
To assess an investor’s performance, we either need more astute, and less intuitive, techniques or we may have to limit our assessments to situations where our judgment is independent of the frequency of these events.
When to use it
Evaluating long-term investment performance where rare losses or gains matter.
Assessing the safety of a system (e.g., nuclear power) when failures are extremely rare.
Designing quality control for manufacturing processes with low defect rates.
FROM THEORY TO EVERYDAY LIFE
In practice
Urn with red and black balls
Drawing from an urn with mostly black balls and very few red ones. After many draws without finding a red, confidence about their absence grows slowly; but the first red found instantly confirms their presence.
Evaluating an investor’s performance
An investor may have a long track record of gains, which slowly increases confidence in their skill, but one major loss (a rare event) can dramatically alter the assessment—statistics struggle to capture this asymmetry.
A MOMENT TO REFLECT
Put it to work.
Why is it harder to confirm the absence of a rare event than to confirm its presence?
How does this asymmetry affect conclusions in fields like finance or medicine when analyzing rare outcomes?
In what ways might traditional statistical methods mislead us when dealing with rare, high-impact events?
CONTINUE THE CONVERSATION
Source texts
- Fooled by Randomness The Hidden Role of Chance in the Markets and in Life (Nassim Nicholas Taleb)