Orders of Magnitude Error

intermediate1 min readstatistics · economics · psychology

Estimates of rare-event probabilities can be highly sensitive to model assumptions. Taleb warns that an apparently small error can leave a risk estimate wrong by orders of magnitude.

THE IDEA

Orders of Magnitude

In The Black Swan, Nassim Taleb warns that errors in estimating extreme-event risk can be much larger than they appear. An estimate may be more than 10% wrong: it might be ten times wrong, or 100 times wrong, or 1,000 times wrong. A claim that an event occurs only every 1,000 years is therefore only as credible as the model and evidence behind it. This sensitivity can produce severe underestimation, although not every small error has that effect.

When to use it

Financial risk management: stress-testing portfolios for events far more frequent than historical models suggest.
Public health planning: preparing for pandemics with higher occurrence rates than typical projections.
Engineering: designing infrastructure with safety margins that account for potential underestimation of extreme weather events.

FROM THEORY TO EVERYDAY LIFE

In practice

Market crash underestimation

In a hypothetical example, a financial analyst models a crash as a 1-in-100-year event. Suppose the risk is actually 1-in-10 because the model omits important changes in the system. The difference illustrates an order-of-magnitude error; the figures are not measured crash probabilities.

Pandemic preparedness

Suppose a hypothetical planning model puts annual pandemic risk at 0.1%, while an alternative model puts it at 1%. The difference is an order of magnitude. These illustrative estimates show why assumptions matter; they do not establish actual pandemic risk or a spending recommendation.

✳

A MOMENT TO REFLECT

Put it to work.

When was the last time you were surprised by an event that seemed 'unthinkable'? Could that event have been more probable than you assumed?
How can you stress-test your plans against errors in probability estimation, not just bad luck?
In your field, what commonly used risk models might be built on false priors, and how could you test for orders-of-magnitude errors?