Undecidability of Probability Estimation from Samples
Taleb describes a proposed result about estimating probabilities from samples without prior restrictions on the admissible distributions. The passage raises the problem of model assumptions but does not supply the proof or its precise conditions.
THE IDEA
An Undecidability Theorem
This problem of self-reference, published with Pilpel after The Black Swan, went unnoticed as such. So Raphael Douady and I re-expressed the philosophical problem mathematically, and it appears vastly more devastating in its practical implications than the Gödel problem.
Raphael is, among the people I know, perhaps the man with the greatest mathematical erudition—he may have more mathematical culture than anyone in modern times, except perhaps for his late father, Adrien Douady.
At the time of writing, we may have produced a formal proof using mathematics, and a branch of mathematics called “measure theory” that was used by the French to put rigor behind the mathematics of probability. The paper is provisionally called “Undecidability: On the inconsistency of estimating probabilities from a sample without binding a priori assumptions on the class of acceptable probabilities.”
When to use it
Examining the model assumptions behind an estimate of extreme market-event risk.
Examining which adverse-event risks a clinical trial sample can and cannot constrain.
Comparing assumptions behind models of earthquake recurrence.
Assessing how limited observations and assumptions about the data-generating process affect a component-failure estimate.
FROM THEORY TO EVERYDAY LIFE
In practice
Rare Event Prediction
Estimating a crash probability from market history requires assumptions about how that history relates to future outcomes and which probability models are admissible. The passage questions what can be inferred without such restrictions; it does not provide a worked estimate or theorem application.
Scientific Hypothesis Testing
When interpreting a sample, identify the distributional and sampling assumptions behind the inference. The proposed result cannot be used from this excerpt alone to declare all frequentist or Bayesian methods inconsistent.
A MOMENT TO REFLECT
Put it to work.
What assumptions connect your sample to the probability you want to estimate? Which would you need to check before applying the proposed result?
How would a risk decision change if several plausible probability models fit the available data?
Can you identify a real-world situation where not specifying a priori assumptions led to a false sense of certainty in probabilistic predictions?
Often confused with
References
- Undecidability: On the inconsistency of estimating probabilities from a sample without binding a priori assumptions on the class of acceptable probabilities — Nassim Nicholas Taleb and Raphael Douady (2010)
CONTINUE THE CONVERSATION
Source texts
- The Black Swan The Impact of the Highly Improbable With a new section On Robustness and Fragility (Nassim Nicholas Taleb)