Broken Leg Problem
A broken leg illustrates a rare event omitted from a statistical model that could justify changing its prediction. The excerpt also considers other possible exceptions, while reporting that clinicians’ discretionary overrides often reduced overall predictive accuracy in the studies discussed.
THE IDEA
Possible Exceptions
If fair comparisons consistently favor the actuarial method, one may ask whether there are circumstances where the clinical judge might beat the actuary. Three hypothesized circumstances are:
- Theory-mediated judgments: Judgments based on theoretical understanding that are difficult to duplicate by statistical frequencies alone. This is successful only when the judge has access to major causal factors, accurate measurement, and a well-corroborated theory—rare in social sciences.
- Rare events (the broken leg problem): Events not included in the actuarial formula due to infrequency can countervail the actuarial conclusion. The classic example is a highly accurate formula predicting weekly movie attendance, which should be overridden if the person is in a cast with a broken leg.
- Configural relations: Complex patterns among predictive variables. Humans have unique observational capacities (e.g., recognizing facial expressions) but not necessarily unique predictive integration skills; these observations can often be coded quantitatively for actuarial analysis.
Despite these possibilities, research consistently finds that:
- Even with an information edge, clinicians fail to surpass actuarial methods (Sawyer, 1966; Wiggins, 1981).
- The broken leg possibility has been studied by allowing clinicians to override actuarial conclusions; overall accuracy is higher when they rely uniformly on actuarial conclusions (Goldberg, 1968; Sawyer, 1966).
- Clinicians identify too many “exceptions,” correctly modifying fewer than incorrectly modifying.
- Formal inclusion of clinician input into the actuarial equation does not enhance accuracy or utility, and informal adjustments can do more harm than good (Sawyer, 1966).
- Unique observational capacities (e.g., pattern recognition) do not guarantee unique predictive capacity; the skilled observer should step aside after observation, leaving interpretation to the actuarial method (Einhorn, 1972).
When to use it
In evaluating a medical risk model, examine whether an omitted condition materially changes the prediction and whether proposed overrides improve accuracy.
When a hiring score omits a relevant skill, assess that skill explicitly and review how exceptions affect the selection process.
When reviewing a prediction tool, record the reasons for overrides and compare their outcomes with the model’s predictions.
When a market event falls outside a model’s inputs, assess how it affects the model’s assumptions before substituting a discretionary forecast.
FROM THEORY TO EVERYDAY LIFE
In practice
Movie Attendance Prediction
An actuarial formula accurately predicts a person's weekly movie attendance. However, the person has recently suffered a broken leg and is in a cast. The rare event (broken leg) should lead the clinician to override the actuarial conclusion, illustrating the broken leg problem.
Medical Diagnosis Override
A readmission model omits a patient’s unusual condition. A clinician considers whether it changes the prediction. The question is whether the additional information improves accuracy, not merely whether the case feels exceptional.
A MOMENT TO REFLECT
Put it to work.
Think of a time when you relied on a statistical model but then overrode it based on a rare event. What was the outcome?
Why might clinicians be overconfident in their ability to identify exceptions to actuarial rules?
Under what conditions might theory-mediated judgment be more accurate than empirical frequencies?
Often confused with
References
- Clinical vs. Statistical Prediction: A Theoretical Analysis — Meehl, P. E. (1954)
- When Shall We Use Our Heads Instead of the Formula? — Meehl, P. E. (1957)
- A Comparison of Clinicians with Statistical Methods — Meehl, P. E. (1959)
- Clinical vs. Statistical Prediction: New Insights — Meehl, P. E. (1976)
- Measurement and Prediction, Clinical and Statistical — Sawyer, J. (1966)
- Clinical and Statistical Prediction: A Decade of Research — Wiggins, J. S. (1981)
- Clinical vs. Actuarial Prediction in Medicine — Martin, G. E., Apostolakos, P. C., & Roazen, H. (1960)
- Simple Models or Simple Processes? Some Research on Clinical and Actuarial Prediction — Goldberg, L. R. (1968)
- Expert Measurement and Mechanical Combination — Einhorn, H. J. (1972)
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)