It Is Not the Experts’ Fault—The World Is Difficult
Kahneman argues that prediction has limits imposed by the environment. Confidence alone does not establish accuracy, and a predictor’s record and time horizon matter when deciding how much to trust a forecast.
THE IDEA
IT IS NOT THE EXPERTS’ FAULT—THE WORLD IS DIFFICULT
Errors of prediction are inevitable in an unpredictable world. High subjective confidence is not a reliable indicator of accuracy; low confidence could be more informative.
Short-term trends can be forecast, and behavior and achievements can be predicted with fair accuracy from previous behaviors and achievements. But we should not expect performance in officer training and in combat to be predictable from behavior on an obstacle field—behavior both on the test and in the real world is determined by many factors that are specific to the particular situation. Remove one highly assertive member from a group of eight candidates and everyone else’s personalities will appear to change. Let a sniper’s bullet move by a few centimeters and the performance of an officer will be transformed. I do not deny the validity of all tests—if a test predicts an important outcome with a validity of .20 or .30, the test should be used. But you should not expect more. You should expect little or nothing from Wall Street stock pickers who hope to be more accurate than the market in predicting the future of prices. And you should not expect much from pundits making long-term forecasts—although they may have valuable insights into the near future. The line that separates the possibly predictable future from the unpredictable distant future is yet to be drawn.
When to use it
Evaluating the track record of financial advisors or investment newsletters before trusting their stock picks.
Assessing political or economic pundits' forecasts by checking their past accuracy and the time horizon of their predictions.
Using psychological test scores (e.g., personality tests) to hire employees, while recognizing that their predictive validity is limited (e.g., validity = .20 -.30).
Deciding how much weight to give to expert opinions in situations where the environment is highly variable, such as predicting military performance or career success.
FROM THEORY TO EVERYDAY LIFE
In practice
Obstacle Course vs. Combat
Performance on an obstacle field is not a reliable predictor of performance in officer training or combat, because behavior in both settings is influenced by many situational factors. Removing one assertive member from a group changes group dynamics; a sniper's bullet shifting by centimeters can change an officer's outcome.
Wall Street Stock Pickers
Kahneman warns against expecting Wall Street stock pickers to forecast prices more accurately than the market. An analyst’s confidence does not establish that their predictions offer an advantage.
Pundits’ Long-Term Forecasts
Pundits making long-term predictions (e.g., economic trends, political outcomes) often have poor accuracy, though they may offer valuable insights for the near future. Their confidence is not a reliable cue to correctness.
A MOMENT TO REFLECT
Put it to work.
Think of a recent confident prediction. What evidence supported your confidence, and how predictable was the situation?
When you hear an expert make a long-term forecast, what cues might you look for to assess whether the forecast is actually informative?
When a prediction fails, what evidence would help distinguish a weak forecasting method from uncertainty that even a good method could not resolve?
CONTINUE THE CONVERSATION
Source texts
- Thinking, Fast and Slow (Daniel Kahneman)