Tetlock's Studies on Expert Prediction and Bias
Philip Tetlock's research examines how experts make predictions and reason about historical counterfactuals, revealing a general tendency toward theory-driven reasoning and biases like overconfidence and belief perseverance, particularly among those favoring parsimony and explanatory closure.
THE IDEA
Expert Forecasts and Historical Counterfactuals
Tetlock examines political experts’ forecasts and explanations of the past. One set of studies tracks the reactions of experts to the apparent confirmation or disconfirmation of conditional forecasts of real-world events in real time. Key issues become: (1) the extent to which experts who “get it wrong” resort to various belief-system defenses; (2) the effectiveness of these defenses in helping experts who got it wrong to preserve confidence in their prior world views; (3) the rationality of these defenses.
The other set of studies examines retrospective reasoning: the tightness of the connection between experts’ conceptions of what would have happened in “counterfactual worlds” and their general ideological outlook. The key issue becomes the extent to which counterfactual reasoning about historical possibilities is theory-driven (predictable from abstract preconceptions) as opposed to data-driven (constrained by the peculiar “fact situation” of each episode).
In these studies of both prospective and retrospective reasoning, individual differences in cognitive style and in conceptual orientations are important moderator variables. Although there is a general tendency among our experts to rely on theory-driven modes of reasoning and to fall prey to theory-driven biases such as overconfidence and belief perseverance, these tendencies are systematically more pronounced among experts with strong preferences for parsimony and explanatory closure.
When to use it
Evaluating the accuracy and reasoning of political analysts’ forecasts
Improving decision-making under uncertainty by identifying and mitigating belief perseverance
Analyzing historical events to determine the role of ideological preconceptions in counterfactual reasoning
Designing training programs for experts to balance theory-driven and data-driven reasoning
FROM THEORY TO EVERYDAY LIFE
In practice
Reaction to Disconfirmation of Forecasts
Imagine an expert who confidently predicts that a country will not invade its neighbor. After an invasion, the expert cites unexpected circumstances to defend the forecast. Those circumstances may be relevant; belief perseverance becomes a concern if the expert dismisses contrary evidence without examining what the forecast missed.
Theory-Driven Counterfactual Reasoning
Imagine a political analyst who argues that avoiding a historical event would have produced an outcome favored by their ideology. The reasoning is theory-driven if the analyst relies on that preference while ignoring records that challenge the proposed chain of events.
A MOMENT TO REFLECT
Put it to work.
How might your own predictions be influenced by theory-driven biases?
Can you recall a time when you defended a wrong belief by citing counterfactuals?
What cognitive strategies could you adopt to reduce overconfidence in your forecasts?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)