Superforecasters: How to Be Wrong
Julia Galef describes Philip Tetlock’s research on forecasting and a group of unusually successful forecasters. She presents their willingness to recognize errors and revise beliefs as a practice worth learning.
THE IDEA
How to Be Wrong
Political scientist Philip Tetlock spent almost two decades measuring people’s ability to forecast global events. The results were disappointing—even so-called experts were barely able to do better than random chance. Or, as Tetlock famously put it, the average expert was “roughly as accurate as a dart-throwing chimpanzee.”1
Yet there were exceptions. A small subset of people turned out to have genuine skill at answering questions such as “Will the Muslim Brotherhood win the elections in Egypt?” or “Will there be a violent incident in the South China Sea in 2013 that kills at least one person?” Tetlock dubbed them the “superforecasters.”
In a forecasting tournament sponsored by the Intelligence Advanced Research Projects Activity (IARPA), a branch of the United States intelligence community, the superforecasters handily beat teams of professors from top universities by margins of up to 70 percent.2 In fact, the superforecasters did so much better than everyone else that IARPA dropped the other teams after only two years, despite originally planning to run the tournament for four.
What made the superforecasters super?
It wasn’t that they were smarter than everyone else. And it wasn’t that they had more knowledge or experience than everyone else. They were mostly amateurs, yet they outperformed even the CIA’s professional analysts, who had the advantage of years of experience, not to mention access to classified information on the topics about which they were making forecasts. The superforecasters, armed only with Google, beat the CIA by 30 percent.
What made the superforecasters so great at being right was that they were great at being wrong.
When to use it
Making quarterly sales forecasts for a business.
Predicting which political candidate will win an upcoming election.
Estimating the likelihood of a project finishing on time and under budget.
Assessing the probability of a scientific hypothesis being confirmed by future experiments.
FROM THEORY TO EVERYDAY LIFE
In practice
Forecasting Egyptian Elections
The source lists whether the Muslim Brotherhood would win elections in Egypt as a forecasting question. It does not give an individual forecast, its probability, or the result of that forecast.
South China Sea Incident
The source asks whether a violent incident killing at least one person would occur in the South China Sea in 2013. This illustrates a question with a specific event and time frame; the extract does not report how an individual forecaster answered it.
A MOMENT TO REFLECT
Put it to work.
How often do you update your beliefs when presented with new evidence that contradicts your initial prediction?
What would it look like to adopt a 'superforecaster' mindset in your own field or daily decision-making?
Can you recall a time when you were wrong about an important prediction? How did you react, and what did you learn?
Often confused with
References
- Superforecasting: The Art and Science of Prediction — Philip E. Tetlock and Dan Gardner (2015)
CONTINUE THE CONVERSATION
Source texts
- The Scout Mindset (Julia Galef)