Theory-Driven Defenses of Conditional Forecasts

THEORY-DRIVEN DEFENSES OF CONDITIONAL FORECASTS

Philip Tetlock examines how experts respond when political events contradict their forecasts.

Assessing the “truth value” of conditional forecasts initially looks more promising than assessing that of historical counterfactuals. When I “retrodict” that “if X had occurred, Y would have,” there is no way of decisively refuting me. But when I predict that “if X occurs, then Y will occur” and X indisputably occurs and Y equally indisputably does not, I am obliged to qualify or abandon my original claim. Closer analysis reveals, however, that judging conditional forecasts in world politics is typically far more controversial than this idealized example suggests. Experts have at least five logically defensible strategies for protecting conditional forecasts that run aground on troublesome evidence:

1. The antecedent was never adequately satisfied – in which case, the conditional forecast becomes an historical counterfactual with the passage of time. Thus, experts might insist that “if we had properly implemented deterrence or reassurance, we could have averted war” or “if real shock therapy had been practiced, we could have averted this nasty bout of hyperinflation.”

2. Although the specified antecedent was satisfied, key background conditions (covered by the ubiquitous ceteris paribus clause) took on unexpected values, thereby short-circuiting the otherwise reliably deterministic connection between cause and effect. Experts might defiantly declare that rapid privatization of state industries would have led to the predicted surge in economic growth but only if the government had exercised tight monetary control.

3. Although the predicted outcome did not occur, it “almost occurred” and would have but for some inherently unpredictable exogenous shock. Examples of close-call counterfactuals include “the hardliners almost overthrew Gorbachev” and “the EU almost disintegrated during the currency crises of 1992.”

4. Although the predicted outcome has not yet occurred, it eventually will and we just need to be more patient (hardline communists may yet prevail in Moscow and the EU still might fall apart).

5. Although the relevant preconditions were satisfied and the predicted outcome never came close to occurring and now never will, this failure should not be held against the framework that inspired the forecast. Forecasting exercises are best viewed as light-hearted diversions of no consequence because everyone knows, or else should know, that politics is inherently indeterminate (Jervis, 1992). As Henry Kissinger wryly conceded to Daniel Moynihan after the fragmentation of the Soviet Union, “Your crystal ball worked better than mine” (Moynihan, 1993, p. 23). On close inspection, of course, this concession concedes nothing.

Tetlock (1999) demonstrated that expert observers of world politics draw on all five strategies of minimizing the conceptual significance of unexpected events. However, tempting though it is to dismiss such intellectual maneuvering as transparently defensive post hocery, it is wrong to issue automatic indictments for the judgmental bias of belief perseverance. Each defense highlights a potentially valid objection to viewing disconfirmation of the conditional forecast as disconfirmation of the underlying theory. Fortunately, it is not necessary here to stake out a detailed position on the merits of specific variants of falsificationism in the philosophy of science (Suppe, 1973). It is sufficient to specify a straightforward procedural test of bias that, if failed, would convince even the most forgiving falsificationist that something is awry. This de minimis test poses the question: When evidence arises bearing on a conditional forecast, do judges who “got it wrong” display much greater interest than judges who “got it right” in questioning whether the antecedent was satisfied, in generating close-call counterfactuals, and in otherwise challenging the probity of the exercise? If so, we still cannot determine who is biased (incorrect forecasters may be too quick to complain about the test or correct forecasters may be too quick to accept it) but we can say that some bias – in the form of theory-driven selectivity of information processing – does exist.

The normative analysis need not, moreover, stop here. With appropriate data, it is possible to construct Bayesian benchmarks for gauging experts’ willingness to change their prior confidence in their beliefs in response to subsequent events. Key questions become: Do experts who “got it wrong” retain more confidence in their prior understanding of the underlying forces at work than they should, given their earlier assertions about the probability of particular events vis-à-vis particular hypotheses concerning underlying forces? Do experts who “got it right” take these confirmations too seriously, inflating their confidence in underlying forces at work even more than they should? And what cognitive mechanisms generate these effects? Is Bayesian underadjustment of “priors” among inaccurate experts mediated by reliance on the five belief-system defenses? These experts may see little reason to adjust their prior beliefs in accord with diagnosticity ratios provided long ago when they now believe that the ceteris paribus clause was not satisfied, the predicted consequences almost happened, and prediction is inherently impossible. By contrast, experts who made correct forecasts may “overadjust.” They may be proud of their accomplishment and disinclined to question it too closely by raising awkward questions about whether their forecasts turned out right for the wrong reasons.

Tetlock (1999) drew on a longitudinal research program described as covering predictions over the period from 1985 to 2010. The analysis reported here concerns specific earlier forecasts, rather than completed results for that entire period. It considers the predictions that experts offered for the 5-year futures of the Soviet Union in 1988, of South Africa in 1989, of Kazakhstan in 1992, of the European Monetary Union in 1991, and of Canada in 1992. It also examines the predictions for the U.S. presidential race of 1992 in a 4-month time frame and for the Persian Gulf crisis/war of 1990–1991 in a 1-year frame. All participants received some graduate training in social science or history, specialized in one or more of the regions under examination, and earned their livelihoods either as advanced graduate students in comparative politics and international relations (n = 58), post-doctoral fellows in international security and related fields (n = 25), university professors (n = 51), policy analysts in think-tanks (n = 17), intelligence analysts in government service (n = 39), or journalists in the employ of the mass media (n = 9).

The Technical Appendix in Tetlock (1999) presents the detailed instructions, assurances, and questions given to experts. Among other things, experts responded to a working-style questionnaire (designed, in part, to assess individual differences in preferences for parsimony and explanatory closure) and also rated a number of probabilities necessary for comparing their belief updating with the normative Bayesian model. Respondents were given detailed guidance on how to use the subjective-probability scales that ranged from 0 to 1.0. Experts rated their probabilities that: (1) their interpretation of the underlying forces operating in the scenario was correct (e.g., that their theory or model of the Soviet Union was correct) (prior probability of H); (2) the most influential alternative interpretation of the underlying forces was correct (e.g., that a different theory or model of the Soviet Union was correct) (prior probability of H∼); (3) various possible futures for [x] would occur, assuming that: (1) their own understanding of underlying forces shaping events was correct (conditional probability of outcome given H); and (2) the most influential alternative interpretation of the underlying forces shaping events was correct (conditional probability of outcome given H∼). The “possible futures” were designed to be logically exclusive and exhaustive. For example, in the Soviet case, the scenarios included a strengthening, a reduction, or no change in communist party control.

After the specified forecasting interval had elapsed, 78% of the original forecasters were successfully contacted and questioned again. After exploring experts’ ability to recall their original answers, experts were reminded of their original forecasts and confidence estimates. Experts rated their agreement with 9 propositions that could theoretically either cushion the disappointment of “disconfirmation” or deflate the euphoria of “confirmation.” Experts also answered “retrospective-probability” questions that assessed the degree to which they updated their prior probabilities in an approximately Bayesian fashion (although this was done in only 4 of the 7 forecasting domains examined here). The retrospective probabilities were compared with the posterior probabilities derived by combining the relevant prior probabilities and conditional probabilities collected at the pre-test (described in the paragraph above).

Across all seven domains, experts were only slightly more accurate than one would expect from chance (the proverbial dart-throwing, blindfolded chimpanzee who poses such tough competition for financial forecasters – Malkiel, 1990). Almost as many experts as not thought that the Soviet Communist Party would remain firmly in the saddle of power in 1993, that Canada was doomed by 1997, that neo-fascism would prevail in Pretoria by 1994, that the EMU would collapse by 1997, that Bush would be reelected in 1992, and that the Persian Gulf crisis would be resolved peacefully. Moreover, although experts only sporadically exceeded chance predictive accuracy, they regularly assigned subjective probabilities that exceeded the scaling anchors for “just guessing.” Most experts, especially those who valued parsimony and explanatory closure, thought they knew more than they did. Moreover, these margins of error were larger than those customarily observed in laboratory research on the calibration of confidence. Across all predictions elicited across domains, experts who assigned confidence estimates of 80% or higher were correct only 45% of the time, a hit rate not appreciably higher than that for experts who endorsed the “just-guessing” subjective probabilities of 0.50 and 0.33 (for 2 and 3 outcome scenarios, respectively). Expertise thus may not translate into predictive accuracy but it does translate into the ability to generate explanations for predictions that experts themselves find so compelling that the result is massive over-confidence.

Our principal interest is, however, in experts’ reactions to the apparent confirmation or disconfirmation of their forecasts. A clear pattern emerges: experts do not change their minds nearly to the degree they should if they were true Bayesians committed to using the diagnosticity ratios they provided at the outset of the forecasting exercises. Table 42.1 shows that, across all four domains with measures of prior probabilities and diagnosticity ratios at the original forecasts, and of posterior probabilities at the follow-up session, experts whose most likely scenarios materialized were pretty good Bayesians; they increased their confidence in their prior understanding of underlying forces that led them to assign varying probabilities to scenarios at the outset of the forecasting period. And they did so in approximately the right amounts. By contrast, experts whose most likely scenarios failed to materialize did not decrease confidence in the understanding of underlying forces that had led them to make their forecasts by as much as the benchmark prescribed. They held their ground, notwithstanding that the Bayesian formula prescribed that they should have lost a great deal of confidence in their prior understanding (and should, in some cases, have outright converted to what they once called “the most influential alternative view of the underlying forces shaping events”). These findings replicate the well known “conservatism bias” in probabilistic reasoning: confronted by the unexpected, people do not change their minds as much as a good Bayesian would. It also merits note that cognitive style emerges again as a predictor among inaccurate (but not accurate) forecasters. Inaccurate experts who value parsimony and explanatory closure were more prone to “conservatism” – a result that is consistent with past laboratory work on both “need for closure” and “integrative complexity.”

Table 42.1. Subjective Probabilities Experts Assigned to their Understanding of the Underlying Forces at the Beginning and End of Forecasting Periods

a Indicates a significant deviation of expert-assigned posterior probabilities from the posterior probabilities that Bayes Theorem stipulates experts should endorse if they were correctly multiplying their subjective prior odds by the diagnosticity ratios provided at original forecast.

b Indicates significant shift in subjective probability assigned to one’s understanding of underlying forces at beginning and end of forecasting (prior versus posterior).

Defenders of the rationality of our experts (especially of the experts who valued parsimony and closure) are not, however, logically compelled to submit to the looming negative verdict. They can rightly point out that, in the intervening periods, experts may have quite sensibly changed their minds about the diagnosticity of various events vis-à-vis various hypotheses concerning the “underlying forces shaping events.” Indeed, Table 42.2 itemizes the five belief-system defenses that experts invoked to neutralize the diagnosticity of unexpected events. These defensive strategies included: arguing that either the antecedent or ceteris paribus clauses underpinning the original forecasts were not satisfied, invoking close-call counterfactuals, claiming that the predicted event might still occur, and minimizing the conceptual significance of forecasting exercises. Table 42.2 shows that, in 26 of 35 cases, t-tests reveal that experts whose most likely scenarios did not occur showed significantly more enthusiasm (p *< .*05) for “defensive” cognitions than experts whose most likely scenarios did occur.