Adjusting Intuitive Predictions for Regression to the Mean

Suppose you are predicting Julie’s college GPA using information about her early reading ability. The proposed correction has four steps.

1. Make your intuitive guess. Your intuition about Julie, or about any case about which you have information, is not worthless. Your fast, system 1 thinking easily places the information you have onto the scale of your prediction and produces a GPA score for Julie. This guess is the prediction you would make if the information you have were perfectly predictive. Write it down.

2. Look for the mean. Now, step back and forget what you know about Julie for a moment. What would you say about Julie’s GPA if you knew absolutely nothing about her? The answer, of course, is straightforward: in the absence of any information, your best guess of Julie’s GPA would have to be the mean GPA in her graduating class—say 3.2 for this illustration. This is the outside view: treat the case as an instance of a relevant class and begin with what is typical for that class.

3. Estimate the diagnostic value of the information you have. This is the difficult step, where you need to ask yourself, “What is the predictive value of the information I have?” The reason this question matters should be clear by now. If all you knew about Julie was her shoe size, you would correctly give this information zero weight and stick to the mean GPA prediction. If, on the other hand, you had the list of grades Julie has obtained in every subject, this information would be perfectly predictive of her GPA (which is their average). There are many shades of gray between these two extremes. If you had data about Julie’s exceptional intellectual achievements in high school, this information would be much more diagnostic than her reading age, but less than her college grades. Your task here is to quantify the diagnostic value of the data you have, expressed as a correlation with the outcome you are predicting. Except in rare cases, this number will have to be a back-of-the-envelope estimate. The source’s chapter 12 gives examples of modest predictive correlations. It treats values above .50 as unusual in the settings discussed, with many meaningful relationships around .20. For Julie’s reading-age example, it uses .20 as a plausible upper bound, rather than a measured value supplied here.

4. Adjust from the outside view in the direction of your intuitive guess, to an extent that reflects the diagnostic value of the information you have. The final step is a simple arithmetic combination of the three numbers you have now produced: you must adjust from the mean, in the direction of your intuitive guess, in proportion to the correlation you have estimated. This step simply extends the observation we have just made: if the correlation were 0, you would stick to the mean; if it were 1, you would disregard the mean and happily make a matching prediction. In Julie’s case, then, the best prediction you can make of GPA is one that lies no more than 20% of the way from the mean of the class in the direction of the intuitive estimate that her reading age suggested to you. This computation leads you to a prediction of about 3.3.

The same correction can be illustrated with a hiring prediction. Consider, for instance, a vice president of sales who is hiring a new salesperson and has just had an interview with an absolutely outstanding candidate. Based on this strong impression, the executive estimates that the candidate should book sales of $1 million in the first year on the job—twice the mean amount achieved by new hires during their first year on the job. How could the vice president make this estimate regressive? The calculation depends on the diagnostic value of the interview. How well does a recruiting interview predict on-the-job success in this case? Based on the evidence we have reviewed, a correlation of .40 is a very generous estimate. Accordingly, a regressive estimate of the new hire’s first-year sales would be, at most, $500K + ($1 million − $500K) ×.40 = $700K.

When the estimated predictive correlation is positive but imperfect, this correction moves the intuitive estimate toward the mean. It may predict fewer future wins for a champion who has won ten Grand Slam titles, or less growth for a successful start-up worth $1 billion, than a matching prediction would. That does not rule out exceptional future performance; it changes the central estimate.

This means that, in hindsight, corrected predictions will inevitably result in some highly visible failures. However, prediction is not done in hindsight. You should remember that outliers are, by definition, extremely rare. The opposite error is much more frequent: when we predict that outliers will remain outliers, they generally don’t, because of regression to the mean. When the reference class and predictive relationship are well chosen, regression-based predictions can reduce mean squared error relative to uncorrected matching predictions. A guessed correlation and an arbitrary intuitive estimate do not guarantee that improvement.