Performance: Test Set ≠ Training Set
In the reported cross-validation task, Take the Best predicted homelessness more accurately than regression while using fewer cues. The comparison illustrates why fitting existing observations and predicting new ones require separate evaluation.
THE IDEA
Performance: Test Set ≠ Training Set
A strategy can “fit” known observations well and perform less accurately on new ones. How do the heuristics compare when the test set differs from the training set? This situation is a version of one-step learning and prediction. The data set is broken into two halves, with random assignment of cities to either one half or the other. The heuristics are allowed to use one half to build their models (calculate regression weights, get cue orders, determine cue direction); then they must make predictions on the other half, using the parameters estimated on the first half, and their accuracy is scored. This process is repeated 1,000 times, with 1,000 random ways of breaking the data into two halves in order to average out any particularly helpful or hurtful ways of halving the data.
Training might not seem to affect Dawes’ Rule and Minimalist, but, in fact, it does. Both strategies use the first half of the data set to estimate the direction of the cue (whether a higher or a lower cue value signals a higher criterion value). When the test set was different from the training set, the performance of Minimalist dropped from 61% to 56% correct, and that of Dawes’ Rule from 66% to 58% (Table 31.2). Take the Best must learn more than merely the direction of the cues; it must also learn the order of the cue validities. With this slight additional knowledge, Take the Best scores 63% correct, down from 69%. Regression must learn not only the direction of the cues, but also their interrelationship in order to determine the best linear weighting scheme. Despite all this knowledge, regression’s performance falls more than that of Take the Best: Whereas regression scored 70% correct when it had to merely fit the data, it scores only 61% correct in the cross-validated case, falling to second place.
In summary, when heuristics built their models on half of the data and inferred on the other half, Take the Best was the most accurate strategy for predicting homelessness, followed closely by regression. This seems counterintuitive because Take the Best looks up only 2.4 of the 6 cues and is simpler computationally.
Profile Memorization cannot supply predictions for cue profiles that appear in the test half but were absent from the training half.
When to use it
Selecting between a simple rule-of-thumb and a complex statistical model for business forecasting when only limited data is available.
Evaluating the generalizability of medical diagnosis rules across different patient populations.
Deciding whether to rely on a few key cues or all available cues when making predictions in a new environment.
FROM THEORY TO EVERYDAY LIFE
In practice
Predicting homelessness with heuristics
When trained on half of the data and tested on the other half, Take the Best achieved 63% accuracy, outperforming regression (61%) despite using only 2.4 of 6 cues. Minimalist dropped to 56% and Dawes' Rule to 58%.
Accuracy drop from training to test
Regression's accuracy fell from 70% (fitting) to 61% (cross-validated), while Take the Best fell from 69% to 63%, illustrating that simpler heuristics can generalize better.
A MOMENT TO REFLECT
Put it to work.
Why might a simple heuristic like Take the Best outperform a complex regression model when making predictions on new data?
How does cross-validation help in assessing the true predictive power of a decision strategy?
What does the decrease in accuracy from training to test set reveal about the tendency of complex models to overfit?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)