Miscalibration in Expert Judgment
Systematic miscalibration is observed across domains of expert judgment, but the magnitude varies: experts with training in statistical modeling (e.g., meteorologists, economists) are less biased than predicted by the direct support account, while others (e.g., physicians, stockbrokers, sports commentators) align with it.
THE IDEA
Calibration concerns how well probability estimates match observed frequencies. The expert-judgment survey described in the source found systematic miscalibration across domains, consistent with qualitative predictions of the direct support account. Its magnitude varied: domains with more statistical-modeling training and technical assistance showed less bias. This association does not establish that every expert in those domains is well calibrated.
When to use it
Evaluating the reliability of expert testimony in legal or policy settings by considering the expert's domain and statistical training.
Designing decision-support systems that prompt users to provide probabilistic estimates and then offer calibration feedback.
Assessing an expert forecast by examining calibration records for comparable predictions in the relevant domain.
FROM THEORY TO EVERYDAY LIFE
In practice
Medical Diagnosis
A physician diagnosing a rare disease often exhibits overconfidence, giving probability estimates that are too extreme compared to actual frequencies, consistent with the direct support account.
Weather Forecasting
A well-calibrated forecaster’s 70% rain predictions correspond to rain about 70% of the time. The survey reports less miscalibration in meteorology, a field with statistical training and technical modeling support.
A MOMENT TO REFLECT
Put it to work.
In your own area of expertise, how well calibrated do you think your judgments are? What evidence supports this?
What specific training or feedback mechanisms could improve calibration in your field?
Why might experts in domains like medicine or stockbroking be more prone to miscalibration than those in meteorology or economics?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)