Expert Judgment
Expert judgment draws on people with relevant knowledge and experience. Their assessments can be combined mathematically, such as by averaging probabilities, or developed through discussion and repeated feedback, as in the Delphi method.
THE IDEA
Expert Judgment
Expert judgment draws on the assessments and forecasts of people with relevant knowledge and experience. Figure 7.11 depicts how evidence can be drawn from these opinions. In the 3E framework, expert judgment includes expertise in tasks and processes, such as insurance claims assessment, and projections, such as weather or economic forecasts.
FIGURE 7.11 Extracting Evidence from Expert Opinions.
Pooling combines expert assessments into a shared estimate, discussed in Chapter 6. Keep assessments of current conditions separate from forecasts of future outcomes. Where several topics are involved, use (1) review and reconciliation within each topic, followed by (2) combination across topics when a meaningful common measure is available. More experts can bring useful information, but adding people does not guarantee greater accuracy. Shared sources and assumptions can leave a large panel repeating the same error.
Linear opinion pooling takes a weighted or unweighted average of experts’ probabilities for the same outcome. Equal weights offer a starting point when there is no reliable basis for distinguishing forecasters. Different weights need a defensible reason, such as relevant forecasting performance; greater experience alone does not establish greater accuracy.
Behavioral aggregation uses interaction and feedback to develop judgments. Experts may meet in person or respond to questionnaires. In the Delphi method, responses are summarized and shared over successive rounds, allowing participants to revise their views. Agreement is not assured, and consensus by itself does not establish that the resulting estimate is right.
When to use it
Aggregating diagnoses from multiple medical specialists to determine a treatment plan.
Forecasting sales figures by pooling opinions from regional managers.
Combining risk assessments from engineers to evaluate the safety of a new product design.
Using expert panels to estimate project completion times in software development.
FROM THEORY TO EVERYDAY LIFE
In practice
Weather Forecasting
Meteorologists with varying expertise provide forecasts for a major storm. A linear opinion pooling approach is used, weighting each meteorologist's prediction based on their historical accuracy, to produce a consensus probability of the storm's path and intensity.
Economic Projections
A panel of economists is asked to forecast GDP growth for the next quarter. Using behavioral aggregation via the Delphi method, their initial estimates are collected, anonymized, and shared. After several rounds of feedback and revision, the group converges on a range of likely outcomes.
A MOMENT TO REFLECT
Put it to work.
How can you determine whether to use equal weighting or weighted pooling in a real-world expert judgment scenario?
What are the main risks of relying on a single expert's opinion, and how does pooling mitigate them?
In what situations might behavioral aggregation fail to produce convergence, and how would you address that?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)