Problem Scoping
Problem scoping makes expert-judgment questions precise enough for respondents to interpret them consistently. Pretesting, clear terms, separate questions for separate issues, and explicit time and location boundaries help.
THE IDEA
Problem Scoping
Expert judgments are easier to compare when respondents understand the question in the same way. Survey-design principles offer practical guidance:
- Plan to pretest. Test questions before collecting expert judgments. People outside the field can help identify awkward wording, while specialists can check the meaning of technical terms.
- Avoid vague or imprecise terms. In an expert survey, a clearly defined technical term may be more useful than everyday wording that permits several interpretations.
- Avoid complex sentences. Use short, direct questions. For example, ‘what is the probability of recession in 2019?’ is clearer than a sentence with several conditions and clauses.
- Avoid double-barreled questions. Ask about one issue at a time. A single question about earthquake likelihood and impact may cause some respondents to answer about probability and others about damage, producing inconsistent frames of reference.
- Provide appropriate frames of reference. Specify the location and period when they affect the answer: for example, an earthquake in San Francisco over the next 10 years.
- Avoid leading, emotional or evocative language. Describe the proposal being assessed clearly. Asking about immigration reforms proposed by President Trump may draw reactions to the politician rather than to the policy’s details.
When to use it
Designing surveys for expert forecasts in economics, such as predicting GDP growth or inflation rates.
Eliciting risk assessments from engineers about structural integrity of a building.
Formulating questions for a panel of medical experts regarding treatment outcomes.
Creating structured interview questions for expert witnesses in legal proceedings.
FROM THEORY TO EVERYDAY LIFE
In practice
Earthquake Risk Assessment Question
When asking experts to assess earthquake risk, avoid double-barreled questions that combine likelihood and impact. Instead, separate them: 'What is the probability of a magnitude 7+ earthquake in San Francisco in the next 10 years?' and 'What would be the expected economic damage from such an event?'
Immigration Policy Question
Asking about 'immigration reforms proposed by President Trump' may evoke emotional responses based on feelings about the president rather than the policy. A neutral framing like 'What are the likely effects of proposed immigration reforms on labor markets?' reduces bias.
A MOMENT TO REFLECT
Put it to work.
Think of a recent expert consultation you conducted or observed. How could the questions have been better scoped to ensure uniform interpretation?
When formulating a question for experts, what steps can you take to pretest for interpretational uniformity without requiring deep subject knowledge?
Identify a double-barreled question you have encountered. How would you rephrase it into two clear, single-focus questions?
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)