Dependability of Probabilistic Evidence
Evidence drawn from data can fail because of errors in the data, analytical methods, or interpretation. Check each before relying on a statistical result.
THE IDEA
Dependability of Probabilistic Evidence
‘Chance favors the prepared mind’ wrote Louis Pasteur, the 19th-century biologist and chemist associated with vaccination, fermentation, and pasteurization. What prepares a mind to evaluate evidence changes as methods develop. Theoretical science gained ground around the 17th century; experiments and, later, large-scale data analysis expanded the available tools. For organizational decisions, the practical question is whether the data and analysis support the conclusion being drawn. Evidence-based practice requires that assessment in management as well as medicine.
An analysis can go wrong through (1) errors in the data, (2) unsuitable or incorrectly applied methods, or (3) mistaken interpretation. Data checks need to cover capture, retrieval, curation, and use. Checking a value also requires knowing what it means and what would make it implausible, which is difficult when documentation is poor. Even accurate data can produce misleading results if the method does not answer the decision-maker’s question. A correctly calculated result can then be misread.
Consider a hypothetical business school whose dean wants to estimate the earnings of alumni who graduated within the past five years. After repeated survey mailings, the director of alumni relations obtains employment and income data from about 70% of the eligible graduates. A few respondents report incomes several orders of magnitude above the next highest group. Including them produces a high mean income, which the director hopes will enhance the school’s reputation.
The high mean is not necessarily an error. If those incomes are accurate, they belong in an estimate of the respondents’ mean. But the mean may poorly describe what a typical graduate earns; reporting the median and the distribution would help readers see the difference. The school should verify unusual values rather than exclude them simply because they are large.
There is also a separate problem: the 30% who did not respond. If lower earners were less willing to report their incomes, even a perfectly calculated mean or median of respondents could overstate earnings across the eligible alumni. The dean needs to know both what the statistic measures and how well the respondents represent the group of interest.
When to use it
Evaluating the effectiveness of a new marketing campaign based on customer feedback data, ensuring representativeness and proper handling of extreme responses.
Assessing the average salary of graduates from a university to rank programs, being cautious of outliers and nonresponse bias.
Deciding whether to invest in a stock based on historical performance data, checking for data quality and appropriate analytical methods.
Interpreting patient outcome data in a clinical trial to determine treatment efficacy, considering potential data errors and methodological flaws.
FROM THEORY TO EVERYDAY LIFE
In practice
Business School Alumni Earnings Example
In a hypothetical survey, a business school collects earnings data from 70% of recent graduates. A few very high incomes raise the mean. Those values should be checked, and a median can help describe typical earnings. The 30% who did not respond create a separate concern: if nonrespondents earn less, statistics based on respondents may overstate alumni earnings.
A MOMENT TO REFLECT
Put it to work.
How might nonresponse bias affect the conclusions you draw from survey data in your own work?
How would you distinguish an erroneous outlier from a genuine extreme value, and how would each affect your analysis?
When would the mean answer your question, and when would a median or a view of the full distribution be more informative?
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)