Nonrepresentative Samples
A cognitive bias where conclusions are drawn from samples that do not accurately represent the broader population, leading to flawed generalizations and decisions.
THE IDEA
Nonrepresentative Samples
Industry benchmarks and best practices discussed in the previous section are largely products of qualitative and observational research – for example, retailers and other direct-to-consumer business organizations often utilize the Pareto principle 4 based ‘80/20 rule’, according to which 80% of revenue can be attributed to approximately 20% of customers. That generalized insight is a ‘rule of thumb’, meaning it is based on practical experience rather than specific empirical insights (it should be noted that the original research published by V. Pareto was based on empirical evidence showing that roughly 80% of land in Italy, at that time, was owned by 20% of the population). Rules of thumb, such as the 80/20 rule, can serve as general guides, but their fit needs to be checked against the population and decision at hand. In contrast to those and other external-to-the-organization standards, businesses can also develop their own, internal-to-the-organization rules by exploring the available data.
When to use it
When conducting market research, design surveys to capture diverse demographics to avoid nonrepresentative samples.
In A/B testing, ensure that the test and control groups are randomly selected and representative of the user base.
When evaluating employee performance, gather feedback from multiple sources (peers, managers, subordinates) rather than just one perspective.
In scientific studies, use stratified sampling techniques to reflect the population's characteristics.
FROM THEORY TO EVERYDAY LIFE
In practice
Customer Satisfaction Survey Bias
A company surveys only customers who filed complaints, concluding overall dissatisfaction, but ignores satisfied customers who did not complain. This nonrepresentative sample leads to an inaccurate picture of customer sentiment.
Clinical Trial Demographics
A drug trial enrolls only young, healthy participants. Its results may not establish the benefits and harms for older people or those with other conditions. The sampling limits what can be inferred; it does not itself cause side effects.
A MOMENT TO REFLECT
Put it to work.
Can you recall a time when you relied on a limited or biased sample to make a decision? How might that have skewed your conclusion?
How can you ensure that future data you collect or rely on is representative of the entire population you care about?
What rule of thumb or common wisdom might be based on nonrepresentative samples, and how could you test its validity?
Often confused with
References
- Original research on land ownership in Italy — V. Pareto
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)