Sample Size Neglect (Law of Small Numbers)
Small samples vary more from sample to sample than large ones drawn in the same way. Sample size neglect gives too little weight to that difference, encouraging confidence in striking results supported by few observations.
THE IDEA
Study l: Sample Size
We first investigate the relative impact of sample proportion (strength) and sample size (weight) in an experimental task involving the assessment of posterior probability. We presented 35 students with the following instructions:
Imagine that you are spinning a coin, and recording how often the coin lands heads and how often the coin lands tails. Unlike tossing, which (on average) yields an equal number of heads and tails, spinning a coin leads to a bias favoring one side or the other because of slight imperfections on the rim of the coin (and an uneven distribution of mass). Now imagine that you know that this bias is 3/5. It tends to land on one side 3 out of 5 times. But you do not know if this bias is in favor of heads or in favor of tails.
Subjects were then given various samples of evidence differing in sample size (from 3 to 33) and in the number of heads (from 2 to 19). All samples contained a majority of heads, and subjects were asked to estimate the probability (from .5 to 1) that the bias favored heads (H) rather than tails (T). Subjects received all 12 combinations of sample proportion and sample size shown in Table 13.1. They were offered a prize of $20 for the person whose judgments most closely matched the correct values.
This experiment demonstrates that people often overlook the importance of sample size (weight) and instead focus disproportionately on sample proportion (strength). The correct Bayesian posterior probability would give more weight to larger samples, but subjects' estimates were insufficiently sensitive to sample size, illustrating the sample size neglect bias.
When to use it
Interpreting poll results before an election: a small but dramatic shift in support may be misleading.
Evaluating scientific studies: news reports often highlight results from small studies without emphasizing their limitations.
Making business decisions from customer feedback: check how many people responded, how they were selected, and what is unknown about nonrespondents before generalizing from the comments.
FROM THEORY TO EVERYDAY LIFE
In practice
Coin Spin Experiment
Compare 8 heads out of 10 spins with 19 heads out of 33 spins. Both size and proportion change, so size alone does not tell us which provides stronger evidence that the coin favors heads. To isolate sample size neglect, compare judgments with the posterior probabilities implied by the stated coin model.
Medical Test Results
A doctor might overinterpret a small study showing a treatment works in 4 out of 5 patients, ignoring that a larger study with 40 out of 50 patients provides stronger evidence. This could lead to premature adoption of the treatment.
A MOMENT TO REFLECT
Put it to work.
When evaluating a claim based on data, how often do you check the sample size? Can you recall a time when you were swayed by a small sample?
Imagine you are deciding between two studies: one with 10 participants and 90% success, another with 1000 participants and 55% success. Which would you trust more? Why?
In what real-world situations might sample size neglect lead to serious errors in judgment?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Heuristics and Biases The Psychology of Intuitive Judgment (Thomas Gilovich, Dale Griffin, Daniel Kahneman)