THE LAW OF SMALL NUMBERS

My collaboration with Amos Tversky in the early 1970s began with a discussion of the claim that people who have had no training in statistics are good “intuitive statisticians.” He told my seminar about researchers at the University of Michigan who were generally optimistic about intuitive statistics. I had strong feelings about that claim, which I took personally: I had recently discovered that I was not a good intuitive statistician, and I did not believe that I was worse than others.

For a research psychologist, sampling variation is not a curiosity; it is a nuisance and a costly obstacle, which turns the undertaking of every research project into a gamble. Suppose you wish to confirm the hypothesis that the average six-year-old girl has a larger vocabulary than the average boy of the same age, and that the hypothesis is true in the population. Girls and boys vary a great deal, however, and by the luck of the draw you could select a sample in which the difference is inconclusive, or even one in which boys actually score higher. Using a sufficiently large sample is the only way to reduce the risk. Researchers who pick too small a sample leave themselves at the mercy of sampling luck.

The risk of error can be estimated for any given sample size by a fairly simple procedure. Traditionally, however, psychologists do not use calculations to decide on a sample size. They use their judgment, which is commonly flawed. An article I had read shortly before the debate with Amos demonstrated that psychologists commonly chose samples so small that they exposed themselves to a 50% risk of failing to confirm their true hypotheses. A plausible explanation was that psychologists’ decisions about sample size reflected prevalent intuitive misconceptions of the extent of sampling variation.

The article shocked me, because it explained some troubles I had had in my own research. Like most research psychologists, I had routinely chosen samples that were too small and had often obtained results that made no sense. Now I knew why: the odd results were actually artifacts of my research method. My mistake was particularly embarrassing because I taught statistics and knew how to compute the sample size that would reduce the risk of failure to an acceptable level. But I had never chosen a sample size by computation. I had trusted tradition and my intuition. When Amos visited the seminar, I had already reached the conclusion that my intuitions were deficient, and we quickly agreed that the Michigan optimists were wrong.

Amos and I set out to examine whether I was the only fool or a member of a majority of fools, by testing whether researchers selected for mathematical expertise would make similar mistakes. We developed a questionnaire that described realistic research situations, including replications of successful experiments. It asked the researchers to choose sample sizes, to assess the risks of failure to which their decisions exposed them, and to provide advice to hypothetical graduate students. Amos collected the responses of a group of sophisticated participants (including authors of two statistical textbooks) at a meeting of the Society of Mathematical Psychology. The results were straightforward: I was not the only fool. Every one of the mistakes I had made was shared by a large majority of our respondents. It was evident that even the experts paid insufficient attention to sample size.

Amos and I called our first joint article “Belief in the Law of Small Numbers.” We explained, tongue-in-cheek, that “intuitions about random sampling appear to satisfy the law of small numbers, which asserts that the law of large numbers applies to small numbers as well.” We also included a strongly worded recommendation that researchers regard their “statistical intuitions with proper suspicion and replace impression formation by computation whenever possible.”

The tendency to underweight sample size is not confined to researchers. It can be illustrated with a simple example: “In a telephone poll of 300 seniors, 60% support the president.” If you had to summarize that sentence in three words, you would likely choose “elderly support president.” The omitted details—that it was a telephone poll with a sample of 300—attract little attention. Unless the sample size is absurdly small or large, people do not react very differently to a sample of 150 than to one of 3,000. This is the core of the insensitivity: we focus on the story, not on the reliability of its source.

The message about the poll contains information of two kinds: the story and the source of the story. Naturally, we focus on the story rather than on the reliability of the results. Only when the source is obviously unreliable—for instance, a partisan group running a flawed and biased poll—do we reject its findings outright. But in less extreme cases, System 1 does not naturally distinguish between degrees of belief. It suppresses ambiguity and spontaneously constructs stories that are as coherent as possible. Unless a message is immediately negated, its associations spread as if it were true. System 2 is capable of doubt, because it can hold incompatible possibilities in mind at the same time. Yet sustaining doubt is harder work than sliding into certainty.

In this light, the law of small numbers is a specific manifestation of a broader bias that favors certainty over doubt. That bias is part of how System 1 operates: it runs ahead of the facts, building a rich and coherent image from scraps of evidence. The exaggerated faith in what a few observations can tell us is closely related to the halo effect—the sense that we know and understand a person about whom we actually know very little. When we overestimate the reliability of small samples, we are, in effect, jumping to conclusions. System 1, as a machine for jumping to conclusions, acts as if it believes in the law of small numbers, producing a representation of reality that makes too much sense.