Statistical Significance: The Detective, Not the Judge

THE DETECTIVE, NOT THE JUDGE

It’s clear that it’s wrong to use “p < .05” as a synonym for “true” and “p > .05” to mean “false.” Reductio ad unlikely, intuitively appealing as it is, just doesn’t work as a principle for inferring the scientific truth underlying the data.

But what’s the alternative? If you’ve ever run an experiment, you know scientific truth doesn’t pop out of the clouds blowing a flaming trumpet at you. Data is messy, and inference is hard.

One simple and popular strategy is to report confidence intervals in addition to p-values. This involves a slight widening of conceptual scope, asking us to consider not only the null hypothesis but a whole range of alternatives. Perhaps you operate an online store that sells artisanal pinking shears. Being a modern person (except insofar as you make artisanal pinking shears) you set up an A/B test, where half your users see the current version of your website (A) and half see a revamped version (B) with an animated pair of shears that does a little song and dance on top of the “Buy Now” button. And you find that purchases go up 10% with option B. Great! Now, if you’re a sophisticated type, you might be worried about whether this increase was merely a matter of random fluctuation—so you compute a p-value, finding that the chance of getting a result at least this good if the redesign weren’t actually working (i.e., if the null hypothesis were correct) is a mere 0.03.*

But why stop there? If I’m going to pay a college kid to superimpose dancing cutlery on all my pages, I want to know not only whether it works, but how well. Is the effect I saw consistent with the hypothesis that the redesign, in the long term, is really only improving my sales by 5%? Under that hypothesis, you might find that the probability of a deviation at least as large as the observed 10% growth might be 0.2. In other words, the hypothesis that the redesign is 5% better is not ruled out by the reductio ad unlikely. On the other hand, you might optimistically wonder whether you got unlucky, and the redesign was actually making your shears 25% more appealing. You compute another p-value and get 0.01, unlikely enough to induce you to throw out that hypothesis.

The confidence interval is the range of hypotheses that the reductio doesn’t demand that you trash, the ones that are reasonably consistent with the outcome you actually observed. In this case, the confidence interval might be the range from +3% to +17%. The fact that zero, the null hypothesis, is not included in the confidence interval is just to say that the results are statistically significant at the corresponding significance level.

But the confidence interval tells you a lot more. An interval of [+3%, +17%] licenses you to be confident that the effect is positive, but not that it’s particularly large. An interval of [+9%, +11%], on the other hand, suggests much more strongly that the effect is not only positive but sizable.

The confidence interval is also informative in cases where you don’t get a statistically significant result—that is, where the confidence interval contains zero. If the confidence interval is [−0.5%, 0.5%], then the data give you good evidence against effects larger than those bounds, rather than proving the intervention does exactly nothing. If the confidence interval is [−20%, 20%], the data leave substantial uncertainty about the size and direction of the effect. Those two outcomes look the same from the viewpoint of statistical significance, but have quite different implications for what you should do next.

The development of the confidence interval is generally ascribed to Jerzy Neyman, another giant of early statistics. Neyman was a Pole who, like Abraham Wald, started as a pure mathematician in Eastern Europe before taking up the then-new practice of mathematical statistics and moving to the West. In the late 1920s, Neyman began collaborating with Egon Pearson, who had inherited from his father Karl both an academic position in London and a bitter academic feud with R. A. Fisher. Fisher was a difficult type, always ready for a fight, about whom his own daughter said, “He grew up without developing a sensitivity to the ordinary humanity of his fellows.” In Neyman and Pearson he found opponents sharp enough to battle him for decades.

Their scientific differences are perhaps most starkly displayed in Neyman and Pearson’s approach to the problem of inference.* How to determine the truth from the evidence? Their startling response is to unask the question. For Neyman and Pearson, the purpose of statistics isn’t to tell us what to believe, but to tell us what to do. Statistics is about making decisions, not answering questions. A significance test is no more or less than a rule, which tells the people in charge whether to approve a drug, undertake a proposed economic reform, or tart up a website.

It sounds crazy at first to deny that the goal of science is to find out what’s true, but the Neyman-Pearson philosophy is not so far from reasoning we use in other spheres. What’s the purpose of a criminal trial? We might naively say it’s to find out whether the defendant actually committed the crime he’s on trial for. But that’s obviously wrong. There are rules of evidence, which forbid the jury from hearing testimony obtained improperly, even if it might help them accurately determine the defendant’s innocence or guilt. The purpose of a court is not truth, but justice. We have rules, the rules must be obeyed, and when we say that a defendant is “guilty” we mean, if we are careful about our words, not that he committed the crime he’s accused of, but that he was convicted fair and square according to those rules. Whatever rules we choose, we’re going to let some criminals go free and imprison some of the blameless. The less you do of the first, the more you’re likely to do of the second. So we try to design the rules in whatever way society thinks we best handle that fundamental trade-off.

For Neyman and Pearson, science is like the court. When a drug fails a significance test, we don’t say, “We are quite certain the drug didn’t work,” but merely, “The drug wasn’t shown to work.” And then dismiss it, just as we would a defendant whose presence at the crime scene couldn’t be established within reasonable doubt, even if every man and woman in the courthouse thinks he’s guilty as sin.

Fisher wanted none of this—for him, Neyman and Pearson stunk of pure mathematics, insisting on an austere rationalism at the expense of anything resembling scientific practice. Most judges wouldn’t have the stomach to let an obviously innocent defendant meet the hangman, even when the rules in the book require it. And most practicing scientists have no interest in following a rigid sequence of instructions, denying themselves the self-polluting satisfaction of forming an opinion about which hypotheses are actually true. In a 1951 letter to W. E. Hick, Fisher wrote:

I am a little sorry that you have been worrying yourself at all with that unnecessary portentous approach to tests of significance represented by the Neyman and Pearson critical regions, etc. In fact, I and my pupils through the world would never think of using them. If I am asked to give an explicit reason for this I should say they approach the problem entirely from the wrong end, i.e. not from the point of view of a research worker, with a basis of well grounded knowledge on which a very fluctuating population of conjectures and incoherent observations is continually under examination. What he needs is a confident answer to the question “Ought I to take notice of that?” This question can, of course, and for refinement of thought should, be framed as “Is this particular hypothesis overthrown, and if so at what level of significance, by this particular body of observations?” It can be put in this form unequivocally only because the genuine experimenter already has the answers to all the questions that the followers of Neyman and Pearson attempt, I think vainly, to answer by merely mathematical considerations.

But Fisher certainly understood that clearing the significance bar wasn’t the same thing as finding the truth. He envisions a richer, more iterated approach, writing in 1926: “A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.”

Not “succeeds once in giving,” but “rarely fails to give.” A statistically significant finding gives you a clue, suggesting a promising place to focus your research energy. The significance test is the detective, not the judge. You know how when you read an article about a breakthrough finding that this thing causes that thing, or that thing prevents the other thing, and at the end there’s always a banal sort of quote from a senior scientist not involved in the study intoning some very minor variant of “The finding is quite interesting, and suggests that more research in this direction is needed”? And how you don’t really even read that part because you think of it as an obligatory warning without content?

Here’s the thing—the reason scientists always say that is because it’s important and it’s true! The provocative and oh-so-statistically-significant finding isn’t the conclusion of the scientific process, but the bare beginning. If a result is novel and important, other scientists in other laboratories ought to test and retest the phenomenon and its variants, trying to figure out whether the result was a one-time fluke or whether it truly meets the Fisherian standard of “rarely fails.” That’s what scientists call replication; if an effect can’t be replicated, despite repeated trials, science backs apologetically away. The replication process is supposed to be science’s immune system, swarming over newly introduced objects and killing the ones that don’t belong.

That’s the ideal, at any rate. In practice, science is a bit immunosuppressed. Some experiments, of course, are hard to repeat. If your study measures a four-year-old’s ability to delay gratification and then relates these measurements with life outcomes thirty years later, you can’t just pop out a replication.

But even studies that could be replicated often aren’t. Every journal wants to publish a breakthrough finding, but who wants to publish the paper that does the same experiment a year later and gets the same result? Even worse, what happens to papers that carry out the same experiment and don’t find a significant result? For the system to work, those experiments need to be made public. Too often they end up in the file drawer instead.

But the culture is changing. Reformers with loud voices like Ioannidis and Simonsohn, who speak both to the scientific community and to the broader public, have generated a new sense of urgency about the danger of descent into large-scale haruspicy. In 2013, the Association for Psychological Science announced that they would start publishing a new genre of article, called Registered Replication Reports. These reports, aimed at reproducing the effects reported in widely cited studies, are treated differently from usual papers in a crucial way: the proposed experiment is accepted for publication before the study is carried out. If the outcomes support the initial finding, great news, but if not, they’re published anyway, so the whole community can know the full state of the evidence. Another consortium, the Many Labs project, revisits high-profile findings in psychology and attempts to replicate them in large multinational samples. In November 2013, psychologists were cheered when the first suite of Many Labs results came back, finding that 10 of the 13 studies addressed were successfully replicated.

In the end, of course, judgments must be made, and lines drawn. What, after all, does Fisher really mean by the “rarely” in “rarely fails”? If we assign an arbitrary numerical threshold (“an effect is real if it reaches statistical significance in more than 90% of experiments”) we may find ourselves in trouble again.

Fisher, at any rate, didn’t believe in a hard and fast rule that tells us what to do. He was a distruster of pure mathematical formalism. In 1956, near the end of his life, he wrote that “in fact no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas.”

Fisher’s appeal to “the light of the evidence” leaves room for scientific judgment about the experiment and the wider body of research.