Clustering Illusion

Cancer Cures

When I return home from an Asian or European trip, my jet lag often causes me to rise at a very early hour. Occasionally, though very rarely, I switch on the TV set searching for market information. What strikes me in these morning explorations is the abundance of claims by the alternative medicine vendors of the curing power of their products. These no doubt are caused by the lower advertising rates at that time. To prove their claim, they present the convincing testimonial of someone who was cured thanks to their methods. For instance, I once saw a former throat cancer patient explaining how he was saved by a combination of vitamins for sale for the exceptionally low price of $14.95—in all likelihood he was sincere (although of course compensated for his account, perhaps with a lifetime supply of such medicine). In spite of our advances, people still believe in the existence of links between disease and cure based on such information, and there is no scientific evidence that can convince them more potently than a sincere and emotional testimonial. Such testimonial does not always come from the regular guy; statements by Nobel Prize winners (in the wrong discipline) could easily suffice. Linus Pauling, a Nobel Prize winner in chemistry, was said to believe in vitamin C’s medicinal properties, himself ingesting massive daily doses. With his bully pulpit, he contributed to the common belief in vitamin C’s curative properties. Many medical studies, unable to replicate Pauling’s claims, fell on deaf ears as it was difficult to undo the testimonial by a “Nobel Prize winner,” even if he was not qualified to discuss matters related to medicine.

Many of these claims have been harmless outside of the financial profits for these charlatans—but many cancer patients may have replaced the more scientifically investigated therapies, in favor of these methods, and died as a result of their neglecting more orthodox cures (again, the nonscientific methods are gathered under what is called “alternative medicine,” that is, unproven therapies, and the medical community has difficulties convincing the press that there is only one medicine and that alternative medicine is not medicine). The reader might wonder about my claims that the user of these products could be sincere, without it meaning that he was cured by the illusory treatment. The reason is something called “spontaneous remission,” in which a very small minority of cancer patients, for reasons that remain entirely speculative, experience an unexplained regression of their disease and appear to recover “miraculously.” The term does not by itself establish a mechanism. A recovery could have occurred without the advertised product; a testimonial alone cannot tell us what would have happened without it. Finally, these spontaneous remissions might not be so spontaneous; they might, at the bottom, have a cause that we are not yet sophisticated enough to detect.

The late astronomer Carl Sagan, a devoted promoter of scientific thinking and an obsessive enemy of nonscience, examined the cures from cancer that resulted from a visit to Lourdes in France, where people were healed by simple contact with the holy waters, and found out the interesting fact that, of the total cancer patients who visited the place, the cure rate was, if anything, lower than the statistical one for spontaneous remissions. It was lower than the average for those who did not go to Lourdes! Should a statistician infer here that cancer patients’ odds of surviving deteriorate after a visit to Lourdes? The visitors could differ from other patients in ways that affect recovery; those rates alone would not establish that the visit caused harm.

Professor Pearson Goes to Monte Carlo (Literally): Randomness Does Not Look Random!

At the beginning of the twentieth century, as we were starting to develop techniques to deal with the notion of random outcomes, several methods were designed to detect anomalies. Professor Karl Pearson (father of Egon Pearson of Neyman-Pearson fame, familiar to every person who sat in a statistics 101 class) developed influential goodness-of-fit tests. Normality and randomness are different properties, so testing one does not amount to testing the other. He examined millions of runs of what was called a Monte Carlo (the old name for a roulette wheel) during the month of July 1902. The account describes a highly significant departure from the tested model, with a probability under that model of less than one in a billion. What! The roulette wheel was not random! Professor Pearson was greatly surprised at the discovery. The interpretation depends on the tested model and the data. Physical imperfections can matter: With enough minutiae one would be able to uncover the nonrandomness somewhere (e.g., the wheel itself may not have been perfectly balanced or perhaps the spinning ball was not completely spherical). The practical question is which probability model adequately describes the process at the precision needed. A departure from that model is not a general proof that randomness is unattainable. Besides, a manager would question whether such nonrandomness can lead to any meaningful, profitable rules. If I need to gamble $1 on 10,000 runs and expect to make $1 for my efforts, then I would do much better in the part-time employment of a janitorial agency.

But the result bears another suspicious element. Of more practical relevance here is the following severe problem about nonrandomness. Even the fathers of statistical science forgot that a random series of runs need not exhibit a pattern to look random; as a matter of fact, data that is perfectly patternless would be extremely suspicious and appear to be man-made. A single random run is bound to exhibit some pattern—if one looks hard enough. Note that Professor Pearson was among the first scholars who were interested in creating artificial random data generators, tables one could use as inputs for various scientific and engineering simulations (the precursors of our Monte Carlo simulator). The problem is that they did not want these tables to exhibit any form of regularity. Yet real randomness does not look random!

This tendency to perceive meaningful patterns in random, unrelated events that occur close together is known as the clustering illusion. It is central to numerous gambling fallacies, including the notion that red is more or less likely to appear on a roulette table after a string of reds. I would further illustrate the point with the study of a phenomenon well-known as cancer clusters. Consider a square with 16 independently thrown darts, each equally likely to hit any location in it. If we divide the square into 16 smaller squares, it is expected that each smaller square will contain one dart on average—but only on average. There is a very small probability of having exactly 16 darts in 16 different squares. The average grid will have more than one dart in a few squares, and no dart at all in many squares. It will be an exceptionally rare incident that no (cancer) cluster would show on the grid. Now, transpose our grid with the darts in it to overlay a map of any region. A cluster might prompt suspicion of an environmental cause. Establishing that cause requires more than noticing an area with an above-average count; the random model shows why a cluster can also arise by chance.

The Dog That Did Not Bark: On Biases in Scientific Knowledge

By the same argument, science is marred by a pernicious survivorship bias, affecting the way research gets published. In a way that is similar to journalism, studies with null or statistically nonsignificant results may be less likely to reach publication. That may seem sensible, as newspapers do not have to have a screaming headline saying that nothing new is taking place (though the Bible was smart enough to declare ein chadash tachat hashemesh—“nothing new under the sun,” providing the information that things just do recur). The problem is that a finding of absence and an absence of findings get mixed together. There may be great information in the fact that nothing took place. As Sherlock Holmes noted in the Silver Blaze case—the curious thing was that the dog did not bark. More problematic, there are plenty of scientific results that are left out of publications because they are not statistically significant, but nevertheless provide information.