Dead Fish Don't Read Minds

DEAD FISH DON’T READ MINDS

Because here’s the thing: the Bible code kerfuffle is not the only occasion on which the standard statistical tool kit has been used to derive a result that sounds like magic. One of the hottest topics in medical science is functional neuroimaging, which promises to let scientists see your thoughts and feelings flickering across your synapses in real time through ever-more-accurate sensors. At the 2009 Organization for Human Brain Mapping conference in San Francisco, UC Santa Barbara neuroscientist Craig Bennett presented a poster called “Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction.” It takes a second to unwrap the jargony title, but when you do, the poster announces pretty clearly the unusual nature of its results. A dead fish, scanned in an fMRI device, was shown a series of photographs of human beings, and was found to have a surprisingly strong ability to correctly assess the emotions the people in the pictures displayed. That would be impressive enough for a dead person or a live fish—for a dead fish, it’s Nobel Prize material!

But the paper, of course, is a deadpan gag. (And a well-executed one: I especially like the “Methods” section, which starts “One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning. . . . Foam padding was placed within the head coil as a method of limiting salmon movement during the scan, but proved to be largely unnecessary as subject motion was exceptionally low.”) The joke, like all jokes, is a veiled attack: in this case, an attack on sloppy methodology among those neuroimaging researchers who make the mistake of ignoring the fundamental truth that improbable things happen a lot. Neuroscientists divvy up their fMRI scans into tens of thousands of small pieces, called voxels, each corresponding to a small region of the brain. When you scan a brain, even a cold dead fish brain, there’s a certain amount of random noise coming through on each voxel. It’s pretty unlikely that the noise will happen to spike exactly at the moment that you show the fish a snapshot of a person in emotional extremity. But the nervous system is a big place, with tens of thousands of voxels to choose from. The odds that one of those voxels provides data matching up well with the photos is pretty good. That’s exactly what Bennett and his collaborators found; in fact, they located two groups of voxels that did an excellent job empathizing with human emotion, one in the salmon’s medial brain cavity and the other in the upper spinal column. The point of Bennett’s paper is to warn that the standard methods of assessing results, the way we draw our thresholds between a real phenomenon and random static, come under dangerous pressure in this era of massive data sets, effortlessly obtained. We need to think very carefully about whether our standards for evidence are strict enough, if the empathetic salmon makes the cut.

The more chances you give yourself to be surprised, the higher your threshold for surprise had better be. If a random Internet stranger who eliminated all North American grains from his food intake reports that he dropped fifteen pounds and his eczema went away, you shouldn’t take that as powerful evidence in favor of the maize-free plan. Somebody’s selling a book about that plan, and thousands of people bought that book and tried it, and the odds are very good that, by chance alone, one among them will experience some weight loss and clear skin the next week. And that’s the guy who’s going to log in as saygoodbye2corn452 and post his excited testimonial, while the people for whom the diet failed stay silent.

The really surprising result of Bennett’s paper isn’t that one or two voxels in a dead fish passed a statistical test; it’s that a substantial proportion of the neuroimaging articles he surveyed didn’t use statistical safeguards (known as “multiple comparisons correction”) that take into account the ubiquity of the improbable. Without those corrections, scientists are at serious risk of running the Baltimore stockbroker con, not only on their colleagues but on themselves. Getting excited about the fish voxels that matched the photos and ignoring the rest is as potentially dangerous as getting excited about the successful series of stock newsletters while ignoring the many more editions that blew their calls and went in the trash.