The Problem of the Single Study
The tendency to overvalue a single peer-reviewed study, ignoring the broader body of evidence, analogous to using an anecdotal counterexample to dismiss a well-established trend.
THE IDEA
The Problem of the Single Study
You’ve probably seen the chilling commercials of emphysema patients wheezing through a breathing tube telling us they wished they never started smoking. There’s little argument today that smoking causes cancer. But you’ll always find some know-it-all who harrumphs that their grandfather smoked three packs of Camels a day and lived till he was 97.
An analogous phenomenon goes on with journal articles—even one that’s peer reviewed. With two million articles churned out each year, you can almost always find one that provides backing for the wackiest claim someone wants to offer—even if this one-shot, stand-alone study flies in the face of a thousand others.
Which brings us back to chess. A Google search produces a study conducted by researchers at Iran’s Islamic Azad University. Groups of fifth-, eighth- and ninth-grade boys practiced chess for six months and afterward posted modest gains in their metacognitive ability as well as their math skill. Does this study put to rest, once and for all, the question of chess’s benefits? Or is it the peer-reviewed equivalent of that chain-smoking granddad?
When to use it
Evaluating health claims in the media (e.g., 'New study shows coffee causes cancer!')
Assessing policy recommendations based on limited research (e.g., educational interventions)
Making personal decisions (e.g., choosing a diet based on one study vs. meta-analyses)
Avoiding overconfidence in scientific findings when teaching or writing about a topic
FROM THEORY TO EVERYDAY LIFE
In practice
The Chain-Smoking Granddad
Despite overwhelming evidence that smoking causes cancer, someone might point to their grandfather who smoked heavily and lived to 97 as a counterexample, dismissing the epidemiological consensus.
The Chess Study
A single study from Iran's Islamic Azad University found that boys who practiced chess for six months showed modest gains in metacognition and math. Relying solely on this study to conclude chess is beneficial ignores the broader (and conflicting) body of research.
A MOMENT TO REFLECT
Put it to work.
When you encounter a news article citing a single study, what questions should you ask before accepting its conclusion?
Think of a time you used a single study to support an argument. How might you have been misled by ignoring the broader evidence?
How can you systematically evaluate the weight of multiple studies on a topic rather than fixating on one?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Verified How to Think Straight, Get Duped Less, and Make Better Decisions about What to Believe Online (Mike Caulfield, Sam Wineburg)