Low Statistical Power and the P-Value Problem
When a study has low power for a small effect, results passing p < .05 can disproportionately reflect chance or exaggerated effect estimates. Low power alone does not determine whether a particular finding is false.
THE IDEA
Doctor, It Hurts When I P
A study with low power for a small genetic effect is unlikely to detect it reliably. If attention is restricted to results passing p < .05, the selected estimates may substantially overstate the effect. A significant result can also be a false positive.
The concern is especially relevant when studies are small and the effects being investigated are modest. Statistical significance alone does not establish the size or reliability of an effect.
The excerpt ends at a reference to a paper in Psychological and does not supply its title or findings.
When to use it
Evaluating the validity of published research findings in psychology or biomedicine.
Planning a new study with adequate statistical power to detect realistic effect sizes.
Interpreting meta-analyses that may include underpowered studies.
Teaching research methods to help students understand the limitations of p-values and low power.
FROM THEORY TO EVERYDAY LIFE
In practice
Small Gene Effect Study
Suppose a study with low power (N=50) tests a gene variant whose true effect on disease risk is small (odds ratio = 1.1). By chance, it finds p=0.04 and estimates an odds ratio of 2.0. The estimate overstates the stipulated true effect; this hypothetical case illustrates exaggeration, rather than establishing that every significant low-powered result is false.
Underpowered Psychology Experiment
Suppose an experiment has 20 participants per condition and tests a small cognitive effect (d=0.2). With low power, a result passing p<.05 may reflect an unusually large estimate and may fail to replicate. The example illustrates a risk, not a reported experiment.
A MOMENT TO REFLECT
Put it to work.
Can you identify a real-world study that might have been underpowered and drew overly strong conclusions?
How would you design a replication study to avoid the low-power pitfall?
What does the phrase 'Doctor, it hurts when I P' suggest about the misuse of p-values?
CONTINUE THE CONVERSATION
Source texts
- How Not to Be Wrong The Power of Mathematical Thinking (Ellenberg, Jordan)