Experiments Over Correlational Methods

Inquiry is fatal to certainty. —Will Durant, philosopher

Institutions increasingly rely on experiments to provide them with information. That’s a good thing, because if you can do an experiment to answer a question, it’s nearly always going to be better than correlational techniques. The correlational technique known as multiple regression is used frequently in medical and social science research. This technique essentially correlates many independent (or predictor) variables simultaneously with a given dependent variable (outcome or output). It asks, “Net of the effects of all the other variables, what is the effect of variable A on the dependent variable?” Despite its popularity, the technique is inherently weak and often yields misleading results. The problem is due to self-selection. If we don’t assign cases to a particular treatment, the cases may differ in any number of ways that could be causing them to differ along some dimension related to the dependent variable. We can know that the answer given by a multiple regression analysis is wrong because randomized control experiments, frequently referred to as the gold standard of research techniques, may give answers that are quite different from those obtained by multiple regression analysis. Even when assignment to condition is not literally random, we can sometimes find “natural experiments.” These can occur when there happen to be groups of cases (people, agricultural plots, cities) that differ in interesting ways with respect to an independent variable, and there is no reason to assume that membership in the group is biased in some way that would prevent us from comparing the groups with respect to some dependent variable. Society pays dearly for the experiments it could have conducted but didn’t. Hundreds of thousands of people have died, millions of crimes have been committed, and billions of dollars have been wasted because people have bulled ahead on their assumptions and created interventions without testing them before they were put into place. When it’s human beings we’re studying, there’s a temptation to rely on verbal reports. Those reports are subject to a wide variety of errors. If we can possibly measure actual behavior rather than verbal reports, we’re more likely to get a correct answer to a research question. You can do experiments on yourself that will provide much more accurate answers about what affects your health and well-being than casual observation can produce.