Eekonomics

Eekonomics

Chapter 11 of Richard E. Nisbett’s Mindware discusses studies and public debates from the period in which the book was written. Its medical examples illustrate research methods.

Do auto salespeople offer deals to women that are more expensive than the deals they offer to men? Does classroom size affect learning? Are multivitamins good for your health? Is there employer prejudice against the long-term unemployed—simply because they’ve been out of a job for a long time? Should postmenopausal women take hormone replacement therapy to reduce the likelihood of cardiovascular disease? Many answers to each of these questions have been proposed. Some answers were based on studies that reached incorrect conclusions because of faulty methodology. Some answers are quite likely correct because good scientific methodology was used. This chapter makes three points that are crucial to understanding scientific findings and deciding whether to believe them.

1. Studies that rely on correlations to establish a scientific fact can be hopelessly misleading—even when the correlations come in complicated packages called “multiple regression analysis” that “control for” a host of variables.

2. Experiments in which people (or objects of any kind) are assigned randomly to one treatment versus another (or no treatment at all) are in general far superior to research based on multiple regression analysis.

3. Assumptions are so often wrong when it comes to human behavior that it’s essential to conduct experiments if at all possible to test important hypotheses about behavior when feasible.

Multiple Regression Analysis

All the questions that began this chapter ask whether some independent or predictor variable —an input or a presumed cause—affects some dependent or outcome variable —an output or an effect. Experiments manipulate independent variables; correlational analyses merely measure independent variables. One technique employing correlational analysis is multiple regression analysis (MRA), in which a number of independent variables are correlated simultaneously (or sometimes sequentially, but we won’t talk about that variant of MRA) with some dependent variable. The predictor variable of interest is examined along with other independent variables that are referred to as control variables. The goal is to show that variable A influences variable B “net of” the effects of all the other variables. That is to say, the relationship holds even when the effects of the control variables on the dependent variable are taken into account.

Consider this example. Smoking cigarettes is correlated with higher incidence of cardiovascular disease. The temptation is to say it looks as if smoking causes cardiovascular disease. The problem is that lots of other things are correlated both with smoking and with cardiovascular disease, such as age, social class, and excess weight. Older smokers have smoked for a longer time than younger smokers, so we need to pull age out of the smoking-disease correlation. Otherwise, what we’re showing is that being older and smoking are associated with cardiovascular disease. But that conflates two variables. What we want to know is just the association between smoking and cardiovascular disease, regardless of how old a person is. We “control for” age by estimating the smoking-disease association after accounting for age in the model. That estimates an age-adjusted association; it does not by itself show that the association is identical in every age group. The same logic applies to social class. Other things being equal, the lower the social class, the greater the likelihood of smoking, and the lower the social class, the greater the risk of cardiovascular disease, independent of any risk factor such as smoking. Ditto for excess weight. And so on. The correlations of these variables with both smoking and degree of cardiovascular disease have to be pulled out of the correlation between smoking and cardiovascular disease.

Multiple regression can help estimate a causal relationship if the study identifies and adequately measures the relevant confounders and satisfies the other assumptions of its causal model. Adding every related variable is not sufficient: controlling for a consequence of the treatment, for example, can distort the effect being estimated. That’s the theory. In practice, many things prevent this ideal case from being the norm. First of all, how do we know we’ve identified all of the possible confounds—the variables linked to both the predictor and outcome variables? We’re almost never able to make that claim. We can only measure what we assume might be important and leave out the infinity of variables that we don’t assume are important. But ATTBW: Assumptions Tend to Be Wrong. So the battle is usually lost right there.

Second, how well do we measure each of the possible confounding variables? If we measure a variable poorly, we haven’t controlled for it enough. If we measure a variable so poorly that it has no validity, we haven’t controlled for anything.

Sometimes MRA is the only research tool available for examining interesting and important questions. An example is the question of whether religious belief and practice are associated with greater or lesser rates of procreation. We can’t do an experiment to test that question, randomly assigning people to be religious or not. We can only use correlational methods such as MRA. As it happens, religiosity is correlated with fecundity at both the individual level and the national or cultural level. Controlling for income, age, health status, and other factors at the level of individuals, at the level of ethnic groups, and at the level of countries, greater religiosity is correlated with greater fecundity. Just why this is we don’t know, and the correlation between religiosity and fecundity might not be causal at all but rather could be due to the fact that some unmeasured third variable influences both religiosity and fecundity. Causality could even run in the reverse direction: having lots of children might make people search for divine support and guidance! The correlational finding is interesting nonetheless, and knowing about it might have real-world consequences.

I want to be very clear that not all correlational research, or all multiple regression research, is valueless. I’ve often used multiple regression myself, even when I’ve conducted experiments establishing a causal relationship. I feel more comfortable knowing that a given relationship exists in the wild and not just in the laboratory or in a possibly atypical ecological environment.

Moreover, there are often clever things that one can do to make us pretty sure that we’ve learned something about causality. Take the correlation between the wealth of nations and the IQ of nations. What’s going on causally there? The correlation taken by itself is hugely problematic. Lots of things are correlated with both wealth and IQ—physical health, for example. “Healthy, wealthy, and wise” is not just an expression; the three go together in a bundle of correlations that include many other potential causal variables as well. Moreover, there’s a plausible causal story running in both directions. As a nation gets smarter it gets richer, because more advanced and complex ways of making a living become possible. As a nation gets richer it gets smarter, because wealth generally increases the quality of education.

But we can sometimes tell a pretty good causal story by looking at what are called “lagged correlations,” that is, the correlation of an independent variable (assumed cause) with another variable (assumed outcome of the cause) at a later time. If a nation gets smarter—because of an increase in education, for example—does it get richer down the road? Indeed it does.

For example, a few decades ago, Ireland made a concerted and highly successful effort to improve its educational system, especially at the high school, vocational school, and college levels. 1 College attendance actually increased by 50 percent over a brief period of time. 2 Within about thirty years, the per capita GDP of Ireland, which previously had IQ scores far lower than that of England (for genetic reasons, according to some English psychologists!), had exceeded the per capita GDP of England.

Finland also made significant educational improvements beginning several decades ago, focusing especially on making sure the poorest students got an education as equal as possible to that of the richest students. By 2010, Finland was ahead of every country on international tests of academic achievement, and its per capita income had risen to be greater than that of Japan and Britain and only slightly less than that of the United States. Nations that have not made heroic efforts to improve education in recent decades, such as the United States, have declined in per capita income relative to other advanced countries.

Such data are still correlational, but they indicate that as a nation begins to break out of the pack educationally, it begins to get richer. As it stagnates educationally, it begins to lose wealth relative to other nations. Pretty persuasive.

Many other circumstances can pull correlational research up to a level of convincingness equal to natural experiments or even randomized control experiments. For example, sheer magnitude of an effect can sometimes make us feel that it must not be a mere artifact due to correlated variables. We also become somewhat more confident that a given treatment is real if the effect is “dose-dependent.” That is, the more intense or frequent the treatment, the higher the level of response. For example, people who smoke two packs a day are much more likely to have poor cardiovascular function than people who smoke half a dozen cigarettes a day. This makes it more likely that smoking really does worsen cardiovascular health than if it were the case that the sheer amount of smoking is unrelated to morbidity.

But there are serious problems with multiple regression analysis as it’s all too frequently conducted. I’m going to be very explicit about the problems, because the media constantly report findings based on this highly fallible method, and important policy decisions are made on the basis of them. Epidemiologists, medical researchers, sociologists, psychologists, and economists all use this technique. It can produce serious errors, and the claims of some devotees that it can reveal causality are usually bogus. In many instances, MRA gives one impression about causality, and actual randomized control experiments give another. In such cases we should believe in the results of the experiments.

Would you think the number of children in a classroom matters for how well schoolchildren learn? It seems reasonable that it would. But dozens of MRA studies by highly regarded investigators tell us that, net of average income of families in the school district, size of the school, IQ test performance, city size, and geographic location, average class size is uncorrelated with student performance. 3 The implication: we now know we needn’t waste money on decreasing the size of classes.

But scientists in Tennessee conducted a randomized experiment in which they varied class size substantially. By the flip of a coin, researchers assigned children in kindergarten through third grade to either small classes (thirteen to seventeen) or larger classes (twenty-two to twenty-five). The study found that smaller classes produced about a .22 SD improvement in standardized test performance; the effect on minority children was greater than the effect on white children. 4

There are now three other experimental studies of the effects of reduction of class size, and their findings are almost identical to those of the Tennessee study. 5

These four experiments provide stronger evidence about the effects of class size in the settings they tested. They change how we should interpret the multiple regression studies, although they do not settle every question about class size in other settings. Why do the multiple regression studies find that class size matters so little? I don’t know, but we don’t have to know in order to have a strong opinion about whether class size can matter.

There’s a lot that the four experiments leave unanswered, of course. We don’t know whether class size makes a difference for every region of the country, every degree of urbanization, every level of social class, and so on. We don’t know what’s going on in classrooms that produces the different educational effects. But those questions can be answered by further experiments, and positive findings for each experiment looking at populations that differ in notable ways from those examined in the available studies would increase our confidence that larger class size does indeed make a difference.

Whether decreasing the size of classes is the best thing to spend the educational dollar on is a separate question, and an answer to it is above my pay grade. Finland doesn’t have particularly small classes; the improved educational outcome was more likely a result of paying teachers more and recruiting them primarily from the top of their college classes rather than the bottom, as the comparison with the United States in this account suggests. And in any case, policy can’t be determined simply by finding a beneficial effect of X on Y; a full-dress cost-benefit analysis is required.

The problem with correlational studies such as those based on MRA is that they are by definition susceptible to errors based on self-selection. Cases—people, classrooms, agricultural plots—differ in any number of ways. Longtime smokers aren’t just longtime smokers, they’ve dragged along lots of other factors associated with smoking—such as greater age, lower social class, and excess weight. Classroom A is larger than classroom B, but it also differs in any number of ways over which the investigator has no control. Classroom A may have a better teacher because the principal thought that teacher could handle large-size classes better than other teachers in the school. Classroom B may have high achievement scores, even though it has a large number of students, because the principal thought that more able students would suffer less from relative lack of attention than less able students. And so on.

The problem doesn’t get solved simply by adding more classrooms, or more control variables, into the mix.

In studies where cases are randomly assigned to an experimental condition, the variability among classrooms on other dimensions remains. But—critically—it’s the experimenter who selects the condition. Random assignment balances teacher quality, student ability and motivation, and resources in expectation, although a particular sample can still have chance imbalances. The classrooms have not “selected” their treatment condition; the experimenter has assigned it. This breaks systematic links between assignment and preexisting differences, making the class-size comparison more credible.

Experiments like those on class size are not conclusive. Neither teachers nor administrators, for example, are blind to condition. They know which classes are small and which are large, and this might affect how teachers teach, including how much effort they put into the job. It’s just that such problems pale in significance to the self-selection problem.

Medical Muddles

Did you know that consuming large amounts of olive oil can reduce your risk of stroke by 41 percent? 6 Did you know that if you have cataracts and get them operated on, your mortality risk is lowered by 40 percent over the next fifteen years compared to people with cataracts who don’t get them operated on? 7 Did you know that deafness causes dementia? Did you know that being suspicious of other people causes dementia? If those claims sound dubious to you, they should. But alleged findings such as these appear constantly in the media. They’re typically based on epidemiological studies. (Epidemiology is the study of disease patterns in populations and their causes.) A great deal of epidemiological research relies on MRA. Studies using it attempt to “control” for factors such as social class, age, and prior state of health. But they can’t get around the self-selection problem. The sorts of people who get a given medical treatment, or who consume large amounts of a particular food, or who take a particular vitamin, differ from those who don’t get the treatment, don’t consume the food, or don’t take the vitamin, in goodness knows how many ways.

Let’s look at the study claiming that, net of control factors including “socio-demographic variables, physical activity, body mass index, and risk factors for stroke,” people who consume more olive oil have fewer strokes. 8 The reduction in one study was 41 percent for “intensive” users of olive oil versus those who never used it. But the lower stroke risk might reflect something correlated with olive-oil consumption rather than the oil itself. For starters, dietary habits can vary with cultural background, and groups can also differ in stroke risk for reasons unrelated to a particular food. Such differences are possible confounders, not evidence that ethnicity explains this study’s result.

The biggest potential confound in any epidemiological study is typically social class. Class is a glaringly obvious candidate for affecting differences in risk for stroke and many if not most other medical outcomes. The rich are different from us. They have more money. People with more money can afford to use olive oil rather than corn oil. People with more money are also more likely to be widely read, and associate with other readers, and therefore to believe that olive oil is better for one’s health than its cheaper competitors. People with more money get better medical care. And people with more money—and with higher social class, whether you measure class by education, personal income, or occupational status—have better life outcomes of all kinds. If social class confounds the relationship being studied, failing to account for it can undermine a causal interpretation.

But supposing the investigator does try to measure social class, how should it be done? Some use income, some use education, some use occupation. Which is best? Or should you somehow combine the three? The truth is that different epidemiological studies may try to measure social class in any or all or none of these ways. And that contributes to the constant churning of “medical findings” reported by the media. 9 (Fat is bad for you. No. Fat is good for you. Red meat is good for you. No. Red meat is bad for you. Antihistamines reduce the severity of the common cold. No. Antihistamines have no effect.) The differing results are frequently just the consequence of defining social class differently, or of not examining it at all.

But social class is only one of an unlimited number of potential confounds present in MRA studies. Almost anything that’s correlated with both the predictor variable and the outcome variable in such studies becomes a candidate for explaining the correlation between the two.

There are thousands of food supplements on the market. MRA studies sometimes find that one or another supplement is beneficial for one thing or another. The media then pass along that finding. Unfortunately, there is usually no way for the reader to tell whether a given study is based on MRA, in which case you probably shouldn’t pay much attention to it, or an actual experiment, in which case it may be very important indeed for you to pay attention. Reporters, even those who specialize in health reporting, typically don’t fully understand the crucial difference between the two methodologies. There are countless examples of MRA studies finding one thing and experiments finding another.

For example, MRA studies report that vitamin E supplements reduce the likelihood of prostate cancer. By contrast, an experimental study was conducted at many locations in the United States, assigning at random some men to vitamin E supplements and others to a placebo. This experiment found a slight increase in the likelihood of cancer by virtue of taking vitamin E. 10

Vitamin E is not the only suspect supplement. There are a host of experimental studies showing that taking multivitamins—something that almost half of Americans do—does little or no good, and very high doses of some vitamins can do genuine harm. 11 There is almost no evidence one way or another about the effects of any of the other fifty thousand or so food supplements that are on the market. Most of the evidence we do have about any given supplement indicates that it’s useless; some indicates it’s actually harmful. 12

The chapter also criticizes the supplement industry’s lobbying and a regulatory system that does not require supplements to pass the same premarket tests of effectiveness as medicines. This is different from being exempt from all federal regulation. As a consequence, billions of dollars are spent every year on nostrums that are useless or worse.

Using Multiple Regression Analysis When Only Experiments Can Do the Job

The longer a person is unemployed, the harder it is to find a job. At the time of this account, the number of people who have been unemployed just for a short time (fourteen weeks or less) is only slightly more than it was just before the Great Recession hit. 13 But the number of long-term unemployed is 200 percent higher than it was then. Is there prejudice against the longtime unemployed on the part of employers? Are they not given consideration simply because they’ve been out of a job for a long time? MRA can’t tell us whether, other things being equal, employers unjustifiably pass over the longtime unemployed in favor of the briefly unemployed. The long-term unemployed may have poor employment records, or be lackadaisical in job hunting, or be too picky about the kind of job they would do. Politicians routinely invoked these alleged causes during the Great Recession. But you can’t know whether these explanations are correct by conducting a multiple regression analysis. Simply “controlling” for measured variables does not guarantee that self-selection bias has been removed. An experiment can test whether employers respond differently to otherwise comparable applications showing different unemployment durations. Such an experiment has been done.

The economists Rand Ghayad and William Dickens sent out 4,800 fictitious applications for six hundred job openings. 14 Even when applications were identical except for alleged length of unemployment, the short-term unemployed were twice as likely to get an interview as the long-term unemployed. In fact, the short-term unemployed were more likely to get an interview for a job for which they were not very well qualified than were long-term unemployed people who were more fully qualified!

Some questions can be tested directly by an experiment, yet researchers also try to answer them through multiple regression analysis. Many experimental studies have shown that African American job applicants with black-sounding names (D’André, Lakaisha) are less likely to get interviews than identical applicants with names that don’t sound black (Donald, Linda). Applicants with white-sounding names have as much as a 50 percent greater likelihood of being granted an interview than applicants with black-sounding names. 15 Having a white-sounding name rather than a black-sounding name is worth as much as eight years of job experience.

Being dubious about whether black names actually produce worse economic outcomes, the highly respected economists Roland Fryer and Steven Levitt conducted a multiple regression study examining the relationship between having a black-sounding name and various economic outcomes. 16

The population they studied consisted of black women born to non-Hispanic blacks in California who had remained in the state as adults. The dependent variables were not job-hunting success, or income, or occupational status, but indirect measures of life outcomes such as average income in the woman’s zip code and whether the woman had private health insurance. The investigators report that the latter variable is “the best measure we have with respect to the quality of her current employment.” 17 (The best that the investigators had, that is. It’s really a rather crude measure of occupational attainment.)

Fryer and Levitt found that women with black-sounding names fare substantially worse on their indicators of occupational success than women with white-sounding names, as we would expect based on the experimental studies. But the relationship between type of name and the outcome variables vanished when they controlled for variables such as percent of black babies in the hospital of the woman’s birth, percent of black babies in the county of her birth, whether the mother was born in California, mother’s age at time of birth, father’s age at time of birth, months of prenatal care, whether the woman was born in a county hospital, her birth weight, her total number of children, and whether she was a single mother.

The authors were aware of the problems with this kind of analysis. They acknowledge that “the clear weakness of this empirical approach is that if unobserved characteristics of the woman are correlated both with life outcomes and her name, our estimates will be biased.” 18 Indeed. Nevertheless, the authors go on to state that there is no connection between how black-sounding one’s name is and life outcomes after they control for other factors. “We find … no negative relationship between having a distinctively Black name and later life outcomes after controlling for a child’s circumstances at birth.” 19

A very large number of variables, many of which would be more predictive of occupational success than the ones Fryer and Levitt examined, would have to have been assessed in order to justify that conclusion. (And when very large numbers of control variables are examined, many of them having stronger relationships with the dependent variable than the correlation of most interest, conclusions become shaky.) Fryer and Levitt imply that parents can give their child a black-sounding name without worrying that there might be negative effects on occupational success. That seems extraordinarily unlikely in light of the experimental studies.

A study by Katherine Milkman and her colleagues shows that a black-sounding name can surely disadvantage candidates for graduate school. 20 Thousands of professors were sent an e-mail message allegedly from a prospective graduate student requesting a meeting a week hence to discuss research opportunities. A male student with a white-sounding name was 12 percent more likely to be granted the interview than a male student with a black-sounding name. Such a difference could have real consequences. Getting one’s first choice for graduate advisor versus not can mean a distinguished career versus a less distinguished one.

Why might Fryer and Levitt have been willing to assume that an MRA study could be powerful and accurate enough to cast doubt on the implications of the experimental studies? I suspect it’s because of what the French call déformation professionnelle—the tendency to adopt the tools and point of view of people who share one’s profession.

For many of the types of research that economists do, MRA is the only available option. Economists can’t manipulate interest rates set by the Federal Reserve. If you want to find out whether austerity or pump-priming was more helpful to a country’s economy during the Great Recession, you can correlate degree of austerity with strength of recovery, but you can’t randomly assign countries to an austerity condition. Economists are taught MRA as their main statistical tool. But they are not taught to be nearly as critical of it as they need to be.

Levitt, in a book cowritten with the journalist Stephen Dubner, 21 reported on an analysis of data collected by the U.S. Department of Education called the Early Childhood Longitudinal Study. The academic achievement of students from kindergarten to fifth grade was examined, along with dozens of other variables, such as parental income and education, how many books were in the child’s home, how much the child was read to, whether the child was adopted, and so forth. Levitt reports on the MRA-based conclusions about the relationship between a host of these variables and academic achievement. He concludes that, net of many variables including number of books in the home, “reading books doesn’t have an impact on early childhood test scores.” 22 MRA simply isn’t up to the job of establishing that reading to children is unimportant for their intellectual development. A well-designed experiment would give stronger evidence about the effect of reading to children.

Levitt had an additional finding indicating that, net of many variables including reading books to children, having books in the home has an important effect on test scores. Thus, owning lots of books makes children smarter, but reading those books to them doesn’t. Levitt’s faith in MRA is such that he actually attempts to give a causal explanation for this state of affairs.

A much more important error is Levitt’s assertion that family environment has relatively little effect on children’s intellectual skills. He bases this conclusion on studies of adoptive children. “Studies have shown that a child’s academic abilities are far more influenced by the IQs of his biological parents than the IQs of his adoptive parents.” 23 But correlations are the wrong data to look at to reach an estimate of the importance of family environment. We need to look instead at the results of the natural experiment of adoption of a child versus leaving the child with the birth parents, who typically are of much lower socioeconomic status. The environments created by adoptive parents are substantially more favorable in many respects than those of families in general.

And, in fact, the school performance of adopted children is half a standard deviation higher than that of siblings who were not adopted; the IQs of adopted children are more than a standard deviation higher than nonadopted siblings. And the higher the social class of the adoptive parents (and therefore the more favorable the intellectual environment on average), the higher the IQ of the adopted child. The effects of family environment on intellectual skills are actually very great. 24

In Levitt’s defense, he didn’t come up with his mistaken conclusions about the effect of adoptive environments on his own. Behavioral scientists and geneticists have been using the correlational data to reach wrong conclusions about environmental effects on intellectual ability for decades.

Some eminent economists don’t seem to recognize the value of experiments at all. The economist Jeffrey Sachs started an extremely ambitious program of health, agricultural, and educational interventions in a small number of African villages, with the intent of improving quality of life. The program’s cost is very high relative to alternatives, and it has been severely criticized by other development experts. 25 Though some of Sachs’s villages improved their residents’ conditions, similar villages in Africa improved more without his intervention. Randomly assigning similar villages to the program or a comparison condition could have addressed this causal criticism, though it would not by itself settle every question about cost or implementation.

Sachs has refused to conduct this experiment on what he described as “ethical grounds.” 26 What’s unethical is not to conduct experiments when they’re feasible. Sachs spent a great deal of other people’s money, but we have no idea whether that money improved people’s lives more than alternative, possibly less expensive programs would have.

As it happens, though, increasing numbers of economists are conducting social psychology–type randomized control experiments. One example is a series of experiments conducted by the economist Sendhil Mullainathan and the psychologist Eldar Shafir showing that resource scarcity can have dire consequences for the cognitive functioning of everyone from farmers to CEOs. 27

In one reported experiment, participants imagined how they would rejigger their budgets to cover an auto repair costing several thousand dollars and then took cognitive tests. Lower-income participants performed worse after considering the expensive repair; higher-income participants did not show that decline. This was a difference in performance under a particular condition, not a demonstration that poverty fixes a person’s intelligence. (And contemplation of an auto repair of a few hundred dollars doesn’t impair the test performance of either poor people or well-off people.)

The economist Raj Chetty is a leader in pushing economists toward finding natural experiments that test economic hypotheses. Does teacher quality really matter in the long run? We can estimate how much difference it makes to have a highly competent teacher versus a much less competent one by looking at the average performance of a given class of students before and just after the entry of a high-value teacher (or after the exit of such a teacher). 28 For example, each cohort of third-grade students in a given school may be getting about the same mediocre scores on achievement tests year after year until the entry of a teacher with a good track record. (Perhaps the previous teacher has left because of an illness.) If the performance of the third-year classes shows a jump that is sustained as long as that teacher is in place, we can look at the effect of that increase on subsequent academic achievement, college attendance, and adult income. And the effects of teacher competence on all these variables are marked.

Such studies count as near-experiments because classes prior to the entry of a given teacher are essentially the control group for classes following the entry of the new teacher. Assignment to condition falls short of random, but when teachers’ assignments are apparently fortuitous, we have a reasonably good natural experiment.

And some of the most important experiments by economists are those on educational interventions carried out by Roland Fryer. He has conducted extremely valuable educational experiments, showing, for example, that financial incentives have little effect on the academic achievement of minority students. 29 They also have little effect on teacher performance—except when loss aversion is triggered by giving the incentives at the beginning of the year with the understanding that failure to improve student achievement will result in losing the incentive. 30 This finding, incidentally, is a lovely example of the greater effect of potential loss than of potential gain, discussed in Chapter 5. Fryer has also played a role in the very successful Harlem Children’s Zone experiments, which yielded large increases in academic achievement for African American students.

My Tribe, Too

I’m afraid I now have to admit that psychologists can be as guilty of misusing MRA as other behavioral scientists. It’s common to report the following kind of finding. Employees of companies with generous parental leave benefits are more satisfied with their jobs than employees of companies that don’t provide that benefit. This correlation is then buttressed with an MRA showing that the better the leave policy, the more satisfied employees are with their jobs, and that this is still true when you “control for” size of company, employee salary, ratings of how pleasant coworkers are, ratings of how much the immediate superior is liked, and so forth.

There are three problems with this kind of analysis. First, a limited number of variables will have been measured, and if one or more have been poorly measured, or if there are other variables not examined by investigators that are correlated both with generosity of parental leave policy and job satisfaction, it may be those associations that account for job satisfaction, not leave policy.

Second, it really makes no sense to pull parental leave policy out of the total picture of the employee’s experience with a company. Generosity of the company in that respect is likely to be bound up with all kinds of other positive qualities of the company. Pulling that one thread out of the complicated ball of relationships among variables, then attempting to “control” for a few out of many variables in that ball, is not likely to protect us from mistakes.

Third, these kinds of analyses leave us particularly vulnerable to the halo effect problem discussed in Chapter 3. People who like their jobs also find the restrooms to be cleaner, the employees to be better looking, and the commutes to be less tedious than people who don’t much like their jobs. Love is blind, and liking’s vision isn’t all that much better.

These problems are perhaps easier to see in the case of personality research. It can make little sense to single out one aspect of a person’s character and assume it’s not strongly related to—enmeshed with—other aspects of the person’s character.

It’s common for psychologists to report findings such as “Self-esteem is correlated with academic performance, controlling for extroversion, measures of self-control, depressive tendencies,” and so on.

But low self-esteem, and other undesirable states, such as depression, are generally found to be correlated: when you’re down you don’t think too highly of yourself, and when you think poorly of yourself that’s likely going to pull your mood down.

Looking at self-esteem as if it could be pulled out of the relationship to depression is just an artificial thing to do.

The two measures often overlap, though the concepts remain distinct. Consider statements such as “I think I’m terrific, too bad I’m so depressed I can hardly see straight,” or “I’ve never been happier, too bad I’m such a jerk.” It’s possible, maybe, but the odd ring of such sentences reflects the fact that self-esteem and depression are normally tangled up together. Their overlap makes the meaning of an adjusted association important; it does not make separate measurement inherently impossible.

Many of my fellow psychologists are going to be distressed by my bottom line here: such questions as whether academic success is affected by self-esteem, controlling for depression, or whether the popularity of fraternity brothers is affected by extroversion, controlling for neuroticism, or whether the number of hugs a person receives per day confers resistance to infection, controlling for age, educational attainment, frequency of social interaction, and a dozen other variables, are not answerable by MRA. What nature hath joined together, multiple regression analysis cannot put asunder.

No Correlation Doesn’t Mean No Causation

Correlation doesn’t prove causation. But the problem with correlational studies is worse than that. Lack of correlation doesn’t prove lack of causation —and this mistake is made possibly as often as the converse error. Does diversity training improve rates of hiring women and minorities? One study examined this question by quizzing human resource managers at seven hundred U.S. organizations about whether they had diversity training programs and by checking on the firms’ minority hiring rates filed with the Equal Employment Opportunity Commission. 31 As it happens, having diversity training programs was unrelated to “the share of white women, black women, and black men in management.” The authors concluded that diversity training did not affect minority hiring. But just a moment. Having diversity training versus not having it is a self-selected variable.

Some corporations that hire diversity trainers may be less interested in hiring women and minorities than corporations that find more effective ways to increase hiring. In fact, they may simply be using such programs as protective cover for their real hiring policies.

Corporations that don’t have diversity training may be effective in hiring minorities by techniques such as setting up diversity task forces or, as the U.S. military does, making successful minority advancement a part of the evaluation of superior officers. Randomized experiments could provide stronger evidence about the effects of particular diversity-training programs. We have to fight the reflexive conclusion that A can’t exert a causal influence on B because there is no correlation between the two.

Discrimination: Look at the Statistics or Bug the Conference Room?

While we’re on the topic of discrimination, let me point out that raw group statistics alone do not establish whether discrimination explains a disparity in an organization—or a society. You often read about “glass ceilings” for women in a given field or about disproportionate school suspensions of boys or minorities. The intimation—often the direct accusation—is that discrimination is at work. But numbers alone won’t tell the story. We don’t know that as many women as men have the qualifications or desire to be partners in law firms or high-level executives in corporations. And we have some pretty good reasons to believe that girls and boys are not equally likely to engage in behavior warranting suspension from school.

Not so long ago, it was common to attribute women’s lower representation in graduate school and faculty rosters to discrimination. And there certainly was discrimination. I know; I was there. I was privy to the conversations the men had about admitting women to grad school or hiring them onto faculties. “Go after the guy; women are too likely to drop out.” Bugged conversations would have proved what raw statistics, comparing percentage of men and women hired, could not.

At the time of this account, 60 percent of college graduates are women, and they constitute a majority of law and medical students as well as graduate students in the humanities, social sciences, and biological sciences. At the University of Michigan, where I teach, two-thirds of the assistant professors hired are women (and they get tenure at the same rate as men).

Do these statistics prove discrimination against men? They do not. And I can assure you that bugged conversations—at least at my school—would not support the discrimination idea either.

On the contrary, we are so frequently confronted with the prospect of admitting huge majorities of women into our graduate program that we contemplate relaxing admission standards for men, though we’ve never carried it out in a conscious way, of that I’m sure.

The statistics on postgraduate education have not stopped some people from claiming there is still discrimination against women in the physical sciences. One book I read recently claimed that women were “locked out” of physics. A numerical disparity alone does not justify that assertion, though carefully designed statistical studies can provide evidence about discrimination.

But we don’t have to resort to bugging conference rooms to establish that discrimination exists. Experiments can do the job. Do car sales people cite higher prices for women and minorities than for white men? Send a white man, a woman, and a minority group member to Mammoth Motors and see what price each gets quoted. The study has been done, and in fact the white man is quoted the lowest price. 32

Do good-looking people get better breaks in life? Many studies show that to be the case. Clip a picture of an alleged delinquent to a file and see what sentence is recommended by undergraduate “judges.” If the kid is good-looking, he’s seen as likely to become a better citizen in the future and is given a relatively light sentence. If the kid is ugly, throw the book at him. 33 “Life is unfair,” as John F. Kennedy said, and experiments are the best instrument we have for revealing just how much more unfair it is for some groups of people than for others.

Summing Up

Multiple regression analysis (MRA) examines the association between an independent variable and a dependent variable, controlling for the association between the independent variable and other variables, as well as the association of those other variables with the dependent variable. A causal interpretation requires an appropriate research design and model, including adequate treatment of relevant confounders. It does not require measuring every possible influence on the outcome. In practice, these conditions are rarely met.

The fundamental problem with MRA, as with all correlational methods, is self-selection. The investigator doesn’t choose the value for the independent variable for each subject (or case). This means that any number of variables correlated with the independent variable of interest have been dragged along with it. In most cases, we will fail to identify all these variables.

In the case of behavioral research, it’s normally certain that we can’t be confident that we’ve identified all the plausibly relevant variables.

Despite the above facts, MRA has many uses. Sometimes it’s impossible to manipulate the independent variable. You can’t change someone’s age. Even when we have an experiment, it adds to our confidence to know that the experimentally demonstrated relationship holds in a natural ecology. And MRA is in general vastly cheaper than experiments, and it can identify relationships that would be important to examine experimentally.

When a competently conducted experiment tells you one thing about a given relationship and MRA tells you another, you normally must believe the experiment. Of course, a badly conducted experiment tells you no more than MRA, sometimes less.

Multiple regression does not require predictors to be logically or statistically independent. The problem is interpreting an adjusted association as a separate causal effect when the variables are closely related. With self-esteem and depression, for example, we need a defensible account of their causal relationship before deciding what controlling for either variable means.

Just as correlation doesn’t prove causation, absence of correlation fails to prove absence of causation. False-negative findings can occur using MRA just as false-positive findings do—because of the hidden web of causation that we’ve failed to identify.