Ignore the HiPPO

9. Ignore the HiPPO

In this excerpt from Mindware, Richard Nisbett describes how experiments can test the assumptions behind a decision. The companies, campaigns, and research findings appear in the context of the book.

Shortly after Barack Obama announced he was running for president in the fall of 2007, Google’s CEO, Eric Schmidt, interviewed him in front of a large audience of Google employees. 1 As a joke, Schmidt’s first question was, “What is the most efficient way to sort a million 32-bit integers?” Before Schmidt could ask a real question, Obama interrupted: “Well, I think the bubble sort would be the wrong way to go,” a response that was in fact correct. Schmidt slapped his forehead in astonishment, and the room broke out in applause. Later, in the question-and-answer period, Obama assured his audience, “I am a big believer in reason and facts and evidence and science and feedback,” and promised that he would run the government accordingly.

In the audience that day was a product manager named Dan Siroker, who made a decision on the spot to go to work for Obama. “He had me at bubble sort.” Siroker had some science to offer Obama’s campaign. He showed workers how to A/B test. When you don’t know which of two treatments or procedures is best to achieve some goal, you compare the two by flipping a coin to decide who gets treatment A and who gets treatment B. You then collect data relevant to the question you’re interested in and analyze the data by comparing the average of A with the average of B using a statistical test of some kind. The same principles apply to professional work and many everyday choices. If you understand how a good experiment is designed, you’ll be better equipped to critique alleged scientific findings that you encounter in the media.

A/B

By the time Dan Siroker joined the Obama campaign website, developers for Google and other Internet companies had for several years been testing variations of web pages online. Instead of basing decisions about web design on HiPPOs—the derisive term for the “highest-paid person’s opinion”—they were comparing measured outcomes under different designs. A certain percentage of web users would be offered a home page design with lots of blue, and other users would be offered a design with lots of red. The information they sought was “percent who clicked.” Potentially every aspect of the page from color to layout to images to text would be tested simultaneously on randomly selected users. The evidence, and not the HiPPO, was the decider about what should be on the website.

The application of A/B testing to political websites was straightforward. A major question was how to design a web page that would optimize the number of e-mail addresses for possible donors. For example, which button would get the most sign-ups—“Learn More,” “Join Us Now,” or “Sign Up Now”? Which image would get more sign-ups—a luminous turquoise photo of Obama, a black-and-white photo of the Obama family, or a video of Obama speaking at a rally? I’m guessing you wouldn’t have predicted that the combination of “Learn More” plus a family photo would be the most effective. And not just a little more effective. That combination produced 140 percent more donors than the least effective combination, which translates into a huge difference for donations and votes.

Website designers have learned what social psychologists discovered decades ago about their intuitions concerning human behavior in novel situations. As Siroker puts it, “Assumptions tend to be wrong.” From 2007 on, A/B testing dictated a wide range of Obama campaign decisions. The campaign specialist and former social psychologist Todd Rogers conducted dozens of experiments for Obama. Some of the experiments were shots in the dark. Is it better for donations and for voter turnout to get a robocall from Bill Clinton or a chatty call from a volunteer? (The latter, it turns out, by a lot.) A visit from a campaign worker just before Election Day is the single most effective way yet discovered to get someone to show up at the polls.

There is now a large body of research on what works for getting out the vote. Which is more effective at getting people to the polls: telling people that turnout is expected to be light or that turnout is expected to be heavy? You might think that telling people that voting is going to be light would make them more likely to vote. A quick cost-benefit analysis shows that your vote would count for more than if turnout was heavy. But remember how susceptible people are to social influence. They want to do what other people like them are doing. If most are drinking a lot, they’ll go along; if they’re not drinking a lot, they’ll cut back. If most people are reusing their towels in their hotel room, so will they. And so telling voters there will be a heavy turnout in their precinct turns out to be much more effective than saying there will be a light turnout.

Is it effective to let people know that you know they voted in the last election—and that you’ll be checking on them after this one? People want to look good in others’ eyes—and in their own. So it’s not surprising to learn that the promise of checking up can be worth 2.5 turnout percentage points or even more. 2 But only A/B testing could show whether the tactic of checking up would produce positive or negative results, or would have no impact at all.

In both 2008 and 2012 the Obama campaign had so many tricks up its sleeve that the Republican campaigns were blindsided. The Romney campaign in 2012 was so confident of victory that no concession speech was prepared for the candidate. Republicans, however, are perfectly capable of playing the A/B game themselves. Indeed, already in 2006, the campaign of Governor Rick Perry of Texas had established that bang for the buck was poor for direct voter-contact mail, paid phone calls, and lawn signs. So the campaign spent no money on those things. Instead, the campaign used TV and radio spots heavily. Just which spots were the most effective was established by isolating eighteen TV markets and thirty radio stations and assigning start dates at random. Opinion polls tracked just which spots produced the biggest shifts toward Perry. The design’s randomized nature added hugely to the accuracy of the results. Campaign workers weren’t allowed to pick which market got which treatment at which time. If they had, any improved poll results could have been due to changed conditions in a given market rather than whether an ad had been placed in that market.

A/B testing can be just as useful for business as for politics, because researchers can segment the population and assign different treatments at random. When the number of cases ( N ) is very large, even very small differences can be detected. And in business as in politics, a small increment can make all the difference to success.

Doing Good While Doing Well

Merchandisers are making ever greater use of A/B testing. And they’re finding that it can be as useful for finding ways to improve people’s lives as it is for improving the bottom line. Researchers in an El Paso, Texas, supermarket conducted A/B tests on a large number of tactics for increasing sales of fruits and vegetables. 3 Placing a divider in the shopping cart with a sign saying “Please place fruits and vegetables in the front of the cart” can double sales of fruits and vegetables, which are more profitable for the store than most other food items and will also have a beneficial effect on customer health. 4 Researchers have also put social influence to work. Signs telling customers that the average shopper in the store purchases X number of produce items can boost produce sales. And the signs turn out to have a massive effect on the purchases of the group with the most to gain by increasing their fruit and vegetable consumption, namely low-income people, many of whom are otherwise particularly likely to purchase processed foods and unlikely to buy fresh produce.

American grocery stores tend to group items by category: starches in aisle 4, sauces in aisle 6, cheeses in aisle 9. Japanese grocery stores tend to group items holistically, by type of meal: pasta, sauce, and cheese in the Italian meal section; tofu, seafood, and soy sauce in the Japanese section. The holistic technique can reduce processed food purchases and make customers who are pressed for time more likely to buy wholesome foods for home cooking. 5

Organizations can conduct far more experiments on the effectiveness of their operations and their work environments than they do. Are employees more productive when they’re allowed to work at home part of the time? All of the time? None of the time? Are high school students more likely to do their homework if one large assignment is given once a week or a small assignment is given every day?

Within Designs Versus Between Designs

A nationwide chain like Sears can randomly direct ads to particular segments of the public in a given media market, and it can randomly select where to put items of a given type in the store—in the back in New Hampshire and North Carolina and in the front in Vermont and South Carolina, for example. The number of Sears stores nationally is sufficiently large that A/B tests can have substantial power . The power of a statistical test is its probability of detecting an effect of a specified size when that effect exists. A larger N generally improves precision and power, provided the design and analysis account for how observations were collected. You can increase power still further by using a within design , for example by flipping locations of products in the same store. This controls for the host of differences that can exist between stores. A typical within design is the before-after design. What are sales like when you put jewelry near the front of the store and undergarments near the back? What are they like when you reverse that?

A/B tests with before/after designs are much more sensitive than simple A/B designs because you can get a “difference score” for each case and use that as your measure. This index compares sales at the Houston location before and after the treatment. Comparing each store with itself accounts for stable differences across locations, such as store size and local preferences. It does not automatically account for changes over time. Those sorts of differences are referred to as error variance , because they reflect variation between stores or between people that have nothing to do with the intervention: Scores may be high or low for reasons that are irrelevant to the question that A/B testing is intended to answer. You’re more likely to know whether a difference in sales under condition A versus condition B is real or not when you reduce error variance by getting a before-score and an after-score for each case.

Note that if you’re using a before/after design, you have to counterbalance the order of treatments. That is, some cases are exposed to the experimental condition first and some to the control condition first. Otherwise, treatment effects and order effects are confounded . What you think is an effect of the treatment may actually be an effect of order of events or simply of time.

An accidental before/after change can suggest a useful hypothesis, even without a controlled experiment. My favorite such case concerns a southwestern gift shop. 6 Turquoise jewelry had been selling poorly, so one evening before the owner left for a brief trip, he decided to put the jewelry on sale and left a sign for his assistant reading “all turquoise in this case X 1/2.” When the store owner returned, almost all the jewelry was gone. He was pleased to see that, but not nearly as pleased as when the assistant told him of his amazement that the jewelry had sold better at twice its regular price than it had before the markup! The assistant had misread the note to indicate that there should be a markup of 100 percent rather than a reduction by 50 percent. One explanation is that customers took the high price as a sign of quality. The anecdote alone cannot rule out other reasons for the change in sales. That wouldn’t work with all kinds of merchandise, of course, but turquoise is particularly susceptible to reliance on the cost cue because few people can make a knowledgeable assessment of its quality.

The same design principles can help with low-stakes questions about everyday habits. A log may suggest an explanation, but changing several things at once makes it hard to tell which change mattered. Randomizing the order of alternatives, repeating the comparison, and varying one factor at a time can make the result more informative. Nisbett returns to personal experiments in Chapter 12, in his discussion of verbal report.

Statistical Dependence and Independence

A larger number of cases, and random assignment of cases to experimental condition, increase our confidence that a given effect is real. But another factor is just as crucial, namely what can count as a case. Suppose you try procedure A in classroom 1 with thirty students. Procedure A might be the standard way of teaching—lectures in class and homework to be done outside of class. And you try unconventional procedure B—lectures to be seen at home on video and guided “homework”—in classroom 2 with twenty-five students. What is the total number of cases ( N )? Alas, not fifty-five, which would be a respectable number likely to show a significant difference if there really is a difference. There are only 2 independently assigned classrooms. Students within each classroom share a teacher and a treatment, and their outcomes may also influence one another. But in the case of a classroom of students or any group of people who interact with one another during the period of the treatment and measurement of its effects, the individual behaviors are not independent of one another. Joan’s confusion may rattle others; Billy’s antics may lower everyone’s score on the test. The behavior of every individual is potentially dependent on the behavior of every other individual. The analysis must account for grouping. With just one classroom receiving each procedure, the treatment is confounded with the classroom: a difference could reflect the teacher, the students, or the procedure. Adding more students to those same classrooms does not provide independent replications of the treatment. If you can’t conduct statistical tests, there is inevitable uncertainty about exactly what was the effect of the different treatments. However, doing next time around whatever worked better the first time around is better than just relying on your assumptions.

The concept of independence is crucial to understanding a limitless range of events. Incredibly, as of 2008, financial ratings services such as Standard & Poor’s used models of probable defaults for mortgages that assumed defaults were independent of one another. 7 Joe Doakes’s default in Dubuque was assumed to mean nothing about the likelihood of Jane Doe defaulting in Denver. This is not completely unreasonable in normal times. But under a wide range of circumstances, including most certainly during a period when you’re in the midst of rapidly rising home prices, you have to assume that there is a possibility that you are in a bubble. In that case, the likelihood of default for mortgage 20031A is statistically dependent on whether mortgage 90014C defaulted. The rating agencies were not—and are not—disinterested parties. They’re paid by the banks to do their ratings, and a rating company’s services are more in demand if the company is in the habit of rating securities as safe. So whether the ratings agencies were unbelievably inept in creating their default models or were simply guilty of fraudulent practice, I’m not in a position to know. But in any case the lesson is clear: faulty scientific methodology can have catastrophic consequences.

Summing Up

Assumptions tend to be wrong. And even if they didn’t, it’s silly to rely on them whenever it’s easy to test them. A/B testing is child-simple in principle: create a procedure you want to examine, generate a control condition, flip a coin to see who (or what) gets which treatment, and see what happens. A well-conducted randomized comparison can support a causal inference about the tested intervention. An observed difference still needs to be assessed for sampling uncertainty, missing data, and problems in how the experiment was carried out. A difference found by using correlational methods can’t guarantee that the independent variable actually exerts an effect on the dependent variable. Correlational designs are weak because the researcher hasn’t assigned the cases to their condition. For example, lots of homework versus little, radio ads versus circulars, high income versus low income. If you don’t randomly assign cases—people, or animals, or agricultural plots—to a condition, you invite on board all kinds of uncertainties. Cases at one level of the independent variable may differ from those at another level in any number of ways, some of which can be identified and some of which can’t. Any of the measured variables, or variables not measured or even conceived of, could be producing the effect rather than the independent variable of interest. And it might even be that the variable presumed to be dependent is actually producing differences in the variable presumed to be the independent one.

More independent cases generally make it easier to detect a real effect; they do not by themselves lower the false-positive rate chosen for a test. Under the null hypothesis and the test’s assumptions, a result at least as extreme as the observed one occurring less than one time in twenty corresponds to significance at the .05 level. Even then, a test can “find” an effect that is not real. Statistical significance does not establish the probability that the null hypothesis is true, or tell us whether the effect is useful.

When you assign each case to all of the possible treatments, your design is more sensitive. That is to say, a difference of a given magnitude found by a “within design” is more likely to be statistically significant than when tested in a “between” design, when repeated measurements reduce irrelevant variation. Stable differences between cases are accounted for, but order effects, changes over time, and carryover between treatments still need attention.

It’s crucial to consider whether the cases you’re examining (people in the case of research on humans) could influence one another. Whenever a given case might have influenced other cases, such that any one case could have had an impact on other cases, there’s a lack of statistical independence. Count independent units at the level of treatment assignment. When a whole classroom receives the same treatment, Classroom A supplies just 1 independently assigned group, even though it contains many students. The analysis can use student-level measurements, but must account for their grouping; preventing conversation during an exam does not remove the shared teacher or classroom treatment.