Good Decisions Sometimes Have Poor Outcomes (Resulting)

GOOD DECISIONS SOMETIMES HAVE POOR OUTCOMES

Dan Levy presents this maxim from Richard Zeckhauser’s teaching: a good decision can have a poor outcome.

If you were asked what was the best decision you made in the past year, what would you say? Annie Duke, former world poker champion, psychologist, and author of two wonderful books on probabilistic thinking and decision making, has asked this question to hundreds of people around the world. The great majority responded with a decision that turned out well. This suggests that many people confuse the quality of the decision (how good was the process we used to make the decision with the information we had at the time?) with the quality of the outcome (how good was what happened after the decision?). Confusing the quality of the decision with the quality of the outcome is so prevalent that poker players have a term for it: resulting.

In statistics classes at Harvard’s Kennedy School, students distinguish decision quality from outcome quality. A good decision followed by a bad outcome falls into the “You were unlucky” category.

Consider the following example inspired by Annie Duke. Imagine you drove to a bar with your partner or roommate. You both had a wonderful time, but you had a few too many drinks whereas your partner just had non-alcoholic Shirley Temples. When it’s time to go home, you ask your partner – who drives as well as you – to drive both of you home. On the way home, a drunk driver hits your car in the back. You and your partner are both fine, but your car will need major repairs.

The decision to ask your partner to drive both of you home was a good one. With the information you had at the time, you reasonably judged that the likelihood of a car accident was higher if you drove rather than your partner drove. Yet the outcome was not good. It was a good decision with a poor outcome.

The maxim applies to a very wide range of situations. For example, imagine you bought a new iPhone. You then needed to decide whether to buy insurance against accidental damage. Being the kind of person who is very careful with your phone, you assessed that the price of insurance was too high given the likelihood that you would have an accident or lose your phone. Three months later, you were at the grocery store; you dropped your phone and it broke.

If you are like most people, the first thought going through your mind is regret about your insurance decision. Worse, you may have recriminations, concluding that not buying insurance was a bad decision. But the fact that you got a bad outcome does not imply that your decision was a bad one. It was a freak accident; you very rarely drop things. This maxim reminds us to judge the quality of your decision by what you knew at the time you made it, not what you found out afterwards.

Posit a more consequential decision. You have to choose between a job in San Francisco or a job in Los Angeles. You have some sense of whether you will get along with your colleagues in each location and how much you will enjoy living in each city, but you are quite unsure. You can make informed judgments about the likelihood of these outcomes, but you don’t know what will ultimately unfold. If you decide to go to San Francisco and end up not liking it, this does not necessarily mean it was a bad decision.

Josh Yardley, a former student of Richard’s, reflects on how learning this maxim in Richard’s course affects how he evaluates his decisions: “Before Richard’s class, my assessment of the quality of a decision was closely – and often unhelpfully – tied to the outcome of that decision. It’s an easy heuristic: If it worked out okay, it must have been a good decision; if not, it must have been a bad one. But this maxim says, “not so fast.” Every time I’m tempted to let the outcome of a decision determine my assessment of that decision, I try to slow my thinking down. How would I judge the process that resulted in that decision before knowing its outcome? And then, how might I update that judgment given the outcome I can now observe? That first question – How was the process? – disciplines my thinking in a way that is incredibly helpful in evaluating decisions made by my favorite sports teams, by my elected officials, and by me.”

The maxim applies to decisions as small and personal as whether to buy insurance for your iPhone and as large and important as how to steer an organization you lead during moments of crisis. In a crisis, every available option may have painful likely consequences. The goal is to choose the option that offers the best prospects given those constraints. A bad result may prompt blame even when the alternatives looked worse.

David Ellwood faced this least-bad-option situation when serving as the dean of the Harvard Kennedy School during the 2008 financial crisis. Any budget cut would be painful, and a series of budget cuts even more so. None of the people he would let go would offer understanding. Few would applaud his chosen action. “The problem is often that no good outcomes are available, forcing you to choose among painful paths. Even more troubling, the best path is knowable only after the fact, when the uncertainty is resolved. Long before you know where the bottom would be or how fast things would recover, we had to decide where and how much to cut back, including how many people to furlough or lay off.”

If we cannot judge the quality of the decision by the quality of the outcome, how should we judge it? The key is to understand what information you had (or could have had) at the time you made the decision, and then determine whether given this information you chose a path that would maximize the expected value of your decision. The formal procedure for doing this is to draw a decision tree with nodes for choices and chance events, the probabilities associated with each chance event, and the outcomes associated with each branch of the tree. Yes, the probabilities will not be well defined, as are the probabilities of a coin flip or selecting a card. Assessing how likely you are to break your phone or like San Francisco can only be assessed subjectively. That is a major reason why decision making is difficult. But not thinking hard about probabilities, or worse not attending to them, is the path to poor decisions.

The logic of a decision tree is simple: When faced with a decision, think about the choices you have and for each of these choices think about what outcomes you associate with each of the scenarios that are uncertain at the moment.

Consider a decision tree for whether to buy insurance for your iPhone. Note that for the two alternatives you face (buy insurance or not), you have two scenarios (whether the phone breaks or not) and each scenario has different consequences depending on the choice you made. You can then calculate for each of the two main branches the expected cost (labeled EC). In this hypothetical example, the expected cost of buying insurance ($102.50) is much higher than the expected cost of not buying it ($42.50). Therefore, a decision based solely on expected monetary cost would favor not buying insurance. The ability to absorb a loss and the value placed on avoiding it can also matter. And for these hypothetical figures, if you break your phone, it would be a bad outcome and it would cost you $850, but this does not mean the decision was a bad one.

Note that the expected cost is what you would expect your cost to be on average if you made this decision many times, say if you were to buy 1,000 smartphones. You might object because you are only buying one phone. Over the course of your lifetime, however, you will make thousands of decisions like this one, so thinking of what will happen on average across many decisions is helpful.

The act of drawing the tree allows you to visualize the choices and uncertainty involved, which can be informative and insightful even if you don’t solve the tree as formally as in the smartphone example. Moreover, even if you do not draw a decision tree, thinking in terms of its components can frequently push you toward superior decisions. And this is what David Ellwood did as dean of the Kennedy School in 2008. “We tried to do what Richard would do: We looked at a variety of scenarios, assigned a rough probability to each, considered how we might achieve the best outcome in that situation, and finally worked backwards to ask which decision seemed likely to give us the best outcomes, knowing full well that we would not select the perfect path with 20-20 hindsight. Ironically, knowing that one cannot possibly know the ideal route actually makes it easier to act when action really is essential.”

Given that good decisions can sometimes result in bad outcomes, why should you go through the trouble of setting up and executing good decision-making processes? The answer is that, on average, good decisions are more likely to result in good outcomes than bad decisions. They cannot ensure good outcomes, but they increase the chances of good outcomes. Consistently using a sound process can improve the prospects across many decisions, though any particular run of outcomes still depends partly on luck.

The fact that good decisions sometimes result in bad outcomes should not lead us to assume that our bad outcomes all result from good decisions with an unlucky result. Bad outcomes often flow from bad decisions, particularly when decisions are made more on whim, and impulse outweighs thinking. The key is to evaluate the decision using what was reasonably knowable at the time, while allowing outcomes to inform future judgments, and ask yourself if you could have done better at the time you made the decision with the information you had. As an example, suppose you don’t like to get wet when you go out. On a day with a 90 percent chance of rain forecast, you decide to leave home without an umbrella. It pours; you get very wet. Unless you have something against umbrellas, this was a bad outcome emanating from a bad decision. Even if it hadn’t rained, bringing the umbrella would still have been the right decision.

Nevertheless, knowing that a good decision can sometimes result in a bad outcome should make us more compassionate with ourselves and with others when bad outcomes occur. This idea is beautifully illustrated by Harold Pollack, a professor and urban public health scholar at the University of Chicago, and one of Richard’s coauthors and former students.

Humanizing insights Harold Pollack Because Richard is an unsentimental thinker, it’s easy to overlook his humanizing insights. Good decisions – or at least defensible ones – sometimes produce poor outcomes. A doctor may practice impeccable evidence-based medicine by avoiding (on-average) wasteful scans. On rare occasions, something important will be missed. Out of an abundance of caution, another doctor may order another test, which leads to further intervention, which causes harm. Once life’s lottery has been run, it is difficult to view such sins of omission or commission with the human sympathy their practitioners deserve. I considered this insight when I read about an unfortunate February 1, 2020, tweet by Surgeon General Jerome Adams: “Roses are red. Violets are blue. Risk is low for #coronarvirus. But high for the #flu So get your #FLUSHOT!” Two months later, the nation was in a grip of a deadly pandemic. Viewed from that perspective, the surgeon general's admonition appears tragically complacent and wrong. Viewed from the vantage point of when it was written, his admonition appeared much more defensible and routine. The doctor was reminding people to focus on an important, mundane risk, not to be distracted by a frightening future problem. Many of his critics might easily have sent the same tweet. Richard would remind us that a little human sympathy is in order.

A corollary to this maxim is that bad decisions sometimes have good outcomes. Kessely Hong, a former student of Richard’s and now his colleague at the Harvard Kennedy School, illustrates this vividly: “Following the 2008 financial crisis, my husband and I were humbled when we had a free financial advising session offered by a representative of the company managing my husband’s retirement plan. We had just had our third baby in five years, were utterly exhausted, and had never bothered to adjust his retirement plan investment away from the default choice of a money market account. That ended up being fortuitous, as we were shielded from loss when the stock market underwent a sharp drop in 2008. The financial representative discussed this with us, looked us in the eye, and said, “You weren’t smart, you were lucky.” This reminded me of Richard’s class. He then advised us to include a mix of investments in our retirement plan selections going forward, so that we could look forward to the possibility of growth in the long run.”

Elizabeth Linos, a former student of Richard’s and now a professor at University of California, Berkeley, was working for the Greek government when the prime minister made a controversial decision that was followed by a decade-long financial crisis that rattled the world. “Every time I think back to this painful time for my country, or any time someone criticizes the decision, I tell them Richard’s maxim: all we can do is make the best decision with the information we have at the time we have it. We should judge our political leaders by the decisions they make, not the outcomes themselves. And I’m still convinced that was the right decision, probabilistically speaking.”

Rob Stavins, a colleague of Richard’s at the Harvard Kennedy School for more than thirty years, summarizes this maxim well: “One of my greatest frustrations when reading the newspaper is when someone explicitly or implicitly judges the quality of a decision on the basis of its outcomes, rather than judging the quality of the decision in the context of information that was available at the time. A related pet peeve is when politicians and others claim that an improved economy, or decreased pollutant emissions, or some other raw change, is evidence of a policy success. The comparison that ought to be made is not how things have changed from time A to time B, but rather how things are at time B, compared with how they would have been without the policy.”