The World Is Much More Uncertain Than You Think

The World Is Much More Uncertain Than You Think

The following account describes Dan Levy’s experiences and contributors’ roles at the time of writing.

On Tuesday, November 8, 2016, Donald J. Trump was elected president of the United States. Regardless of your political views, how surprised were you when you heard the news? I have asked this question to hundreds of students at Harvard; a sizable portion were “shocked” or “beyond shocked.” While there are many explanations for their answers, one underlying theme is that they did not think this event could possibly happen.[19] This maxim asks us to examine whether our certainty is justified and whether we have dismissed plausible outcomes too quickly.[20] So the next time you find yourself thinking that some event will happen for sure or that some other event has no chance of happening, pause to remind yourself of this maxim.

COVID-19 illustrates how an event can surprise much of the public even when specialists have warned of the broader risk. At the beginning of 2020, few of us considered the possibility, however remote, that our lives and those of everyone on the planet would be upended by a global pandemic. As Marie-Pascale Grimon, a PhD student at the Harvard Kennedy School who served as a teaching assistant in Richard’s course, remarked a few months into the pandemic “Who would have thought a few months ago that we would soon be facing the greatest pandemic in the world since the 1918 Spanish flu? We make predictions and those are helpful to make decisions, but we should always bear in mind that the world is much more uncertain than you or I think.”

We ignore this maxim at our peril. The dangers stretch even into high-level policy making. Jason Furman, a top economic adviser to U.S. President Obama for eight years and Richard’s Kennedy School faculty colleague, recounts that when he was at the White House, “the best political and legislative aides had only two possible probabilities for events occurring: zero and one. One of the top ones would say ‘I was in Congress for years, and I can tell you there is no chance they will pass this’ – weeks before it passed. Or ‘In all my decades in Washington attaching such and such to the bill has never failed to get it passed,’ just before months and months of failure to get it passed.”

This maxim applies not only to large events of a global scale such as a presidential election or the advent of a pandemic, but also to our personal and professional lives. Think about key events that happened in your life in the past few years and assess what chances you would have assigned to these events happening if someone had asked you even a few years earlier. My guess is that for at least one of these events you would have said “there is no way this could possibly happen.” In my own professional life, if you had asked me in 2004 what were the chances that at some point in my life I would be working on a research project in Burkina Faso, I would have said zero. I grew up in Latin America, was living in Washington DC, and all my research projects were in the United States or Latin America, where I was comfortable with the landscape and the language. Yet less than three years later I was getting on a plane to Ouagadougou, a place I had not even heard of a year earlier.

In a similar vein, Anna Dreber Almenberg, professor of economics at the Stockholm School of Economics and one of Richard’s coauthors, recounts that early in 2008 Richard argued to her when she was a visiting PhD student at Harvard that if you wanted to observe risk taking in the “wild,” the world of bridge (a card game that Richard plays with remarkable skill) is a great setting to do so, given that players must make dozens of probability judgments, at least implicitly, within a single session. “That’s how we ended up at the Fall 2008 North American Bridge Championship in Boston, Massachusetts, collecting data on risk taking in bridge and other domains plus DNA samples to link risk-taking to variation in a dopamine receptor gene. Before meeting Richard, I would have assigned a zero probability to the idea that I would ever attend a bridge championship.” Anna ended up not only going to the Championship but also publishing a paper about this topic with Richard and other coauthors a few years later.[21]

When a low probability event occurs (say an underdog wins a sports championship), we tend to come up with reasons why we might have expected it. This phenomenon is often referred to as hindsight bias, a tendency to perceive past events as having been more predictable than they actually were. But if we consider that many events occur in a year, we should expect at least some to be low-probability events. For example, there are many championships in a given year, so an unlikely championship outcome in some sport (say tennis, golf, football, etc.) need not be surprising in itself.[22]

Recognizing that the world is an uncertain place can improve planning. To illustrate, let’s begin with a historical example provided by William Samuelson, one of Richard’s coauthors and now professor at Boston University.[23] In June 1967, Egypt closed the Suez Canal due to the Arab-Israeli Six Day War. Oil companies scrambled for alternative means of transport, such as chartering additional tankers. As part of its planning, the management team of one oil company insisted that its analysts provide a best-guess target date for when the canal would reopen. The analysts gave a date in July 1969, and the company organized all its transport operations assuming the canal would reopen on that date. As Samuelson recounts, “On the fateful date, the company’s extra charters expired, prevailing charter rates were sky high, and there was no prospect in the near future of the canal reopening.” Planning for a range of reopening dates could have helped the company avoid having all its extra charters expire on the same unconfirmed date.[24]

Rich Krumholz, who learned about the world of risk and insurance when Richard advised his undergraduate thesis, is now a managing partner of a large chain of restaurants. He credits this maxim with guiding how he has operated businesses and invested during his post-college life. “We cannot know what the future may hold, but it is likely that over the course of our lives, we will experience several low frequency but high severity events. After devastating occurrences like 9/11 or the COVID-19 pandemic, people accurately remark that it was nearly impossible to predict the specific event in advance. However, just because we do not know what type of high severity event will happen, does not mean that we should not prepare our personal lives, businesses, and investment portfolios to withstand these types of dramatic events. Human psychology leads most of us to extrapolate the near past too far into the future. It lulls us into assurance after periods of calm. However, if we live based on this maxim – that the world is much more uncertain than you think – we will be alert to potential risks and conservatively positioned when these severe events inevitably happen.”

Carolyn Kousky, a former student of Richard’s and now executive director of the Wharton Risk Center at the University of Pennsylvania, studies risk management for disasters ranging from hurricanes to wildfires to terrorist attacks. She illustrates the benefits of taking this maxim into account in her work: “Post-disaster, there is a strong push to enact new policies that (had they been in place) would have mitigated the specifics of what was just experienced. But the next disaster is never a perfect replica of the previous disaster. We have to keep our field of vision wider, to understand our full range of futures, and prepare policies that will be robust and effective across the spectrum of possible surprises.” One criticism of the response to 9/11 is that it concentrated too heavily on another airplane attack at the expense of other possible threats.

Estimating uncertainty is very challenging, even for people who recognize the world is a very uncertain place. Richard illustrates this in his classes by asking students to estimate some quantities drawn at random from a source such as an almanac, as in the table below.

Estimation Exercise

	Quantity to be estimated	Your 1% surprise point	Your best guess	Your 99% surprise point
1	Birth rate in Andorra 2020 per 1,000 population
2	Air Distance from Amsterdam to Atlanta, in kilometers
3	Number of votes Hillary Clinton got in the 2016 U.S. Presidential Election from voters in the state of Alabama
4	Total area of Afghanistan (in square kilometers)
5	Number of McDonald’s restaurant locations in the United States, in 2020
6	Population of Algeria, in 2020
7	Life Expectancy at birth in Angola, 2020, in years.
8	Total Exports in Argentina in 2017, in US dollars
9	Number of votes that Mitch McConnell received to win his Kentucky Senate seat in the 2020 Elections
10	Total worldwide net sales of Apple in 2020, in U.S. dollars

Most of us are in the dark for most or all of our guesses. The exercise is meant to force us to express the extent of this uncertainty by asking us to indicate not only our best guess but also a range that we feel pretty confident will contain the true value of the quantity.

Try doing this exercise yourself before proceeding. Fill in the table above or write the numbers on a sheet of paper (without consulting the internet, of course). The 1 percent surprise point is a value below which you assign only a small chance to the true quantity; it is not an absolute lower bound. If you were making 100 of these estimations, only once should the true value be below the 1 percent surprise point. Similarly, the 99 percent surprise point is an upper quantile, not the highest possible value; out of 100 such estimations, only once should the true value be above the 99% surprise point. There should only be a 2 percent chance that the true quantity is outside the range formed by the surprise points (1 percent chance below and 1 percent chance above).

By this stage of the course, students know that the world is very uncertain and that we tend to understate the uncertainty in our predictions and beliefs. So they understand that Richard is out to show them that they will not sufficiently recognize their own level of uncertainty about these quantities. To provide incentives for reasonable guessing, Richard offers a prize, usually a book, to the student with the best guesses. Good assessments combine useful ranges with coverage that matches the stated probabilities; even well-calibrated assessments can occasionally produce a surprise.

Despite all this, students tend to choose intervals that are too narrow, reflecting overconfidence in their own beliefs. In a typical year 30 percent of the answers fall outside students’ 1%–99% surprise range. Remember, if students were correctly estimating their own uncertainty, only 2% of the answers should be surprises. This means that students are very overconfident about their own knowledge.

Answers to the questions in the table are at the end of this maxim. Make sure you have filled in the table before you look. How many of the correct quantities were outside the ranges you set (either below your 1 percent surprise point or above your 99 percent surprise point)? If more than one, examine whether you made the ranges too narrow. A small exercise can suggest a calibration problem, though it cannot establish a person’s overall accuracy.

Experience with estimating probabilities reduces but does not eliminate the penchant for underestimating uncertainty. When students complete the exercise again with a new set of questions, they are surprised about half as often. Jason Furman argues that White House economists, who are used to thinking in probabilistic terms, are typically better than the legislative aides mentioned earlier at recognizing uncertainty. He says that when asked to predict the upcoming jobs number, they would stress uncertainty and give a 90 percent confidence interval. And yet “about 75 percent of the time, I would guess, the number ended up falling within that 90 percent confidence interval (so even they were not sufficiently internalizing the degree of uncertainty).”

There are some endeavors where uncertainty is so large that it becomes central to our understanding of the issue. Bill Hogan, an expert on energy policy and Richard’s Kennedy School colleague for more than four decades, wrote back in 1985: “A policy that depends on accurate oil price forecasts is a policy in trouble…the range of uncertainty is impressive; it is so large that the uncertainty may be the most important feature of the analysis.”

In medicine, uncertainty looms large although this is not always recognized by the people involved. Surgeon and writer Atul Gawande describes the problem: “The core predicament of medicine — the thing that makes being a patient so wrenching, being a doctor so difficult, and being a part of a society that pays the bills they run up so vexing — is uncertainty. With all that we know nowadays about people and diseases and how to diagnose and treat them, it can be hard to see this, hard to grasp how deeply the uncertainty runs. As a doctor, you come to find, however, that the struggle for caring for people is more often with what you do not know than what you do. Medicine’s ground state is uncertainty. And wisdom — for both patients and doctors — is defined by how one copes with it.”[25]

In thinking about these matters, it is helpful to distinguish between risk, uncertainty and ignorance (see table below). We face risk in situations with known probabilities and known states of the world. For example, if you play roulette in a casino, the states of the world (represented by all the possible slots the ball can land on) are known, and the probability associated with each state (the likelihood that the ball will land in any specific slot) is also known. As Richard has written “Casinos, which rely on dice, cards and mechanical devices, and insurance companies, blessed with vast stockpiles of data, have good reason to think about risk. But most of us have to worry about risk only if we are foolish enough to dally at those casinos or to buy lottery cards to a significant extent.”[26]

Categorization of situations where outcomes are unknown

Probabilities	Known states of the world	Unknown states of the world
Known	Risk	Not applicable
Unknown	Uncertainty	Ignorance

Source: Richard Zeckhauser

When the possible states of the world are known, but the probabilities that those states occur are unknown, we face uncertainty. Uncertainty is a much more common occurrence for most of us than risk. For example, when you are trying to decide which medical treatment to undertake, you might have a good sense of the possible states of the world (cured vs. not cured) for each of the treatments you are considering, but you will not know for sure the probabilities associated with each of these outcomes and will have to make educated guesses about them. In Richard’s words: “Uncertainty, not risk, is the difficulty regularly before us.”

Richard has also written extensively about ignorance, a situation involving possibilities the decision maker has not identified.[27] In situations with unknown probabilities and unknown states of the world, we are simply ignorant. This is the most extreme form of uncertainty. Many earlier examples concern events we could conceive of but judged exceedingly unlikely. Ignorance concerns outcomes missing from a decision maker’s view of the possibilities. The attack on 9/11 illustrates this for people who had not considered such a use of passenger planes or the possibility of the towers collapsing; it does not mean that nobody had imagined those threats. Richard says, “Ignorance is an important phenomenon, I would argue, ranking alongside uncertainty, and way above risk. Ignorance achieves its importance, not only by being widespread, but also by involving outcomes of great consequence.”

Richard also explored ignorance in literature with Devjani Roy, an English PhD who did a postdoctoral fellowship with Richard at the Harvard Kennedy School and is now at Texas Tech University Health Sciences Center. In this work, they explored ignorance in the world of literature.

Ignorance in literature Devjani Roy In the summer of 2013, Richard and I were writing the initial draft of what became our work on ignorance. We were discussing Leo Tolstoy’s novel Anna Karenina, in which Anna, wife to the well-respected, if dour, Alexei Karenin, indulges in a ruinous adulterous affair with the young cavalry officer Count Vronsky. Our discussion veered toward marriage as a decision-making situation; more specifically, we discussed how marriages such as Anna’s, with the stolid and unimaginative Karenin, take place with no thought to future compatibility over a lifetime. Richard then remarked that marriage is one of life’s most important decisions, possibly one with the greatest impact on health, happiness, and material well-being; yet, when we make the decision, we typically have the least available or the least salient information to enable a robust decision. Marriage, then, is a situation of ignorance. “Consequential Amazing Developments” (or CADs, to use Richard’s term) are important outcomes the ignorant decision maker could not have envisioned – such as how Anna’s chance encounter with Vronsky at the Moscow railway station that sets off Tolstoy’s novel. I do not know if Anna, impulsive and reckless, would recognize the importance of ignorance in her life. But we all should. Indeed, we might do well to pay attention to Richard’s reflections on information scarcity as we enter marriage and extend these to other important areas of our lives.

Even Richard, who understands the maxim “The world is much more uncertain than you think” better than anyone I know, can underestimate the extent of uncertainty in the world. Jeff Bielicki, a former student of Richard’s and now a professor at Ohio State University, recalls a conversation with Richard in the fall of 2004 when he was his teaching assistant. At this time, the Boston Red Sox, the baseball team beloved by most people in the Boston area including Richard, were doing well during the post-season. But then they lost two games of the American League Championship Series against their arch-rivals, the New York Yankees. “Richard was a bit despondent about the prospects of the Red Sox making it to the World Series. I mentioned that I didn’t think the prospects were as poor as he was stating them, in part justified by how uncharacteristically well many of the Red Sox players had been performing. My case was not helped when the Yankees subsequently clobbered the Red Sox 19-8 (and at Fenway Park!). But the Red Sox turned it around: they won four straight games, advanced to the World Series, and ultimately won it – breaking the 86-year Curse of the Bambino.”[28]

Answers to the estimation exercise [29]

	Quantity to be estimated	Correct answer	Was the correct answer below your 1% surprise point or above your 99% surprise point?
1	Birth rate in Andorra 2020 per 1,000 population	7 births/1000
2	Air Distance from Amsterdam to Atlanta, in kilometers	7,082
3	Number of votes Hillary Clinton got in the 2016 U.S. Presidential Election from voters in the state of Alabama	729,547
4	Total area of Afghanistan (in square kilometers)	652,230
5	Number of McDonald’s restaurant locations in the United States, in 2020	13,226
6	Population of Algeria, in 2020	42,972,878
7	Life Expectancy at birth in Angola, 2020, in years.	61.3
8	Total Exports in Argentina in 2017, in US dollars	$58.45 billion
9	Number of votes that Mitch McConnell received to win his Kentucky Senate seat in the 2020 Elections	1,222,749
10	Total worldwide net sales of Apple in 2020, in U.S. dollars	$274.52 billion