Odds and N s

7. Odds and Ns

In 2007, Governor Rick Perry of Texas issued an executive order mandating that all twelve-year-old girls in Texas receive a vaccination for human papillomavirus, which can result in cervical cancer. In her effort to score points on Rick Perry in the Republican primary of 2012, the candidate Michele Bachmann announced that a woman had told her that her “little daughter took that vaccine, that injection, and she suffered from mental retardation thereafter.” What was wrong with Bachmann’s inference—or at least her invitation to us to infer—that HPV vaccinations cause mental retardation? Let us count the ways.

We need to think of Bachmann’s evidence as a report about a sample of the population of all twelve-year-old girls in the United States who got the vaccination. One reported case of intellectual disability constitutes a very small sample (low N) and isn’t remotely enough to establish that the population of girls receiving the injection was at risk from it.

In fact, there have been several rigorous randomized control studies with follow-ups in which girls were randomly selected either to be inoculated or not. These studies all had a very large N—number of cases. In none of those studies were vaccinated girls found to have a higher rate of intellectual disability than girls who were not vaccinated.

Bachmann’s sample of twelve-year-old girls who had the vaccination consisted of one case—an example of reliance on the Man Who statistic, as in “I know a man who.” Bachmann’s sample was haphazard at best rather than random. The closer that sample-selection procedures approach simple random sampling, in which each possible sample of the specified size has an equal chance of being selected—the more we should trust them. If we don’t know whether a sample is random, any statistical measure we conduct may be biased in some unknown way.

Actually, Bachmann’s sample is not even as good as a haphazard one. Assuming Bachmann was telling the truth, she had a strong motive to present this one case to the public. And she may not have been telling the truth, or her informant may not have been telling the truth. Which is not to say that the informant was lying. She may well have believed what she allegedly told Bachmann.

If her daughter had the vaccination and then received a diagnosis of intellectual disability, it’s possible that the mother’s reasoning is an instance of the post hoc ergo propter hoc error: after this, therefore because of this. The fact that thing 1 precedes thing 2 doesn’t mean it necessarily caused thing 2. In any case, it seems to me that we should regard Bachmann’s claim as not quite reaching the very low bar set by the Man Who statistic.

One of my favorite examples of the post hoc ergo propter hoc error combined with the Man Who statistic was provided to me by a friend who overheard the following conversation between two elderly men. First man: “My doctor told me I have to quit smoking or I’ll die from it.” Second man: “No! Don’t quit! I have two friends who quit smoking because their doctors told them to and they were both dead within a few months.”

Sample and Population

Consider the hospital problem discussed in Chapter 1. Smaller hospitals will have more days than larger hospitals when the number of baby boys is greater than 60 percent of the total births. The only way to know that is to understand the law of large numbers: under suitable sampling conditions, means and proportions tend toward their population values as N, the sample size, grows. A particular larger sample need not be closer than every smaller one. At the extremes of population size, it’s easy to see the force of the law of large numbers. Suppose there were ten births at a given hospital on a particular day. How likely would it be that 60 percent or more of the births were male? Quite a good chance, of course. We wouldn’t be suspicious of a coin that came up heads six times out of ten. Suppose there were two hundred births at another hospital on a particular day. How likely would that deviant value be? Extremely unlikely, obviously. That’s exactly comparable to tossing an allegedly fair coin two hundred times and coming up with 120 or more heads instead of the expected one hundred.

As an aside, I’ll note that the accuracy of a sample statistic (mean, median, standard deviation, etc.) is essentially independent of the size of the population the sample is drawn from. Most national polls in elections sample about a thousand people, for which they claim a margin of sampling error of about + or − 3 percentage points. A sample of a thousand people gives about as good an estimate of the exact percentage supporting a given candidate when the population size is 100 million as when the population size is ten thousand. So when your candidate is ahead by eight points, don’t be concerned when the other candidate’s campaign manager pooh-poohs the poll, saying millions of people will vote and the poll sampled only a thousand people. The small fraction of the population sampled is not by itself a good objection. Representativeness, nonresponse, turnout assumptions, and the time remaining before the election still matter.

Which brings us to sample bias. Even an unbiased sampling procedure produces random error. Bias arises when the procedure systematically favors some observations in a way that distorts the estimate. A large sample does not remove that distortion. If you’re trying to find out what proportion of the workers in a factory would like to have flexible work time and you sample only men, or only people who work in the cafeteria, those people could be different in some important respect from the factory population as a whole, giving an erroneous estimate of the proportion of workers who favor flex time. With biased sampling, increasing the sample can reduce random variability while leaving the systematic error in place.

It has to be noted that the national polls don’t actually sample randomly from the population. In order for that to be the case, every voter in the country would have to have an equal chance of being in the sample. When this is not the case, you run the risk of serious bias.

One of the very first national election polls in the United States, conducted by the now defunct Literary Digest, reported that Franklin Roosevelt was about to lose the 1936 election, which he in fact won in a landslide. The Digest’s problem? Its mailed poll drew heavily on lists such as telephone directories and automobile registrations, and respondents selected themselves by returning it. The people reached and those who replied did not adequately represent the electorate.

A similar source of bias characterized some of the polls in the 2012 election. The Rasmussen polling firm did not call cell phone lines, neglecting the fact that people who have only cell phones are disproportionately likely to be young and Democrat-leaning. The Rasmussen poll systematically overestimated support for Romney compared to polls that sampled both landline numbers and cell phone numbers.

Once upon a time, when people answered their phones and opened their doors to interviewers, you could come close to a random sample of the population. For the polls discussed here, accuracy depended in part on the pollster’s data and intuitions about how to doctor the sample—weighting it by stirring into the brew of poll numbers the likelihood that respondents will vote, respondents’ party identification, gender, or age, how the community or region has voted in the past, eye of newt, toe of frog …

Finding the True Score

Consider the following pair of problems. University X has a renowned musical theater program. The program awards scholarships to a small number of high school graduates who show unusual promise. Jane, the director of the musical theater program, has friends who are drama teachers at high schools in the area. One afternoon she goes to Springfield High to watch a student who has been highly touted by her teachers as a superb young actress. She sees a rehearsal for a Rodgers and Hammerstein musical in which the student has the lead role. The student flubs several lines, seems not to have the correct conception of the character she’s playing, and comes across as having little stage presence. The director tells her colleagues that she now doubts the judgments of her teacher friends at the high school. Is this a wise conclusion or not?

Joe is a talent scout for the football team at university Y. He visits high school practice sessions all over the state, looking for prospects whose coaches have told Joe he ought to consider them for his team. One afternoon he goes to Springfield High to see a quarterback who has an excellent win-loss record, has impressive TD and completion-percentage statistics, and is highly praised by his coaches. At the practice, the quarterback misthrows several passes, is sacked a few times, and gains little yardage overall. The scout’s report states that the quarterback has been overrated and recommends that the university cease to consider him for recruitment. Is this a wise recommendation or not?

If you said that Jane is wise and Joe is not, it’s a fair bet that you’re knowledgeable about sports but not dramatics. If you said Joe is wise and Jane is not, it’s a fair bet that you’re knowledgeable about dramatics but not sports. I find that people who don’t know much about sports often say Joe is probably right that the quarterback may not be all that talented, but people who do know sports are likely to think Joe may be being too hasty.

They recognize the possibility that Joe’s (rather small) sample of the quarterback’s behavior may be an extreme performance and that there is a strong possibility that the quarterback’s ability is closer to Joe’s informants’ evaluation than to his own. People who don’t know much about dramatics are likely to say that the actress probably isn’t very good, but people who do know dramatics think Jane may be being too dismissive of the judgments of Jane’s friends at the high school. Other things being equal, the more you know about a given domain, the more likely you are to be able to use statistical concepts to think about that domain. In this case, the important concept is the law of large numbers.

Here’s why the law of large numbers is relevant. A quarterback’s performance over one or more seasons can be assumed to be a pretty reliable indicator of his skill. If his coaches back that up with the insistence that he is really excellent, we have a great deal of evidence—lots of data points—indicating that the quarterback that Joe observes really is extremely good. Joe’s evidence is pretty piddling by comparison: one set of observations on one day. The inherent variability of a player’s performance, even the inherent variability of the performance of an entire team, is recognized in the adage that on any given Sunday, any team in the NFL can beat any other team in the NFL. This certainly doesn’t mean that all teams are equally talented, it just means that you need a fairly large sample of behavior to determine with confidence the relative skill of different teams.

The same logic applies to the theater program director’s opinion of the actress she sees. If several people who know the actress well say that she’s highly talented, then the director ought to place relatively little weight on the sample she has. I find that very few people recognize this, except for people who have done some acting and are well aware of the variability of performance in that field. In his autobiography, the comedian and actor Steve Martin writes that almost any comedian can be great some of the time. The successful ones are those who can be at least good all the time.

In statistical lingo, the coach and the theater program director are trying to determine the true score of the candidate they are observing. The observation = true score + error. This is the case for measurements of all kinds, even the height of people or the temperature of the atmosphere. There are two ways to improve the accuracy of the score. One is to get better kinds of observations—better yardsticks or thermometers. The other is to “cancel out” random errors by averaging enough suitably independent observations. This helps when errors vary around the target; repeating a systematically wrong measurement does not correct it.

The Interview Illusion

Even when we’re highly knowledgeable about some domain, and highly knowledgeable about statistics, we’re likely to forget the concept of variability and the relevance of the law of large numbers. In the practice described here, the Department of Psychology at the University of Michigan interviewed its top applicants for graduate study before it makes a final decision as to whether to admit a student. My colleagues tend to put substantial weight on their twenty- to thirty-minute interviews with each candidate. “I don’t think she’s a good bet. She didn’t seem to be very engaged with the issues we were discussing.” “He looks like a pretty sure thing to me. He told me about his excellent honors thesis and made it clear he really understands how to do research.”

The problem here is that judgments about a person based on small samples of behavior are being allowed to weigh significantly against the balance of a much larger amount of evidence, including college grade point average, which summarizes behavior over four years in thirty or more academic courses; Graduate Record Exam (GRE) scores, which are in part a reflection of how much a person has learned over twelve years of schooling and in part a reflection of general intellectual ability; and letters of recommendation, which typically are based on many hours of contact with the student.

In fact, the studies discussed here found that college grade point average predicted performance in graduate school to a significant degree [1] (a correlation of about .3, a modest relationship), and GRE scores also predict performance to about the same degree. And the two are somewhat independent of each other, so using them both improves prediction above the level of each separately. Using letters of recommendation adds yet a bit more to the accuracy of predictions.

The brief, unstructured interview judgments criticized here had correlations below .10 with later performance across several groups studied, including students, army officers, businesspeople, and Peace Corps volunteers. That is a weak relationship; it does not establish that every interview format is equally uninformative.

It wouldn’t be so bad if people gave the interview as much weight as it deserves, which is little more than to let it be a tiebreaker, but people characteristically undermine the accuracy of their predictions by overweighting the value of the interview relative to the value of other, more substantial information. In fact, people overweight the value of the interview so much that they’re likely to get things backward. They think an interview is a better guide to academic performance in college than is high school GPA, and they think an interview is a better guide to quality of performance in the Peace Corps than letters of recommendation based on many hours of observation of the candidate. 2

The practical warning is to keep a brief, unstructured interview from overruling substantial evidence in the candidate’s record. A common set of relevant questions and explicit scoring criteria can also keep an interview from becoming only a test of first impressions. If you could weight the interview as little as it deserves, that wouldn’t be true, but it’s almost impossible not to overweight it because we tend to be so unjustifiably confident that our observations give us very good information about a person’s abilities and traits. It’s as if we regard the impression we have of someone we’ve interviewed as resulting from an examination of a hologram of the person—a little smaller and fuzzier to be sure, but nevertheless a representation of the whole person. We ought to be thinking about the interview as a very small, fragmentary, and quite possibly biased sample of all the information that exists about the person. Think of the blind men and the elephant, and try to force yourself to believe you’re one of those blind men.

Note that the interview illusion and the fundamental attribution error (FAE) are cut from the same cloth, and both are amplified by our failure to pay sufficient attention to the quantity of evidence that we have about a person. A better comprehension of the FAE, namely that we overestimate the relevance of stable dispositions relative to situations, would lead us to be dubious about how much we can learn from an interview. A firmer grasp of the law of large numbers makes us less vulnerable to both the FAE and the interview illusion.

I wish I could say that my knowledge of the utility of interviews always makes me skeptical about the validity of my own conclusions based on interviews. My understanding of that principle has a limited dampening effect, however. The illusion that I have valuable and trustworthy knowledge is too powerful. I just have to remind myself not to weight the interview—or any brief exposure to a person—very heavily. This is especially important when I have presumably solid information about the person based on other people’s opinions formed after long acquaintance with the candidate, as well as records of academic or job achievements. I have no difficulty, however, in remembering the limitations of your judgments based on a brief interview!

Dispersion and Regression

I have a friend—let’s call her Catherine—who consults with hospitals about management practices. She loves her job in part because she likes to travel and meet new people. She’s something of a gourmet and enjoys going to restaurants that she has reason to believe are very good ones. But she says that she’s typically disappointed when she goes back to restaurants that she initially thought were excellent. Second meals rarely seem as good to her as the first. Why do you suppose this is? If you said “maybe the chefs change a lot” or “maybe her expectations are so high that she’s likely to be disappointed,” you’re ignoring some important statistical considerations.

A statistical approach to the problem would start by noting that, after all, there is a chance element to how good a meal Catherine gets at any given restaurant on any given occasion. For any individual sampling a given restaurant on different occasions, or for a group of diners who eat at a given restaurant at a given time, there will be variation in judged meal quality. That first meal Catherine gets at the restaurant could be anywhere from so-so (or even worse) to fabulous. This variation is what makes us refer to judged meal quality as a variable. A numerical variable has a distribution, and, when its mean is defined, we can describe how observations vary around it. Continuous measurements, such as height, differ from categories such as political affiliation; neither distinction by itself guarantees a bell-shaped distribution. Given that fact alone, it’s not surprising that Catherine is often disappointed: it’s a virtual certainty that some of the time her second experience of a restaurant is going to be worse than the first (as well as some occasions when the second meal is better than the first).

But we can say more than that. If Catherine returns because a first meal was unusually good, and chance helped make it exceptional, the next meal is expected to be less extreme. This is a prediction about selected excellent experiences, not a guarantee of disappointment. An outstanding meal on occasion 1 can reflect both the restaurant’s usual quality and favorable circumstances. Regression does not require a normal distribution. A normal distribution, often called a bell curve, is a separate useful model, discussed with Figure 2.

The normal distribution is a mathematical abstraction, but it’s approximated surprisingly often for continuous variables—some measurement distributions and standardized test scores have approximately this shape. Combining many small, sufficiently independent effects is one reason normal approximations can arise. Counts of eggs or manufacturing errors need not be normally distributed, so the fit should be checked.

There are a number of ways of describing the dispersion of cases around the mean. One is the range—the highest value in the available cases minus the lowest value. A more useful measure of dispersion is the average deviation from the mean. If the average quality of meals that Catherine assigns to the first meal in each of the restaurants she eats in in different cities is, say, “pretty good,” and the average deviation from that mean is equal to, say, “very good” on the plus side, and just “fairly good” on the minus side, we would say that the degree of dispersion—the average deviation—around Catherine’s mean of first-meal-quality judgments is not very great. If the average deviation ranged from “superb” on the plus side to “rather mediocre” on the minus side, we’d say that the dispersion was quite large.

But there’s a much more useful measure of dispersion that we can calculate for any variable that can be given continuous numerical values. This is the standard deviation (SD), often represented by the Greek letter sigma, σ. The standard deviation is (essentially) the square root of the average of the squared distance of each observation from the mean. Conceptually it’s not all that different from the average deviation, but the standard deviation has some extremely useful properties. Figure 2, not reproduced here, marks standard deviations on a normal curve.

About 68 percent of values are within plus and minus one standard deviation from the mean. As an example, consider IQ test scores. Most IQ tests are scored so that the mean is arbitrarily set at 100 and the standard deviation at 15. Someone with an IQ of 115 is a standard deviation above the mean. The distance between the mean and one standard deviation above the mean is pretty large. An IQ of 115 places a score above the standardized mean; it does not by itself determine education or occupation. Likewise, a score of 100 locates someone on the scale without fixing what that person can learn or do.

Figure 2 caption: Distribution of IQ scores around the mean of 100, with corresponding standard deviations and percentile ranks. The figure itself is not included in this article.

For a normal distribution, percentiles correspond to standard deviations in a predictable way. About 84 percent of all observations are less than one standard deviation (SD) above the mean. An observation one SD above the mean is approximately at the eighty-fourth percentile. About sixteen percent of all observations lie above it. Almost 98 percent of all observations lie below two SDs above the mean. A score two SDs above the mean is approximately at the 98th percentile. Just over 2 percent of all observations lie above it. Nearly all observations will fall between three SDs below the mean and three SDs above it.

The relation between these percentages and standard deviations applies when the normal model fits. Continuity alone is not enough. For example, the standard deviation is a measure frequently used in finance. The standard deviation of the rate of return on an investment is a measure of the volatility of the investment. If a given stock has averaged a 4 percent rate of return for the past ten years, with a standard deviation of 3 percent, a stable normal model would place about 68 percent of future observations the rate of return will be between 1 percent and 7 percent, and about 95 percent of the time it will be greater than −2 and less than 10 percent. Those percentages are properties of the assumed model, not guarantees. A short return history cannot establish that future returns will have the same distribution.

If the standard deviation is 8, the same normal model puts about 68 percent of returns between − 4 percent and + 12 percent. You could do very well indeed with the stock. Sixteen percent of the time you’ll get more than a +12 percent return. On the other hand, 16 percent of the time you’ll lose more than 4 percent. That’s fairly volatile. And 2 percent of the time you will lose more than 12 percent; 2 percent of the time you will gain more than 20 percent. You could make a killing or lose your shirt.

A stock might have a dividend yield of 2, 3, or 4 percent, but that does not establish how much its price can fall. Labels such as value and growth do not guarantee low risk or future growth. The point of the illustration is to compare variation under stated assumptions, not to choose investments by age or by a normal-curve calculation alone.

Normal distributions all have the same shape when expressed in standard-deviation units. Other distributions can have different tail behavior. A very peaked curve may resemble a rocket ship in a 1930s comic book, but the peak alone does not describe its tails. Leptokurtic distributions have higher kurtosis than a normal distribution; platykurtic distributions have lower kurtosis. Neither category guarantees that 68 percent of observations lie within one standard deviation. That percentage belongs to the normal model.

But back to our question of why it is that Catherine is typically disappointed when she returns to a restaurant where she got an excellent meal. We’ve agreed that her evaluation of restaurant meals is a variable: it ranges from, say, execrable (1st percentile) to, say, ambrosial (99th percentile). Let’s say that an excellent meal is one that is about at Catherine’s 95th percentile or higher—better than about 94 percent of the meals she eats. Now ask yourself the following question about your own meal experiences. Do you think it’s more likely that every meal you might happen to eat in a restaurant where you haven’t eaten before would be excellent, or that only some meals would be excellent? If you think that you wouldn’t expect all meals to be excellent, and you happen to get an excellent one the first time, then the expected value of a second meal is at least slightly lower than the quality of that excellent first meal. Catherine’s second-meal experiences can illustrate regression to the mean if her first meal was selected for being exceptional and repeated meal quality is imperfectly related. Less extreme follow-ups are then expected on average; they are not inevitable for every restaurant.

Regression effects are lying in plain sight everywhere. Why is the baseball rookie of the year so often a disappointment the second year? Regression. The rookie may have benefited from unusually favorable conditions as well as skill, making some decline unsurprising without making it inevitable. Why is the stock that grew in value more than any other in year 1 so often a mediocre performer or worse in year 2? Regression. Why does the worst-performing kid in the third grade perform a little better the next year? Regression. None of this is to say that the only thing going on is regression. It’s not that the mean of a distribution is a black hole that sucks all extreme observations into it. Any number of other things could be operating to push performance level up or down. But in the absence of knowing exactly what those things are, we have to recognize that extreme scores are usually followed by less extreme ones because the combination of forces resulting in extreme values is not likely to sustain itself over time and trials. The rookie of the year happened to have a coach who was having an unusually good year; he played against relatively weak opponents in his first few games, which built his confidence; he just got engaged to the girl of his dreams; his health was perfect; he had no injuries that impaired his play, and so on. The next year an injured elbow kept him out of several games; the coach moved to another team; there was a serious illness in his family. Or whatever. And there will always be whatevers.

Two questions for which the regression principle is (surprisingly) relevant: (1) What is the likelihood that an American aged twenty-five to sixty will have an income placing the person in the top 1 percent at least once during those years? (2) What is the likelihood that the person will be in the top 1 percent for ten consecutive years? The chances that a person will have a top 1 percent income once are over 110 in 1,000. Bet you wouldn’t have guessed that. The chances that a person will do that ten years in a row are 6 in 1,000. Very surprising given the odds for one year. These historical figures, as reported here, seem surprising because we don’t spontaneously think about a value like income as being highly variable and thus susceptible to substantial regression effects. But there is in fact a great deal of variability for an individual’s income from year to year (especially at the high end of the income distribution). Extreme incomes are surprisingly common in the population as a whole. But precisely because they are extreme, such incomes are not likely to be repeated all that frequently. The great majority of those much-resented 1 percenters are on their way down, so you may want to take it easy on them!

The same kinds of figures apply for low incomes. More than 50 percent of Americans will be impoverished or nearly so at least once in their lives; conversely, not all that many people are in poverty forever. People perennially on the dole are a rarity. The great majority of people who are ever on welfare are there only for a couple of years or so. 3 You may want to take it a little easier on people on welfare, too.

We can make some pretty serious mistakes by failing to conceptualize events in terms of the possibility of regression to the mean. The psychologist Daniel Kahneman once told a group of Israeli flight instructors that praise was more effective than criticism at changing someone’s behavior in the direction that’s desired. 4 One of the instructors contradicted him, saying that praise of a pilot’s execution of a maneuver seemed to make the pilot worse at that maneuver, whereas yelling at the pilot for a poor execution resulted in improvement the next time around. But the instructor had failed to pay sufficient attention to the fact that novice pilot performance is a variable, and regression to the mean is expected after an especially good performance—or an especially bad performance. An execution that’s better than average is expected on probabilistic grounds alone to be followed by something closer to the mean, that is, worse. An execution worse than average is expected to get better. This pattern alone cannot establish whether the instructor’s criticism helped or hurt. To assess the teaching method, he would need evidence beyond the tendency for unusually good or bad performances to be followed by less extreme ones.

The flight instructor’s error gets a boost from a two-edged cognitive sword we all carry around. We are superb causal-hypothesis generators. Given an effect, we are rarely at a loss for an explanation. Seeing a difference in observations over time, we readily come up with a causal interpretation. A change need not be an effect of the intervention under discussion; random variation can also account for it. The compulsion to explain is particularly strong when we habitually see that one event typically occurs in conjunction with another event. Seeing such a correlation almost automatically provokes a causal explanation. It’s tremendously useful to be on our toes looking for causal relationships that explain our world. But there are two problems: (1) The explanations come too easily. If we recognized how facile our causal hypotheses were, we’d place less confidence in them. (2) Much of the time, no causal interpretation at all is appropriate and wouldn’t even be made if we had a better understanding of randomness.

Let’s try a couple of other applications of the regression principle. If the mother of a child has an IQ of 140 and the father has an IQ of 120, what is your best guess about what the IQ of the child would be? 160 155 150 145 140 135 130 125 120 115 110 105 100

Psychotherapists speak of a hello/goodbye effect for many patients. Patients report that their condition is worse than it really is before beginning therapy and report that their condition is better than it is at the end of therapy. Why might this be?

If you said the expected value for the child’s IQ—given that one parent’s IQ is 140 and the other’s is 120—is 140 or higher, you haven’t taken into consideration the phenomenon of regression to the mean. An IQ of 120 is higher than the mean and an IQ of 140 is higher still. Unless you think the correlation between parents’ IQ and child’s IQ is perfect, a linear prediction generally moves toward the population mean rather than reproducing the parents’ average. Using .50 as the regression coefficient on this scale gives a prediction halfway between the midparent value and the population mean, namely 115. A correlation alone would not determine that slope without the relevant standard deviations. Supersmart parents are going to have kids who are merely smart on average. And supersmart kids are going to have parents who are merely smart on average. Regression works in both directions.

The usual explanation of the hello/goodbye phenomenon is that patients fake bad in order to be eligible for therapy but want to ingratiate themselves with the therapist at the end of therapy. Regardless of whether there is some truth to this explanation, we expect patients to be better at the end of therapy than at the beginning because they’re probably less emotionally healthy than usual at the time they seek therapy, and because the mere passage of time makes regression to the mean likely. You would expect the hello/goodbye effect in the absence of any treatment at all. In fact, doctors of all kinds generally have time on their side: some conditions improve naturally, while others persist or worsen. A change after treatment therefore needs a suitable comparison before it can be credited to the treatment. For that matter, any intervention at all has a good shot at being deemed effective. “I had some dandelion soup and my cold cleared right up.” “My wife took agave root extract as soon as she got the flu and she had it for half as long as I did.” The combination of the Man Who statistic plus the post hoc ergo propter hoc heuristic has made rich many a manufacturer of nostrums. And they can truthfully claim that the great majority of people got better after taking their remedies. These examples connect sampling variability with correlation: how strongly repeated measurements or related traits tend to move together.

Summing Up

Observations of objects or events should often be thought of as samples of a population. Meal quality at a given restaurant on a given occasion, performances of a given athlete in a particular game, how rainy it was during the week we spent in London, how nice the person we met at the party seems to be—these all have to be regarded as samples from a population. And all assessments that are pertinent to such variables are subject to error of some degree or other. The larger the sample, other things being equal, the more the errors will cancel one another out and bring us closer to the true score of the population.

The law of large numbers applies to events that are hard to attach a number to just as much as to events that can readily be coded.

The fundamental attribution error is primarily due to our tendency to ignore situational factors, but this is compounded by our failure to recognize that a brief exposure to a person constitutes a small sample of a person’s behavior. The two errors lie behind the interview illusion—our drastic overconfidence that we know what a person is like given what the person said or did in a thirty-minute encounter.

Increasing sample size reduces random sampling variation under suitable conditions; it does not remove systematic bias. A well-designed probability sample makes the selection process explicit and supports appropriate estimates. At the very least we have to be attentive to the possibility of sample bias: Was I relaxed and in pleasant company when I was with Jane at Chez Pierre or was I uptight because my judgmental sister-in-law was also there? A larger biased sample can leave us with a more precise estimate of the wrong target.

The standard deviation is a handy measure of the dispersion of a continuous variable around the mean. The larger the standard deviation for a given type of observation, the less confident we can be that a particular observation will be close to the mean of the population of observations. A big standard deviation for a type of investment means greater uncertainty about its value in the future.

If we know that an observation of a particular kind of variable comes from the extreme end of the distribution of that variable, then it’s likely that additional observations are going to be less extreme. The student who gets the highest grade on the last exam is probably going to do very well indeed on the next exam, but isn’t likely to be the one who gets the highest grade. The ten stocks with the highest performance in a given industry last year are not likely to constitute the top ten this year. Extreme scores on any dimension are extreme because the stars aligned themselves just right (or just wrong). Those stars are probably not going to be in the same position next time around.