The Gendarme's Hat

The Gendarme's Hat

By the early eighteenth century, Jakob Bernoulli had worked out a precise statement and mathematical proof of the Law of Large Numbers. It was now no longer an observation but a theorem.

Imagine a coin-flipping contest: each Big player flips many coins, each Small player only a few, and the highest proportion of heads wins. The theorem helps explain why this Big-Small game isn’t fair. With fair, independent flips, the Big players’ scores tend to cluster more closely around 50%, while those of the Smalls are apt to vary much more widely. But it would be nuts to conclude that the Small team is “better” at flipping heads, even when that team supplies the winner. For if you average the proportion of heads flipped by all the Small players, not just the top scorer, they’ll likely be at just about 50%, same as the Bigs. And if we look for the player with the fewest heads instead of the most, Team Small suddenly looks bad at getting heads: it’s very likely one of their players will have only 20% heads, and the Big players are much less likely to score that badly. Scoring by raw number of heads gives the Big team an insuperable advantage; but using percentages slants the game just as badly in favor of the Smalls. The smaller the number of coins—what we’d call in statistics the sample size—the greater the variation in the proportion of heads.

It’s the very same effect that makes political polls less reliable when fewer voters are polled. And it’s the same, too, for brain cancer. Small states have small sample sizes—they are thin reeds whipped around by the winds of chance, while the big states are grand old oaks that barely bend. Measuring the absolute number of brain cancer deaths is biased toward the big states; but measuring the highest rates—or the lowest ones!—puts the smallest states in the lead. That’s how South Dakota can have one of the highest rates of brain cancer death while North Dakota claims one of the lowest. Small-population variation can produce that pattern without Mount Rushmore or Wall Drug being toxic to the brain. The ranking alone does not establish an environmental cause.

That’s a mathematical fact you already know, even if you don’t know you already know it. Who’s the most accurate shooter in the NBA? A month into the 2011−12 season, five players were locked in a tie for the highest shooting percentage in the league: Armon Johnson, DeAndre Liggins, Ryan Reid, Hasheem Thabeet, and Ronny Turiaf.

Who? That’s the point. These were not the five best shooters in the NBA. These were people who barely ever played. Armon Johnson, for instance, appeared in one game for the Portland Trail Blazers. He took one shot. He made it. The five guys on the list took thirteen shots between them and hit them all. Small samples are more variable, so an unrestricted shooting-percentage leaderboard can be topped by somebody who’s taken only a handful of shots and got lucky every time. (Any doubt on this point can be put to rest by looking at Johnson’s 2010−11 season, when he shot a steadfastly ordinary 45.5% from the field.) That’s why the standard leaderboard doesn’t show guys like Armon Johnson. Instead, the NBA restricts the rankings to players who’ve reached a certain threshold of playing time; otherwise, part-time nobodies with their small sample sizes would dominate the list.

But not every ranking system has the quantitative savvy to make allowances for the Law of Large Numbers. The state of North Carolina, like many others in this age of educational accountability, instituted incentive programs for schools that do well on standardized tests. Each school is rated on the average improvement of student test scores from one spring to the next; the top twenty-five schools in the state on this measure get a banner to hang in the gym and bragging rights over the surrounding towns.

Who wins this kind of contest? The top scorer in 1999, with a 91.5 “performance composite score,” was C. C. Wright Elementary in North Wilkesboro. That school was on the small side, with 418 students in a state where elementary schools average almost 500 kids. Not far behind Wright were Kingswood Elementary, with a score of 90.9, and Riverside Elementary, with 90.4. Kingswood had just 315 students, and tiny Riverside, in the Appalachian town of Newland, had only 161.

In fact, the small schools cleaned up on North Carolina’s measure in general. A study by Thomas Kane and Douglas Staiger found that 28% of the smallest schools in the state made the top twenty-five at some point in the seven-year window they studied; among all schools, only 7% ever got the banner in the gym.

It sounds like small schools, where teachers really know the students and their families and have time to deliver individualized instruction, are better at raising test scores. But maybe I should mention that the title of Kane and Staiger’s paper is “The Promise and Pitfalls of Using Imprecise School Accountability Measures.” And that smaller schools did not show any tendency, on average, to have significantly higher scores on the tests. And that the schools that were assigned state “assistance teams” (read: that got a dressing-down from state officials for low test scores) were also predominantly smaller schools.

In other words, as far as we know, Riverside Elementary is no more one of the top elementary schools in North Carolina than Armon Johnson is the sharpest shooter in the league. The reason small schools dominate the top twenty-five isn’t because small schools are better, but because small schools have more variable test scores. A few child prodigies or a few third-grade slackers can swing a small school’s average wildly; in a large school, the effect of a few extreme scores will simply dissolve into the big average, hardly budging the overall number.

So how are we supposed to know which school is best, or which state is most cancer-prone, if taking simple averages doesn’t work? If you’re an executive managing a lot of teams, how can you accurately assess performance when the smaller teams are more likely to predominate at both the top and bottom tiers of your rankings?

There is, unfortunately, no easy answer. If a tiny state like South Dakota experiences a rash of brain cancer, you might presume that the spike is in large measure due to luck, and you might estimate that the rate of brain cancer in the future is likely to be closer to the overall national number. You could accomplish this by taking some kind of weighted average of the South Dakota rate with the national rate. But how to weight the two numbers? That’s a bit of an art, involving a fair amount of technical labor I’ll spare you here.

One relevant fact was first observed by Abraham de Moivre, an early contributor to the modern theory of probability. De Moivre’s 1756 book The Doctrine of Chances was one of the key texts on the subject. (Even then, the popularization of mathematical advances was a vigorous industry; Edmond Hoyle, whose authority in matters of card games was so great that people still use the phrase “according to Hoyle,” wrote a book to help gamblers master the new theory, called An Essay Towards Making the Doctrine of Chances Easy to those who Understand Vulgar Arithmetic only, to which is added some useful tables on annuities.)

De Moivre wasn’t satisfied with the Law of Large Numbers, which said that in the long run the proportion of heads in a sequence of flips gets closer and closer to 50%. He wanted to know how much closer. To understand what he found, let’s go back and look at those coin flip counts again. But now, instead of listing the total number of heads, we’re going to record the difference between the number of heads actually flipped and the number of heads you might expect, 50% of the flips. In other words, we’re measuring how far off we are from perfect head-tail parity.

For the ten-coin trials, you get: 1, 1, 0, 1, 0, 1, 2, 2, 1, 0, 0, 4, 2, 0, 2, 1, 0, 2, 2, 4 . . . For the hundred-coin trials: 4, 4, 2, 5, 2, 1, 3, 8, 10, 7, 4, 4, 1, 2, 1, 0, 10, 7, 5 . . . And for the thousand-coin trials: 14, 1, 11, 28, 37, 26, 8, 10, 22, 8, 7, 11, 11, 10, 30, 10, 3, 38, 0, 6 . . .

You can see that the discrepancies from 50-50 get bigger in absolute terms as the number of coin flips grows, even though (as the Law of Large Numbers demands) they’re getting smaller as a proportion of the number of flips. De Moivre’s insight is that the size of the typical discrepancy is governed by the square root of the number of coins you toss. Toss a hundred times as many coins as before and the typical discrepancy grows by a factor of 10—at least, in absolute terms. As a proportion of the total number of tosses, the discrepancy shrinks as the number of coins grows, because the square root of the number of coins grows much more slowly than does the number of coins itself. The thousand-coin flippers sometimes miss an even distribution by as many as 38 heads; but as a proportion of total throws, that’s only 3.8% away from 50-50.

De Moivre’s observation is the same one that underlies the computation of the standard error in a political poll. If you want to make the error bar half as big, you need to survey four times as many people. And if you want to know how impressed to be by a good run of heads, you can ask how many square roots away from 50% it is. The square root of 100 is 10. So in the illustrative trial with 60 heads in 100 tries, that was exactly one square root away from 50-50. The square root of 1,000 is about 31; so the illustrative result of 538 heads in 1,000 tries was even more surprising, even though I got only 53.8% heads in the latter case and 60% heads in the former.

But de Moivre wasn’t done. He found that, with many fair, independent flips, the distribution of appropriately scaled discrepancies from 50-50 is approximated by a bell curve, or, as we call it in the biz, the normal distribution. (Statistics pioneer Francis Ysidro Edgeworth proposed that the curve be called the gendarme’s hat, and I have to say I’m sorry this didn’t catch on.)

The bell curve/gendarme’s hat is tall in the middle and very flat near the edges, which is to say that the farther a discrepancy is from zero, the less likely it is to be encountered. And this can be precisely quantified. For a large number N of fair, independent flips, the chance of being within the square root of N heads of an even split—50% heads—is about 95.45% by the normal approximation. The square root of 1,000 is about 31; indeed, eighteen of our twenty big thousand-coin trials above, or 90%, were within 31 heads of 500. If I kept playing the game, the fraction of times I ended up somewhere between 469 and 531 heads would be expected to be close to that approximate 95.45% figure.

It feels like something is making it happen. Indeed, de Moivre himself might have felt this way. By many accounts, he viewed the regularities in the behavior of repeated coin flips (or any other experiment subject to chance) as the work of God’s hand itself, which turned the short-term irregularities of coins, dice, and human life into predictable long-term behavior, governed by immutable laws and decipherable formulae.

It’s dangerous to feel this way. Because if you think somebody’s transcendental hand—God, Lady Luck, Lakshmi, doesn’t matter—is pushing the coins to come up half heads, you start to believe in the so-called law of averages: five heads in a row and the next one’s almost sure to land tails. Have three sons, and a daughter is surely up next. After all, didn’t de Moivre tell us that extreme outcomes, like four straight sons, are highly unlikely? He did, and they are. But if you’ve already had three sons, a fourth son is not so unlikely at all. In fact, you’re just as likely to have a son as a first-time parent.

This seems at first to be in conflict with the Law of Large Numbers, which ought to be pushing your brood to be split half and half between boys and girls. But the conflict is an illusion. It’s easier to see what’s going on with the coins. I might start flipping and get 10 heads in a row. What happens next? Well, one thing that might happen is you’d start to suspect something was funny about the coin. For this example, let’s assume the coin is fair and the flips are independent. So the law demands that the proportion of heads must approach 50% as I flip the coin more and more times.

Common sense suggests that, at this point, tails must be slightly more likely, in order to correct the existing imbalance. But common sense says much more insistently that the coin can’t remember what happened the first ten times I flipped it! I won’t keep you in suspense—the second common sense is right. The law of averages is not very well named, because laws should be true, and this one is false. Coins have no memory. So the next coin you flip has a 50-50 chance of coming up heads, the same as any other. The way the overall proportion settles down to 50% isn’t that fate favors tails to compensate for the heads that have already landed; it’s that those first ten flips become less and less important the more flips we make. If I flip the coin a thousand more times, and get about half heads, then the proportion of heads in the first 1,010 flips is also going to be close to 50%. That’s how the Law of Large Numbers works: not by balancing out what’s already happened, but by diluting what’s already happened with new data, until the past is so proportionally negligible that it can safely be forgotten.