Not All Sample Sizes Are Created Equal

Not All Sample Sizes Are Created Equal

One of the most important considerations with sample sizes is their representative diversity. The insights you get from a large number of data points are only going to be as good as the range of possibilities they cover. If, for example, you want to know how effective airbags are in preventing serious injury across a car’s occupants, data only on drivers will leave questions about people in other seats unanswered.

The authors of “The Weirdest People in the World?” explain, “Behavioral scientists routinely publish broad claims about human psychology and behavior in the world’s top journals based on samples drawn entirely from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies.”[16] They go on to demonstrate that the people in WEIRD societies are outliers in many ways; therefore, we probably shouldn’t be using studies based entirely on the behavior of WEIRD subjects as representative of the entire human population.

If we want to uncover universals about the behavior of human three-year-olds, studying more children in California alone will not establish whether the findings apply elsewhere. If we want to say something meaningful about human nature, then our data set ought to contain information from a sample of humans who represent the diversity found on the planet.

In the book Invisible Women, author Caroline Criado Perez explores how data sets often used to make decisions that impact women don’t actually contain any information about women. She argues, “When your Big Data is corrupted by big silences, the truths you get are half-truths, at best.”[17]

It is important to study samples representative of the overall population. However, it can also be important to study subsets with different features that might not be apparent from averages.

Ask whether your collection methods capture the experiences relevant to the decision. Perez reports that more than 90 percent of people who experience unwanted sexual behavior on public transport in New York and London don’t report it. Nor do female metro users in Azerbaijan. So if someone were to conclude, based on official police reports, that safety for women isn’t an issue in these places, they would be dead wrong.[18] Therefore, those making rules and changes based on data need to look hard at the data being used.

Not including women creates a data set that eliminates half the human race. Perez gives examples of when this is annoying, like phones too big for the average female hand, and when it creates serious negative outcomes for women, such as in medical treatment. She explains, “Nearly all pain studies have been done exclusively on male mice,” and many clinical trials done exclusively on men will have their results applied equally to women, despite women having different physiognomies.[19]

Results from a narrow sample should not be applied as though every patient were the same two-hundred-pound adult. Drug response can vary with body size, age, sex, health, genetics, and other factors; broad ethnic categories do not define a single metabolic profile. A trial’s recruitment and analysis should address the patients for whom the treatment is intended, including differences that an overall average may conceal.

Adding observations does not correct a sampling method that systematically misses relevant people or experiences. Perez writes, “If you aren’t aware of how those biases operate, if you aren’t collecting data and taking a little time to produce evidence-based processes, you will continue to blindly perpetuate old injustices.”[20]

Perez argues, “The introduction of Big Data into a world full of gender data gaps can magnify and accelerate already-existing discriminations.”[21] It is not a big leap to conclude that Big Data based on narrow data sets can exacerbate many different forms of discrimination. As the authors of “The Weirdest People in the World?” conclude, “We need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity.”[22] Detailed data on a homogeneous population does not, by itself, establish that the results apply to other populations. If a data set is being used to describe the broad category of “human,” then it needs to be representative of the diversity of the species.

Perez explains, “having an accurate measure is important because data determines how resources are allocated.”[23] Sampling choices matter because the resulting decisions affect people. Assess both how much data you have and whose circumstances it captures.