Picking the ‘Right’ Data
Analysis can lose its purpose when it starts with available data and reports rather than a decision or question. Another mistake is treating statistical significance as evidence of practical importance: a large sample can estimate a very small effect precisely.
THE IDEA
Picking the ‘Right’ Data
Organizations can collect more data than they know how to use. One response is to produce reports covering every available measure. A more useful starting point is the question the analysis needs to answer. That question helps determine which variables, time periods, and comparisons matter. Exploration can suggest questions, but a report does not become useful merely by containing more metrics.
Sample composition and timing also matter. Older observations may be less relevant when customer behavior, operating conditions, or measurement methods have changed. They may still be valuable for detecting longer-term patterns. The analyst needs to assess whether the observations describe the population and conditions relevant to the decision.
Standard error measures how much an estimator would vary across repeated samples. For a sample mean based on independent observations with a common variance, it is estimated by dividing the sample standard deviation by the square root of the sample size. A larger sample therefore lets the analysis detect smaller differences.
This does not mean a large sample inherently inflates the false-positive rate of a correctly specified test. It means a small but real effect can be statistically significant while having little practical value. Assess the effect’s size and uncertainty alongside the costs, benefits, and decision threshold. Testing many possibilities or violating a test’s assumptions creates separate problems that sample size alone does not resolve.
When to use it
A product manager evaluating user behavior data should first define the key question (e.g., 'What features drive retention?') before selecting which metrics to analyze.
A scientist planning an experiment must consider whether an extremely large sample might yield trivial effects that are statistically significant but not meaningful.
A business analyst reviewing sales trends checks whether older observations reflect the current market before deciding how to include them.
A policymaker interpreting a meta-analysis examines effect sizes and uncertainty alongside the practical consequences of the findings.
FROM THEORY TO EVERYDAY LIFE
In practice
Upside-Down Data Approach
A marketing team has access to a vast database of customer transactions. Instead of starting with the question 'What factors predict repeat purchases?', they generate a monthly report summarizing all available metrics. The report contains hundreds of numbers but offers little actionable insight because it was not aligned with any specific business question.
Large Sample Size Pitfall
In a hypothetical trial with 100,000 patients, a drug produces a statistically significant average blood-pressure reduction of 0.5 mmHg. The large sample makes a small effect easier to detect. Whether the effect warrants treatment is a separate question involving its clinical importance, costs, and side effects.
A MOMENT TO REFLECT
Put it to work.
Think of a recent data analysis you performed. Did you start with a clear question, or did you simply explore available data? How might the outcome have differed if you had reversed your approach?
When reviewing a study with a very large sample size, what questions should you ask to determine whether the statistically significant result is also practically significant?
How can you guard against the temptation to include all available data in an analysis, especially when some data is old or irrelevant to the current decision?
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)