Exploratory Data Analysis
Exploratory data analysis looks for patterns and examines what remains unexplained. Banasiewicz recommends focusing that search on a decision, reviewing earlier findings and checking data quality to reduce the distraction of chance associations.
THE IDEA
Exploratory Data Analysis
EDA looks for patterns in data. One way to describe the process is Data = Smooth + Rough.12 The smooth is a modeled relationship; the rough is what remains unexplained. After fitting an initial pattern, an analyst examines the residuals for further structure. An unexplained residual is not necessarily meaningless noise.
Banasiewicz contrasts the small datasets common in social-science research in the 1970s with the much larger datasets available when he wrote, around 2018. Searching across many variables can produce numerous apparent relationships, including chance associations.13 He therefore recommends directing the search toward the decision at hand and comparing findings with earlier analyses to look for persistent relationships.
He proposes three phases:
- Review previous findings. Identify which earlier analyses can inform the current question.
- Examine data quality. Check whether the available data can support the intended analysis.
- Analyze the data. Look for additional patterns relevant to the decision, including structure left in the residuals.
When to use it
Identifying key drivers of customer churn by iteratively exploring a transactional dataset, extracting main effects and then examining residuals for interaction effects.
Exploring sensor data from manufacturing equipment to find patterns that predict failures, using the smooth-rough decomposition to isolate systematic signals from noise.
Analyzing survey responses to uncover relationships between demographic factors and satisfaction scores, with phase-based EDA including review of prior studies and data quality checks.
Investigating longitudinal financial data to find persistent market indicators, using EDA to smooth out short-term fluctuations and focus on stable relationships.
FROM THEORY TO EVERYDAY LIFE
In practice
Analyzing Customer Purchase Patterns
A data scientist explores a dataset of customer transactions to find relationships between purchase frequency and demographics. After extracting the main effect of age on frequency, they examine residuals and find an association with geographic region that was initially masked. This iterative process follows the EDA formula of smooth + rough.
Social Science Survey Data
A researcher in the 1970s studies a small survey dataset on voting behavior. They first plot correlations between education level and party preference, then examine the residuals for interactions with income, refining their understanding of voter segments step by step.
A MOMENT TO REFLECT
Put it to work.
How can you constrain your exploratory data analysis to focus on patterns that are relevant to a specific decision, rather than chasing every statistically significant relationship?
What methods could you use to incorporate prior analytical results into your current EDA process to identify longitudinally stable relationships across datasets?
When examining residual patterns in your data, how do you decide when to stop extracting additional 'smooth' and accept the remaining 'rough' as noise?
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)