Research Synthesis Challenge
Synthesizing findings from a large body of empirical research faces challenges similar to big data: high volume, high variety, and the need for subjective human evaluation, leading to issues of inter-rater reliability.
THE IDEA
Thematic Analyses of Empirical Research
Research synthesis shares two challenges with large operational datasets: volume and variety. The source estimates close to 29,000 academic journals, alongside trade publications and reports from other organizations. It cites a University of Ottawa estimate of approximately 2.5 million scientific papers per year. If 1% concerned a particular field, that would mean about 25,000 potentially relevant contributions annually. These are the source’s estimates, not a current count.
The studies also differ in their questions, methods, settings and outcomes. Finding and pooling papers is only the beginning. Reviewers must judge their quality and determine which findings can reasonably be combined. Search and text-processing tools can assist, but their usefulness depends on the material and the review task.
Human review introduces another problem: two reviewers may interpret the same study differently. Multiple reviewers, explicit criteria and a process for resolving disagreements can help, but they also require resources. For a field producing 25,000 potentially relevant papers a year, deciding what to review and how to maintain agreement is a substantial part of the synthesis task.
When to use it
Conducting a systematic review for a medical guideline.
Evaluating the cumulative evidence on a business strategy.
Synthesizing research for a policy decision in education.
Combining findings from multiple studies in a meta-analysis.
FROM THEORY TO EVERYDAY LIFE
In practice
Education Research Synthesis
Imagine a teaching-method review whose initial search returns over 20,000 potentially relevant papers. That illustrative workload would require screening criteria and a calibrated review team; the count is not a reported finding about an actual teaching method.
Business Evidence Review
A business leader wants to gather evidence on remote work productivity. They find hundreds of studies with varying methodologies and need to synthesize findings, facing challenges of volume and variability in study design and outcomes.
A MOMENT TO REFLECT
Put it to work.
How would you design a process to synthesize research findings from thousands of studies while maintaining reliability?
What are the trade-offs between using automated text mining and human reviewers for research synthesis?
How can inter-rater reliability be improved when multiple reviewers evaluate the same studies?
Often confused with
CONTINUE THE CONVERSATION
Source texts
- Evidence-Based Decision-Making How to Leverage Available Data and Avoid Cognitive Biases (Andrew D. Banasiewicz)