The Eight Steps of Hypothesis Testing
Jones’s eight-step method compares competing hypotheses in a matrix of evidence. It emphasizes diagnostic inconsistencies, revision of the alternatives, and a final check of assumptions. A bakery case illustrates how the comparison can lead investigators to a new explanation.
THE IDEA
The Eight Steps of Hypothesis Testing
Jones proposes the following process for comparing competing explanations.
Step 1: Generate hypotheses. Write down as many hypotheses as possible to explain the problem, then eliminate implausible ones and combine similar ones.
Step 2: Construct a matrix. Label the first column "Evidence." Label the other columns to the right "Hypotheses," and enter descriptors of the hypotheses atop the columns. The hypotheses must be mutually exclusive. They need not be collectively exhaustive.
Step 3: List "significant" evidence down the left-hand margin, including "absent" evidence. In keeping with the principle of major factors, list only significant evidence. Include the absence of evidence one would expect if the hypothesis were true or untrue. Ask "Alexander’s question": What evidence not included in the matrix would refute one or more hypotheses?
Step 4: Working across the matrix, test the evidence for consistency with each hypothesis, one item of evidence at a time. Determine if the evidence is consistent (C), inconsistent (I), or ambiguous (?). "Consistency" means the evidence is compatible with the hypothesis; it does not validate it.
Step 5: Refine the matrix. Add or reword hypotheses, add "significant" evidence relevant to new or reworded hypotheses, and test that new evidence against all hypotheses. Delete evidence that is consistent with all hypotheses (it has no diagnostic value), but keep a record of it.
Step 6: Working downward, evaluate each hypothesis. Re-evaluate and confirm the validity of inconsistent evidence, and check underlying assumptions. Delete any hypothesis for which there is significant inconsistent evidence.
Step 7: Rank the remaining hypotheses by the weakness of inconsistent evidence. Under this method, the hypothesis facing the weakest inconsistent evidence becomes the leading candidate. This is not arithmetical; degree of inconsistency matters.
Step 8: Perform a sanity check. Review findings, hypotheses, evidence, and assumptions.
Example Application: The Bakery Problem
- Hypotheses: sabotage by employees, malfunctioning oven gas gauges, fluctuations in gas pressure.
- Evidence listed: three batches overbaked, only specially prepared bread overbaked, only two ovens overbaked, maintenance man found nothing wrong with heat gauges, overbaking during busiest season, employees unhappy, employee anger over firing, baker seen talking with fired employee, flour deliveries delayed.
- After testing and refinement, a fourth hypothesis "new flour is defective" was added. Only that hypothesis had no inconsistent evidence.
Alexander’s Question: What fresh information from anywhere would be inconsistent with a certain hypothesis? (from Neustadt & May, Thinking in Time)
When to use it
Choosing a new business strategy by evaluating competing market entry hypotheses
Investigating a technical failure in engineering, such as multiple possible root causes
Evaluating competing scientific theories against experimental data
Deciding on a course of treatment when several plausible diagnoses exist
Diagnosing a recurring manufacturing defect by testing competing root causes (e.g., machine malfunction, operator error, raw material quality).
Comparing explanations for a financial forecast error, such as changed demand, incorrect assumptions, or a data-processing fault.
Troubleshooting a customer service issue by generating hypotheses about process breakdowns and testing them against evidence.
Assessing potential reasons for a drop in website traffic: technical glitches, content issues, algorithm changes, or competitor actions.
FROM THEORY TO EVERYDAY LIFE
In practice
Medical Diagnosis
A doctor compares viral infection, bacterial infection, and an autoimmune reaction as possible explanations of symptoms. The matrix records findings that support, contradict, or fail to distinguish them. The working diagnosis should account for the reliability and diagnostic value of those findings, rather than simply counting consistent items.
Legal Case Investigation
A detective proposes two hypotheses: the suspect committed the crime or an unknown third party did. They list evidence (alibis, fingerprints, witness statements) and evaluate which hypothesis is better supported. After ranking, they perform a sanity check to ensure no critical evidence was overlooked.
The Bakery Problem: Diagnosing Overbaked Bread
Henry Willis faces overbaked bread during a peak week. Connie Stewart compares sabotage, faulty gas gauges, and fluctuating gas pressure against the evidence. Refining the matrix adds defective new flour as an alternative; it is the only hypothesis in the account with no inconsistent evidence.
Swine Flu Immunization Decision (Alexander’s Question)
President Ford's committee considered immunizing US population against swine flu. Dr. Alexander asked: what fresh data would cause reversal of support? This question helps identify evidence that could refute a hypothesis, applied by Stewart in step 3.
A MOMENT TO REFLECT
Put it to work.
How can you ensure that your hypotheses are truly mutually exclusive?
What might happen if you skip the sanity check in step 8?
In what situations could the eight-step process become overly cumbersome, and how would you adapt it?
Think of a recent decision you made: how could you have framed it as a hypothesis test? What evidence would have been inconsistent with your chosen hypothesis?
What combination of causes might be missed if you treat every proposed explanation as mutually exclusive?
How would Alexander’s question change the way you gather information for a business or scientific problem?
References
- Thinking in Time: The Uses of History for Decision Makers — Richard E. Neustadt and Ernest R. May (1986)
CONTINUE THE CONVERSATION
Source texts
- The Thinkers Toolkit by Morgan D Jones
- The Thinkers Toolkit by Morgan D Jones