Confirmatory Data Analysis

Confirmatory Data Analysis

Evidence is information offered in support of a belief or inference, as discussed in Chapter 5. It may point in different directions. An organization's marketing records, for example, might show that temporary price reductions help in some circumstances and not in others. Exploring those records could suggest that the depth of a discount affects its value. That observation can become a hypothesis for a confirmatory study.

The distinction rests on when the hypothesis and methods are chosen. Exploratory analysis looks for patterns; confirmatory analysis tests a specified claim under a plan set before examining the test results. Testing a pattern on the same data that suggested it risks mistaking a chance finding for a reliable relationship. Independent data provide a stronger test.

Clinical trials illustrate a structured progression, though the phases can overlap in purpose. Phase 1 typically examines safety and dosage. Phase 2 further examines safety and looks for evidence of effectiveness. Phase 3 evaluates efficacy and safety in larger studies with prespecified methods. Business projects may use a less formal process, but they still need to distinguish generating a hypothesis from testing it.

A confirmatory study might test whether sales incentives increase purchases or whether price is associated with aggregate sales. Its results can remain uncertain because of weak data, questionable assumptions, or low statistical power.20 Planning helps make the test interpretable; it cannot ensure a decisive result.

Use reliable data and choose a sample size suited to the smallest effect that matters for the decision. The source recommends statistical power of 0.90 or higher; the appropriate target depends on the study and the costs of errors. Before analyzing the test data, specify:

• any transformations or adjustments to the data;
• the criteria for excluding observations;
• the statistical tests to use;
• whether each test is one-sided, such as A > B, or two-sided, such as A ≠ B; and
• the threshold for statistical significance, commonly α = 0.05, and how results will be interpreted.

The significance threshold controls the planned Type I error rate under the test's assumptions. A p-value is compared with that threshold; it is not the probability that the hypothesis is false. A nonsignificant result also does not, by itself, establish that an effect is absent.

The source calls another preparatory step informational domain specification (IDS). This means defining the proposed relationships, the concepts involved, and the variables used to measure them. An analysis of discounts, for example, needs to specify what counts as a discount, which outcome represents promotional value, and what other conditions could affect the comparison. That makes the claim and the limits of the available data clear before testing begins.