Normative Evidence

Normative Evidence

Recurring instances of a particular event may exhibit noticeable amounts of cross-time consistency. For example, the aggregate frequency of automotive accidents is relatively consistent from year to year, especially when adjusted for driving volume, as illustrated by the widely used statistic, Number of Fatalities per 100 Million Vehicle Miles Traveled, the value of which ranged from a low of 1.08 to a high of 1.15 during a five-year period from 2010 to 2015. When a recurring event shows such longitudinal stability, it often becomes enshrined in the form of an industry benchmark, or a standard point of reference against which more specific instances may be compared.7 Given their aggregate foundations, benchmarks tend to be very general and slow changing which tends to limit their informational value when applied to a specific context. Nonetheless, being rooted in often considerable volumes of accumulated experience, those generalizations can offer a sound basis for face-validity assessments.

Some benchmarks circulate as informal estimates with little information about their origin. Before relying on one, check its source, date, population and method. A widely repeated number may still be poorly suited to the decision at hand.

Another widely used source of normative evidence is best practices, broadly characterized as solutions to recurring problems utilizing validated applied principles. Unlike the quantitatively minded benchmarks, best practices are qualitative and process oriented, aiming to single out and describe the best means of accomplishing a particular objective. Whether it addresses a fairly broad system, such as customer relationship management or insurance claim management, or a comparatively narrower process of identifying at risk of attrition customers or claims that exhibit heightened likelihood of adverse development, best practices can offer useful starting points for organizational decisions. Still, there are dangers in over-reliance on such unchecked information, as illustrated by a fairly well-known meatloaf story, which goes like this: A man, let’s call him John, with a great liking for meatloaf observed that every time his wife Mary made that dish she always cut off the ends prior to cooking it. When asked about it, Mary responded that she learned it from her mother. Not satisfied with that explanation, John then asked his mother-in-law about it, who told him that the cooking pan she was using for her meatloaf was quite small, and so she always ended up having to trim the ends. And so what her daughter internalized as being a part of the general meatloaf making ritual was a step that only made sense under specific circumstances. The point here is that it is important to critically assess the validity of any general practice to make sure it indeed reflects sound thinking, or that it stands on robust and sufficiently current and applicable informational foundation.