The Calibration of Legal Judgments

Domain 3: The Calibration of Legal Judgments

The appropriateness of lawyers’ probability judgments has important implications for the quality of their service – decisions about whether to sue, settle out of court, or plead guilty to a lesser charge all depend on a lawyer’s judgment of the probability of success. Surprisingly, then, there is relatively little research assessing the calibration of lawyers’ probability judgments in their day-to-day practice. Figure 39.7 presents calibration curves representing Dutch (Malsch, 1989) and American (Goodman-Delahunty, Granhag, & Loftus, 1998) lawyers’ judgments of whether they would win their cases. Base rates in both studies were near 50%, but predictions of the Dutch lawyers were somewhat discriminating (α = .54), whereas the American lawyers’ judgments showed no predictive validity (α = .15, hardly above chance). Malsch (1989) describes a number of differences in the Dutch and American legal systems that might explain the differential sensitivity to outcomes. In both data sets, there is surprisingly little evidence that lawyers’ judgments show a confirmatory bias given the pressures of the adversarial system; if confirmatory biases were rife, then the calibration curves should fall largely under the identity line. Instead, consistent with the case-based model, both sets of judgments show an overextremity bias (underprediction of success for low probabilities and overprediction of success for high probabilities). As expected given their poorer discrimination, this bias is much more marked for the American than for the Dutch lawyers.

Figure 39.7. Lawyers: Calibration curves from Dutch lawyers (Malsch, 1989; represented by circles) and American lawyers (Goodman-Delahunty et al., 1998; represented by triangles), compared with predictions of neglect model.

Note: Neglect model predictions based on RST simulations assuming no focal bias (β = 0) and fixed judgmental extremity (σ = 1). Discriminability (D) and base rate (BR) are matched to values from original studies, with α = 0.54 and BR = 47.5% for Malsch (1989), and α = 0.15 and BR = 56% for Goodman-Delahunty et al. (1998). Neglect model curves are truncated for purposes of presentation.

Goodman-Delahunty et al. (1998) found other interesting patterns within their data. When lawyers were surveyed close to their trial date (and had somewhat valid information about a given case, α = .33) they were moderately well calibrated, and showed a moderate extremity bias; when surveyed far from the trial date (with completely nondiagnostic cues, α = .03) they showed a strong extremity bias and outcomes were at the base rate level regardless of expressed probability. Again, this neglect of discriminability is consistent with the direct support account. There was also some evidence for optimistic overconfidence when the predictions were personally relevant: the subset of lawyers who were working for plaintiffs on a contingency fee basis were just as confident as other lawyers (generally about 65% confident) but won their cases much less often (42% compared to 56% overall).