Reliability of Case Evidence

Reliability of Case Evidence

Prior probabilities

At the dinner table John tells Mary: "One of our employees got caught today for stealing, but she said she never did it before and will never do it again."

How likely is it that she never did it before? Relevant prior information would include how often a detected theft is a first offense, as well as the evidence in this case.

Charles Munger tells us how John should think:

If you're going to catch 10 embezzlements a year, what are the chances that any one of them - applying what Tversky and Kahneman called base rate information - will be somebody who only did it this once? And the people who have done it before and are going to do it again, what are they all going to say? Well in the history of the ... Company they always say, "I never did it before, and I'm never going to do it again." And we cashier them. It would be evil not to, because terrible behavior spreads.

Does eyewitness identification or DNA evidence mean that a person is guilty? Does a positive medical test mean that a person has a disease?

In the eighteenth century, the English mathematician and minister Thomas Bayes laid a foundation for a method of evaluating evidence. The French mathematician Pierre Simon de Laplace brought the method to its modern form. Bayes' Theorem makes it possible to update the prior probability of an outcome in light of new evidence. It is easier to use if we change probability formats into frequency formats. Let's use Bayes's Theorem with a modified version of the classic cab problem, originally developed by the Psychologists Daniel Kahneman and Amos Tversky.

John testifies in court: "I witnessed the accident and the cab involved was green." John's vision has been reliably tested and the tests establish that he can identify the color green correctly 8 out of 10 times. John said "green" in 8 out of 10 cases when something was green and said "green" 2 times out of 10 when something was blue. This means that John misidentified the color 2 out of 10 times.

How trustworthy is John as an eyewitness? Eyewitness testimony contains uncertainty. The probability that John’s identification is correct depends on both his ability to distinguish the colors and how often each color is involved in an accident. First we ask: What is the prior probability of outcome - how probable is an event prior to considering the new evidence? How probable is it that a green cab was involved in an accident before we consider John's evidence? Assume that the relative frequency (the proportion of cabs of a certain color in a particular population at a specific point in time) of blue and green cabs gives us information about the prior probability of involvement in the accident. What was the proportion of blue and green cabs out of all cabs at the time of the accident? Assume there were a total of 100 cabs in town. 90 blue and 10 green. This means that the prior probability the cab involved was green is 10% (10 green cabs out of 100).

What is the (posterior - after considering case evidence) probability the cab was green given that John said it was green?

	John says "green"
Given green	8 (10 x 0.8)
Given blue	18 (90 x 0.2)
Total	26

If 10 out of 100 cabs are green and John is right 8 out of 10 times, then he identifies 8 cabs as green. If John misidentifies blue cabs at the stated rate, he is expected to identify 18 of the 90 blue cabs as green. Out of a total of 26 cabs John has identified as green, only 8 are green. This means that the likelihood that the cab was green given John's testimony that "the cab was green" is 31% (8/26). It seems the cab involved is more likely to have been blue.

Before John testified the prior probability was only 10% that the cab involved was green. When he testified "green" the probability rose to 31%.

"Based on 50 years of accidents involving cabs, given the same proportion of color, 3 out of 4 times the cab involved was green."

Independent of the frequency of blue and green cabs, the appropriate prior probability may have been past evidence of accidents. What we want is the correct evidence that is representative for what was likely to happen before we considered the new evidence.

When we get new representative evidence, we must update the prior probability. Ask: What has happened in similar cases in the past? Are there any reasons this probability should be revised? Have the circumstances or the environment changed? When the case evidence does little to distinguish the alternatives, the posterior remains closer to the prior.

How strong is the evidence?

One factor when evaluating evidence is the coincidental or random match probability. It answers the question: What is the probability that a person other than the suspect, randomly selected, will match a certain profile? For example, when evaluating DNA evidence a random match happens when two different people share the same DNA profile.

After five days of searching, the police found the missing woman strangled to death. John's brother, Bill, is on trial for her murder.

There is a DNA profile match. The forensic evidence against Bill is blood and tissue samples taken from the crime scene, which match Bill's. Either Bill left the evidence or someone else did.

What is the probability of a coincidental match? How likely is it that a match occurs between the DNA profile found at the scene of the murder and a randomly chosen person? How likely is it that Bill's profile matches the profile of the person who did leave the evidence at the crime scene? How rare is this profile? The rarer the profile, the lower the probability that Bill's matches only by chance. The prosecution's medical expert witness estimates (estimation of the frequency of the profile in the most appropriate comparison population) the probability that there would be such a match if Bill was innocent and the match was just a coincidence as only 1 in 20,000. This means that out of every 20,000 individuals, only one will have the same DNA profile as the one found at the murder scene. The prosecutor argues: "There is only a one in 20,000 chance that someone other than Bill would by chance have the same profile as the one found on the murder scene. The probability is therefore only 1 in 20,000 that someone other than Bill left the evidence." The figure had a dramatic impact on the media and the jury. Bill was found guilty and given a life sentence.

Where did justice go wrong? The prosecutor confused two probabilities. The probability that Bill is innocent given a match is not the same as the probability of a match given Bill is innocent. Even the replacement wording needs care: "The probability is one in 20,000 that some person other than Bill would leave the same blood and tissue as that found on the crime scene." A random-match rate concerns a randomly selected person’s profile, not the probability of that person leaving evidence at the scene.

The jury also needs to consider prior (before the evaluation of the forensic evidence is considered) probabilities of guilt. The probability that Bill is the murderer can't only be calculated from the forensic evidence alone. Other evidence needs to be considered. The significance of forensic evidence always depends on other evidence. What other data do the police have? What else is known about Bill? Does he have an alibi? Was he near the crime scene? Each piece of evidence must be considered together, and not in isolation. There may also have been a non-criminal explanation of why Bill left the blood and tissue samples.

For a simplified calculation, suppose the prior probability that Bill is the source of the evidence is 10% (90% that he is not). Assume the probability of a match when he is the source is 1 (sensitivity 100%, with no false negatives). Combining those assumptions with a random-match probability of one chance in 20,000 gives a posterior probability that Bill is the source of 99.96% (0.1/0.100045). This calculation assumes accurate laboratory handling and reporting; it does not establish that leaving the evidence would make him guilty.

	Match
Bill is the source	0.1 (10% x 100%)
Bill is not the source	0.000045 (90% x 1/20,000)
Total	0.100045

One way of determining the prior probability is by asking: What is the population from which the murderer could have come? We need to know the appropriate comparison population to estimate this number. The murder has taken place in a city of 500,000 men. Assume any man in the city could have committed the crime. One of them is the murderer. Of the 499,999 people innocent, we can expect about 25 coincidental DNA matches. This means there are 26 men (25 + the murderer) who could have committed the crime. If Bill was selected only through the match and is otherwise no more likely than any of these 26 men to be the source, the approximate source probability is 3.8% (1/26).

This is an alternative model with a different prior, not an additional update to the preceding calculation. It assumes the men were equally plausible sources before the match—for example, that they had comparable access to the crime scene.

The choice of appropriate comparison population also matters. How did the expert witness estimate the random match probability? What is the real prevalence of this profile? And does the random match probability mean that this profile would occur only once in 20,000 individuals? No, the calculated frequency is only an estimate that may be wrong in either direction.

The possibility of contamination, transfer, or deliberate planting is a separate question from the rarity of a matching profile. In Scientific Conversations by Claudia Dreifus, forensic mathematician Charles Brenner says about the O.J. Simpson case and DNA evidence: "The defense did something very clever from the DNA point of view: They said the evidence was planted. Their basic strategy was even if it matches, it was a plant. They gave up on the strategy of disproving the DNA evidence. There obviously was a match in the blood. They never denied it."

In the O.J. Simpson trial, the defense argued that fewer than one out of 1,000 wife abusers kill their wives. Therefore evidence of abuse is irrelevant and should not be admissible in a murder trial. But the appropriate probability is not the probability that a man who abuses his wife kills her. The relevant comparison population to consider is wives who have been abused by their husbands and thereafter been murdered by someone. The relevant question is therefore: What is the probability that a man killed his wife given that he abused her and given that she was actually killed by someone? And Nicole Brown Simpson was killed, not just abused.

John Allen Paulos says in Innumeracy that given reasonable facts about murder and abuse, it has been shown that if a man abuses his wife or girlfriend and she is later killed, the abuser is the killer more than 80 percent of the time. But this doesn't mean that the probability that the husband or boyfriend is guilty of murder is 80%. It is just one piece of evidence among many that needs to be considered.

Do errors happen in testing?

Yes, one type of error is a false positive and another is a false negative. A false positive is akin to a false alarm. A false negative is the missing of a real effect. For example, some factors that can influence the reliability of medical test results and cause false positives are: the clinical accuracy of the test method (compared with some "gold standard"), patient preparation, medical conditions, medications and laboratory errors. People may also make errors when collecting and handling samples, in their interpretation of test results or in reporting test results correctly.

John tested positive for a rare disease with a mortality rate of 80%. How scared should he be?

What is the chance that anyone (a person chosen randomly and belonging to the same risk group as John) actually has the disease given that they tested positive? The predictive value of the test depends both on the clinical accuracy of the test and prior probabilities or the proportion of individuals with the disease in the population we are testing at a given time (prevalence). Sensitivity is the proportion of people with the disease who test positive. Specificity is the proportion without the disease who test negative.

Assume a population of 100,000 people.

The frequency of people with the disease in the population is 0.1% i.e. one person in 1,000 has the disease. Before the test John had a 0.1% chance of having the disease and a 99.9% chance of not having the disease. With a 100% accurate test, 100 people would test positive and 99,900 would test negative. These counts reflect the assumed prevalence before considering a particular test result.

The test has a 97% sensitivity or true positive rate. This means that 97 out of 100 people with the disease correctly test positive. It also means that 3 people out of 100 with the disease wrongly test negative (false negatives).

The test has a 95% specificity or true negative rate. This means that 95 out of 100 people without the disease correctly test negative. The false-positive rate among people without the disease is 5%. That means 5% of that group, or 4,995 people, wrongly test positive (false positives).

Since John is told he tested positive, the information he needs is the frequency of people that test positive and have the disease (true positives) and the frequency of people that test positive but don't have the disease (false positives).

	Test positive
Given disease	97 (100 x 0.97)
Given not disease	4,995 (99,900 x 0.05)
Total	5,092

Among every 1,000 positive results in this population, we would expect about 19 to be true positives (97/5,092). Under these assumptions, the probability that John has the disease given a positive test is about 1.9%. Of the 5,092 positive results, most are false positives.

What if a randomly tested person tests negative? There are 3 false negatives and 94,905 true negatives, meaning that there is more than a 99.9% chance that the person doesn't have the disease.

The label "tested positive" can be scary but remember that the test is not the disease. A test could fall in the group of false positives. But which is worst? To belong to the group of false positives - diagnosed as having the disease without having it, or false negatives - diagnosed as not having the disease but having it? With sensitivity and specificity held constant, a positive result is more likely to be a true positive when the disease is more common in the tested group. A negative result becomes less reassuring as prevalence rises. The relevant prevalence is that of the person’s risk group, which may differ from the population average.

Ask: What is the frequency of people with the disease in the relevant comparison population before I consider specific case evidence? How accurate is the medical test?

The above reasoning can be used to help evaluate the reliability of diagnostic tests or screening procedures. Some examples are screening for or diagnosis of breast cancer, prostate cancer, colorectal cancer, HIV or drug use.

Estimating the frequency of false positives and false negatives are also important when evaluating the reliability of polygraph tests (used in criminal investigations or for screening employees) and identification systems.

In polygraph tests, false positives occur when innocent persons are found deceptive. False negatives occur when guilty persons are found non-deceptive.

In identification systems, a false positive occurs when a system accepts a match where there is none. A false negative occurs when a system fails to recognize a match where there is one.

The probability of false positives is also a factor to consider when evaluating the value of DNA profile evidence. This means that the jury in Bill's case also needs to consider the false positive rate. The jury needs to ask: What is the probability that the laboratory reports a match between two samples that don't match? A reported match doesn't necessarily mean a true match. Errors happen. A possible explanation for the forensic match may be error due to contamination (accidental or deliberate), mishandling the evidence, or switching the samples. For example, in one rape case, technicians from the Houston police crime laboratory told the jury that they found a DNA match between a rapist's DNA and a male suspect. The man was convicted in 1999 and sent to prison for 25 years. In 2003, the Houston Police Department said that the DNA was not from the convicted man.

When evaluating case evidence we must consider the prior probability, the probability of a random match, and the probability of a false positive.