RSS fellow and lawyer Anthony Cutler went to the launch of our new guide to statistics for those working in law. Here he reports on what both legal and statistical experts had to say about it.
'Probability is not foreign to the law. Evidence … is primarily a legal notion …' - Ian Hacking, The Emergence of Probability.
I was reminded of this quote on the evening of 27 November which saw the launch of Statistics and Probability for Advocates, a joint initiative of the Royal Statistical Society and the Inns of Court College of Advocacy (ICCA). The guide is an educational resource and ready reference for judges, barristers, solicitors and others who have to present and challenge statistical evidence at trial.
Derek Wood CBE QC (chair of governors at ICCA) opened and closed the evening. Many lawyers, he observed, had entered the profession never imagining that it was going to draw upon their numeracy and critical thinking about data.
Ruth Morgan (director, UCL Centre for the Forensic Sciences) set the tone by inviting the audience to step back from the headline statistics. Experts’ concise numerical conclusions were the outcome of the whole history of how evidence had developed from the purported locus. The physical tokens had interacted with, and been transformed by, a variety of environments, agents and processes. All those together formed an evidence base that demanded to be scrutinised. Moreover, the evidence base was neither static nor certain. The evolution of the evidence might well be vague. A proper evaluation required awareness of evidence dynamics. There were layers of uncertainty that needed to be sifted. I was left with the suspicion that, as Richard Jeffrey put it, 'It’s probabilities all the way down.' Professor Morgan had some statistics (sic) on recently quashed criminal verdicts in the Court of Appeal. In 39% of successful appeals, scientific test methodology had been criticised and in 32% the forensic science interpretation. The correct and accurate use of statistics and statistical evidence involved more than narrow mathematical interpretation and there remained differences in how science and law approach the problems.
RSS President Professor David Spiegelhalter stepped in to deliver a talk that had been promised by Professor David Ormerod QC (Law Commissioner for Criminal Law and Evidence) sadly absent due to other commitments. There were three principal issues that Professors Spiegelhalter and Ormerod wanted to urge upon the legal profession to improve confidence in evidence. The first was fundamental, awareness of the prosecutor’s fallacy. I think if I asked you to give me the probability that somebody with measles has spots you would reply close to 100%. However, if I ask you the probability that somebody with spots has measles I suspect you would be more circumspect and offer a much lower probability. Many things other than measles cause spots. The probability of a theory, the measles, given the evidence, the spots, is not at all the same thing as the probability of the evidence given a theory. That maps over into the court room where the probability of innocence given the evidence is not the same thing as the probability of the evidence given innocence. Professor Spiegelhalter warned of the subtle dangers in the words used to express the relationship between evidence and case theory. Confidence in evidence demanded accurate and critical use of language, grasp of mathematical relationships and the ability to act within the guidance of the appeal courts.
The second of Professor Spiegelhalter's points again picked up on one of the central messages of the evening. It is often the courts’ readiness to grasp individual numbers that risks leading them into error. Likelihood ratios are increasingly common in litigation. These tests compare the probability of the observed evidence under two competing accounts of how it came to be, perhaps the guilt of the defendant on one hand or an accidental transfer of bodily fluids on the other. Such crisp numerical summaries were often salient features on summing up. Professor Spiegelhalter picked up on the earlier point that the numbers had no meaning without looking at, as Professor Morgan had put it, the evidence dynamics and the various reliabilities of the likelihoods in question.
His third point was that the language of uncertainty mattered. Mathematics could capture probabilities such as one in a million or one in a billion but they had limited meaning for a lay audience. Expressing risk in publicly meaningful and communicable terms is David’s speciality and he reminded the audience that there is empirical research as to how this can be done well.
The final talk of the evening was from Dr Claire McIvor of Birmingham Law School. Dr McIvor is a practising lawyer but she wanted to champion epidemiologists in the eyes of the law. Epidemiological evidence is often critical to issues of factual causation in complicated disease litigation. However, judges tend to see the profession as a species of statistician and to devalue their evidence. Clinicians and engineers always carry a credibility with judges that epidemiologists and physicists don’t. The common law tends to favour the concrete over the abstract. Dr McIvor felt that the courts did not handle epidemiological evidence very well. The same messages come across. The courts like witnesses, expert of lay, who give crisp confident responses. Expressions of doubt or admissions of inherent uncertainty are accepted but avocations of certainly are not routinely challenged.
Dr McIvor referred to the case of Novartis v Cookson ( EWCA Civ 1261) which dealt with causation in the onset of bladder cancer. The exactly stated probabilities of the claimant’s consultant urologist were preferred over the, perhaps more descriptive, evidence of the defendant’s epidemiologist. Lawyers and judges were too readily impressed by the numbers. There was not enough probing of where they came from or what they meant. Experts who declared 'Studies have shown that 75% of …' were insufficiently challenged on the nature of the studies, what the sampling frame, how participants were recruited, what the dosage was, etc.
Dr McIvor made some further trenchant points about the civil standard of proof, the balance of probabilities, and the courts’ use of relative risk. I suspect that these are issues that appeal courts may well be invited to revisit before much longer. Dr McIvor ended with comments on the unfortunate clinical negligence case of Gregg v Scott. I won’t here add to the volumes that have already been written about that case but if you are not familiar with it, statistician or lawyer, it is well worth looking up.
The RSS/ ICCA guide is a powerful reminder to advocates and judges that professions of great confidence should invite systematic challenge as to their bases. There is a great deal in the guide that will help anyone dealing with expert evidence before the courts to handle witnesses knowledgably and surely. If the guide is about statistics then it is about statistics in the broadest sense. It is about how we challenge professions of expert knowledge and how we weigh and express confidence. That is never a bad thing.
At the end of the evening, Derek Wood reflected on how unprepared so many lawyers were to challenge science and unpack mathematics. Yet, it is now 120 years since US Supreme Court justice Oliver Wendell Holmes predicted: 'The black-letter man may be the man of the present but the man of the future is the man of statistics and the master of economics.'