Statistics and the law: a recent history of an uneasy relationship

Written by Oz Flanagan on . Posted in Features

When statistics are abused in politics or the media, the rebuke that follows can inflict a dose of embarrassment on the abuser. But a far more disturbing and personally destructive backdrop for the misuse of statistics is in the courtroom. In this setting, the power of statistics can provide the vital link in evidence needed for justice, or the distortion that sets a prosecution down the path to injustice.

As far back as the trial of Alfred Dreyfus in 19th century France, the story of statistics and the law has been a long, fraught but ultimately necessary association. This is why the Statistics and the Law Working Group of the RSS has now become a permanent Section of the Society. The working group was brought together in 2005 after a spate of legal cases with flawed statistical evidence, but one case in particular prompted the statistical community to sit up and take note.


 

Sally Clark

The most troubling case of the time was Sally Clark’s wrongful conviction for the murder of her infant sons, who were in fact victims of cot death. That case prompted the then president of the RSS, Peter Green, to write to the Lord Chancellor about the crucial probability statistic given as evidence against Clark by the expert prosecution witness, Sir Roy Meadow.

As with any use of statistics, they can either be used for illumination or manipulation. In the Clark case, Meadow stated that the chance of two infants dying from cot death was 1 in 73 million. This statement became known as ‘Meadow’s law’ and resonated with the jury and the wider public.

The figure was born out of taking the probability that a child like Clark’s would die of cot death, and then multiplying it by itself (because two of Clark’s children had died). Peter Green, in his letter to the Lord Chancellor, put it this way. ‘The calculation leading to 1 in 73 million is invalid. It would only be valid if SIDS cases [cot death] arose independently within families, an assumption that would need to be justified empirically. Not only was no such empirical justification provided in the case, but there are very strong reasons for supposing that the assumption is false.’

Nevertheless, Meadow’s argument went on to have a significant role in the similar criminal cases of Angela Cannings, Donna Anthony and Trupti Patel. At Clark’s second appeal in 2003, she was acquitted due to new medical evidence being presented to the court. A fresh examination of the statistics was due to follow, but in the end, the medical evidence proved enough to secure her release. The judicial comments following her acquittal did consider the statistics, and they stated that Meadow’s evidence was ‘manifestly wrong’ and ‘grossly misleading’.

Sally Clark had experienced the death of two of her children in infancy and spent more than three years in prison before her acquittal. She died tragically in 2007.

Lucia de Berk

Around the same time as Clark’s acquittal, another infamous miscarriage of justice was beginning its legal course in the Netherlands. Lucia de Berk worked as a hospital nurse in The Hague, but when the suspicious death of a baby happened during one of her shifts, she was singled out for scrutiny. A review of past incidents then found a pattern of deaths or near-deaths which all took place when de Berk was on duty.

She was charged and convicted of multiple murders on the basis of this perceived pattern and how out of the ordinary it was. Henk Elffers, a law professor with a graduate degree in statistics, was called as an expert witness. His key piece of evidence to the court was a calculation that there was a 1 in 342 million chance of this pattern being natural. He calculated this number by analysing cross tabulated data from Lucia’s shifts over a year, on certain wards and whether there was an incident on her watch. What Elffers had done to arrive at this figure was to multiply three p-values.

Richard Gill, a statistician at Leiden University became involved in the case after de Berk’s first appeal failed. In his opinion, had the defence understood Elffers mistake at the first trial, it would have fundamentally changed the statistical evidence against Lucia. ‘I think a good statistician would have been able to show that the 1 in 342 million number was seriously wrong because of the mistake of multiplying three p-values. It’s difficult to explain to judges why you can’t do this, but any serious textbook would tell you the correct recipe for analysing three two by two tables.’

The first appeal of de Berk’s case came in 2004, but the verdict was upheld and Elffers statistical evidence stood. Moreover, since the initial murders had been ‘proven’, she was further convicted of other deaths linked to her where no natural cause could be found.

‘After Lucia’s failure at her appeal, statisticians began to get involved’ Richard explains. ‘Essentially what we did was explain to the public that the conviction was unsafe. But what those statisticians did had no formal legal impact at all on her eventual release.’ By that time, medical evidence had also made the conviction unsound and de Berk was declared innocent in 2010 at her second appeal.

During her ordeal, de Berk had been labelled as one of the worst serial killers in Dutch history and spent seven years of her life in prison. Her story was the subject of a film released last year.

Ben Geen

The Lucia de Berk case has been brought to attention again due to its remarkable similarities with the case of another nurse, Ben Geen in the UK. He was convicted in 2006 of poisoning patients in a way that resulted in respiratory arrest. The prosecution said his motive for doing this was so that he could enjoy the thrill of resuscitating the patients.

Richard sees a stark parallel with the de Berk case. ‘Events are happening during his shifts and then a trigger event occurs. The system goes into overdrive and people become paranoid. So the doctors are told to go through past records looking for more victims. They already know there is a killer because someone died suspiciously and there has been an arrest. ‘

‘What they are actually doing is instinctively using Bayesian updating of their assumptions to go back and find other victims. The trouble is, the prior evidence being used is later found to be flawed. But nobody goes back to do a re-evaluation,’ he says.

There were two main strands to the evidence against Geen at his trial. One was a syringe found in his pocket that was said to contain the drugs he used on his victims. The other was professional medical opinion that the pattern identified was a very rare occurrence in an A&E department.

At Geen’s appeal hearing in 2009, Jane Hutton (professor of statistics at Warwick University), submitted a report on whether this pattern was statistically unusual. The appeal failed and in the judges notes they acknowledged Jane’s report, but discounted it because the medical professionals all agreed that the pattern was ‘rare’. In the judgement, they wrote: ‘A statistician might have explained for many hours to a jury the dangers of drawing inferences from clustering, however, what the statistician could not challenge was the basic premise that these incidents were rare.’

Richard Gill finds this reliance on medical opinion problematic. ‘There was an NHS inquiry into what happened at that hospital after the conviction. They looked at the numbers from the year before and they turned out to be virtually the same, but this has never been used in court. The inquiry excused the authorities from not noticing there was killer, because the number of events did not give any warning.’

A further parallel with the de Berk case is how both defendants were well aware of the events surrounding them before their arrest. Geen commented to colleagues on how there always seemed to be a resuscitation on his shifts, this was later used against him in court. Lucia de Berk also noticed the incidents happening on her shifts and called attention to them through one of her supervisors.

The Clark and de Berk cases included evidence of incorrect statistics in their initial trials, and both were acquitted when medical evidence began to unravel in tandem with the statistical aspects. However, Geen’s case is different because the statistics have been discounted, in favour of what the defence describe as anecdotal medical opinion.

Beyond the de Berk and Geen cases, similarities can also be found in the case of Colin Norris, another nurse convicted in 2008 of murdering patients in a Leeds hospital. A BBC Panorama programme recently outlined how he too came under suspicion after making offhand comments about patients on his shifts. A cluster of suspicious deaths was then discovered with Norris identified as the common link.

Ben Geen has served nine years of a 30 year sentence and is currently imprisoned in HM Prison Long Lartin. He continues to protest his innocence.

‘An invasion of hard science into the courtroom’

In the wider context, Richard Gill says there has been ‘an invasion of hard science into the courtroom and it is bringing with it an increasing level of quantitative thinking.’ But he says the onus of this rests with both statisticians and the legal profession. ‘The statisticians have to realise what is going on and learn to communicate better. We are doing that, especially with all the probabilities that come with DNA evidence today. But I also think that the younger generation of lawyers have an increasing awareness that they don’t know enough about statistics and probability.’

Statisticians do recognise the increasingly vital role that statistics and probability are playing in the courtroom. Especially given the advances in DNA evidence that often require likelihood ratios to be communicated to juries. This is why the Statistics and the Law Working Group have compiled practitioner guides (with funding from the Nuffield Foundation) to assist judges, lawyers and expert witnesses in communicating and interpreting statistics.

However, statistical theory is something that the legal profession generally come to research when a case with statistical evidence presents itself. Colin Aitken, chairman of the Statistics and the Law Section and co-author of the practitioner guides, believes that in an ideal world, teaching of an understanding of statistical and probabilistic reasoning in a legal context would be embedded into legal education within the UK. Both at university level for students and with continuing professional development for practitioners. However, there are difficulties fitting such teaching into what is an already very crowded curriculum.

Ben Geen’s legal team are now making a new application to the Criminal Cases Review Commission to have the case reopened. As part of the submission, several prominent statisticians including David Spiegelhalter, Stephen Senn and David Hand have commented on the statistical aspects involved. They stress that they hold no opinion on Geen’s innocence or guilt, but they do believe that the statistical evidence needs to be properly examined and considered.

David Spiegelhalter says that ‘assessing whether events are unusual or surprising should not be left to intuition. I don't trust my own intuition when an apparent “coincidence” occurs, I have to sit down and do the calculations to check whether it's the kind of thing I might expect to occur at some time and place.’

He continues, ‘In general, I don't feel statistical evidence is handled well by courts. They like either incontrovertible numerical “facts”, or overall expert opinions. But statisticians deal with a delicate combination of data and judgement that often gives rise to “rough” numbers, and these don't seem to fit well with the legal process.’

 

The Statistics and Law Section is planning its first meeting at the RSS in May. Anyone interested In its work is invited to contact This email address is being protected from spambots. You need JavaScript enabled to view it. for further information.

The March 2005 issue of Significance magazine was a statistics and the law special issue and explores many of these issues in detail.

Probability Statistics and the Law section

Join the RSS

Join the RSS

Become part of an organisation which works to advance statistics and support statisticians

Copyright 2019 Royal Statistical Society. All Rights Reserved.
12 Errol Street, London, EC1Y 8LX. UK registered charity in England and Wales. No.306096

Twitter Facebook YouTube RSS feed RSS feed RSS newsletter

We use cookies to understand how you use our site and to improve your experience. By continuing to use our site, you accept our use of cookies and Terms of Use.