If you haven’t played Guess The Correlation yet, you definitely should – but please read this article first. And make sure you’re not at work. And that you’ve eaten something recently. Once you start playing, you’ll forget whatever else you were doing, or that you were meant to be doing. It’s an addictive little thing.
In September 2015, the World Health Organisation’s (WHO) International Agency for Research in Cancer (IARC) announced the finding of their expert group that processed meat was a ‘Group I carcinogen’, putting it in the same category as cigarettes and asbestos. Cue headlines such as "Bacon, ham and sausages have the same cancer risk as cigarettes warn experts". The WHO had to try and sort out the subsequent confusion by pointing out that the ‘Group 1’ classification was about the confidence of an increased risk of cancer existing, and said nothing about the magnitude of the risk.
Statistics play an important role in improving human lives, not least through helping to identify potential policy needs and by measuring the impact of policy implementations. This was noted more than a month ago, on 20 October, when we celebrated World Statistics Day – which sought to promote the message that better data helps create better lives. But what do we mean by ‘better data’?
I have a mild addiction to playing backgammon on my smartphone. I have tried several free game apps, but none have proven fully satisfying to me. I reserve the status of ‘satisfying’ for a game that is fair and challenging, regardless of the end score. This means that the computer player (CPU) needs to demonstrate some skill and its dice rolls should be random and probabilistic.
Publishing a book is almost always a gamble, since it is very difficult to predict the fate of the year’s novelty. For example, this summer we are assisting to the end of the Harry Potter’s saga of blockbuster films, inspired in the still more successful series of books by J. K. Rowling. In fact, the manuscript of Harry Potter and the Philosopher’s Stone (Sorcerer’s Stone in the US) was turned down by several UK publishers before Bloomsbury accepted the challenge and printed 500 copies to begin with in June 1997 (a standard novel author print run).
Analytics projects often present us with situations in which common sense tells us one thing, while the numbers seem to tell us something much different. Such situations are often opportunities to learn something new by taking a deeper look at the data. Failure to perform a sufficiently nuanced analysis, however, can lead to misunderstandings and decision traps. To illustrate this danger, we present several instances of Simpson’s Paradox in business and non-business environments. As we demonstrate below, statistical tests and analysis can be confounded by a simple misunderstanding of the data. Often taught in elementary probability classes, Simpson’s Paradox refers to situations in which a trend or relationship that is observed within multiple groups reverses when the groups are combined.