Improved statistical practice is badly needed in biomedical science research, and statistical experts should be called upon more often to ensure this, according to experts working in this particular area of science.
These assertions were made during a symposium held in April this year on the 'Reproducibility and reliability of biomedical research', run jointly by the Academy of Medical Sciences, the Biotechnology and Biological Sciences Research Council (BBSRC), the Medical Research Council (MRC) and the Wellcome Trust. More than 70 experts from academia, industry and academic publishing were in attendance and contributed to the speeches, breakout sessions and panel discussions across the two-day symposium.
The report from the event notes that practices such as 'p-hacking', cherry-picking of data and 'underpowering' field research can hinder the reproducibility of research that drives scientific progress. There is also a call for less emphasis on p-values in publications. P-hacking - also known as data dredging - involves repeatedly searching a dataset or trying alternative analyses until a 'significant' result is found. Underpowered studies are those which are too small to reliably indicate whether or not an effect exists (and if an effect does exist, to estimate its size to any useful level of precision).
The report highlights the importance of good statistical practice to help avoid these pitfalls. 'There is often a need for expert statistical advice, particularly before designing and carrying out an experiment', it states, adding that there might be a greater role for statistics experts on grant review panels and editorial boards. Nature is flagged as an example of a journal committed to examine statistics more closely, which commissions expert input from statisticians when deemed necessary.
The report also notes the 'pressing need to provide opportunities for biomedical researchers to continually update their knowledge of statistics and quantitative skills', acknowledging that it can be difficult to keep up-to-date in this complex and changing field. However, researchers should, according to the report, understand the statistical concepts and assumptions they are employing, 'as a bare minimum'.
The RSS naturally welcomes these findings in the report. 'There is a need, in particular, to understand the limitations of simple methods of analysis, especially when analysing data from complex observational studies,' says RSS president Peter Diggle.
Peter also welcomes the report's call for greater openness and transparency with the data used in research and that data from a project should be considered a 'deliverable', just as a publication is. 'The importance of this cannot be over-stated,' he says. 'Not only data, but also the computer code that produces the results quoted in any publication, should be made accessible to any bona fide researcher who wishes to check their reproducibility.'
The report cites examples of good practice from other areas, such as clinical trials, which it says, 'are generally conducted to a very high standard', and particle physics, where the significance threshold is 'extremely stringent', ie above five-sigma. 'The reason for this is that in particle physics, a 'five sigma' result is paradigm-shifting, whereas most clinical, biomedical and public health research is aimed at incremental improvements in our understanding,' Peter explains.
'There needs to be a different balance between protecting against false positive and false negative findings,' he continues. 'Whether significance testing, at whatever level, is the right way to declare positive results is a moot point. Altman and Bland, among many other medical statisticians, have long argued that estimation (with an associated measure of precision) is usually more useful than testing.'
The organisations involved in this report are now expected to develop and implement the changes suggested. Each will publish an update on its progress within the next 12 months.