In significance testing, a null hypothesis typically posits that changing the input to a system does not change the resulting output. To determine whether a result is statistically significant, a researcher has to calculate a p-value, which is the probability of observing an apparent effect given that the null hypothesis is true. If the p-value is less than 0.05 it is conventionally deemed a statistically significant result.
‘We believe that the p < 0.05 bar is too easy to pass and sometimes serves as an excuse for lower quality research,’ BASP authors David Trafimow and Michael Marks of Mexico State University write in their editorial. They go on to say that they hope other journals will follow suit.
Confidence intervals are also banned, since they ‘do not provide a strong case for concluding that the population parameter of interest is likely to be within the stated interval.’ Bayesian alternatives, however, 'are neither required nor banned' from the journal.
The use of p-values to judge whether research is 'significant' has been thought by many statisticians to be problematic, ever since the idea was introduced by Ronald Fisher in the 1920s. In 2005, Stanford School of Medicine professor John Ioannidis wrote a well-known paper, 'Why Most Published Research Findings Are False', stating that reliance on p-values alone was ‘ill-founded’.
More recently, in November 2014, Royal Society Fellow David Colquhoun published ‘An investigation of the false discovery rate and the misinterpretation of p-values’ stating: 'If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time'.
The American Statistical Association (ASA) argues that banning p-values altogether will have other ‘negative consequences’ in a formal statement regarding the editorial. 'The statistical community is aware of problems associated with the use and interpretation of inferential methods,' it says. 'However, the journal proposes to fall back entirely on descriptive statistics and use "larger sample sizes than is typical in much psychology research." We believe this policy may have its own negative consequences and thus the proper use of inferential methods needs to be analyzed and debated in the larger research community.'
RSS president, Peter Diggle, said: 'The RSS welcomes and shares the BASP editors’ concerns that inferential statistical methods are open to mis-use and mis-interpretation, but does not feel that a blanket ban on any particular inferential method is the most constructive response.'
In our Opinion section, Peter Diggle, Stephen Senn, Andrew Gelman, Geoff Cummings and Robert Grant give their initial reaction to the issues raised.