It is common practice to deem a result ‘significant’ if its calculated p-value is less than 0.05. However, many have questioned this method, to such a degree that last year the editors of the journal Basic and Applied Social Psychology banned the use of it altogether in its published articles.
In order to address the doubts cast over the use of p-values, the ASA assembled more than two dozen experts in this area, including the RSS fellow Stephen Senn, to produce some guidance around proper use. The group met for two days in October 2015 to discuss the issue and over the following months a statement was drafted.
The final statement, published on 7 March 2016, acknowledges that ‘While the p-value can be a useful statistical measure, it is commonly misused and misinterpreted.’ The statement articulates six principles that would improve the understanding of what a p-value represents.
Collectively, the six principles emphasise the limitations of relying on p-values alone as a measure of significance and warns against using p < 0.05 as a line that divides whether something is ’true’ on one side and ‘false’ on the other. ‘Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold,’ the statement stresses.
The principles also warn against cherry picking research findings (also known as ‘data dredging’, ‘selective inference’ and ‘p-hacking’) and supports full disclosure by researchers of all data collection decisions, all statistical analyses conducted and all p-values computed.
The statement concludes by defining what constitutes good statistical practice when presenting the findings of research:
'Good statistical practice, as an essential component of good scientific practice, emphasizes principles of good study design and conduct, a variety of numerical and graphical summaries of data, understanding of the phenomenon under study, interpretation of results in context, complete reporting and proper logical and quantitative understanding of what data summaries mean. No single index should substitute for scientific reasoning.’
ASA President Jessica Utts said: ‘Over time it appears the p-value has become a gatekeeper for whether work is publishable’. This, she says, can lead to potentially important research never being published and can encourage practices such as p-hacking. Ron Wasserstein, ASA’s executive director, said he hopes the statement will steer research into a ‘post p < 0.05 era’.
RSS president Peter Diggle welcomes the ASA statement, saying: 'There is no doubt that p-values are over-used and often wrongly used, but a more constructive response than banning them is to educate people about their uses and limitations - as the ASA statement does and the RSS will also continue to do.'