Why We Should Showcase Research Without Significant Results
Academic journals and peer reviewers across different fields currently favour the visibility and publication of those manuscripts which highlight the statistical significance of their results. In turn, this creates an incentive for researchers to produce strongly significant effects with often outdated research designs. In his latest PNAS paper, Bocconi Professor Scott Williamson joins a consortium of authors in the social sciences arguing that the persistence of such publication bias increases the risk of "false positives", discourages scientific dissemination, hinders replicability and contributes to inefficient spending of research funds. To mitigate these problems, Professor Williamson and hi co-authors propose a new framework called "RARE", or Report All Results Efficiently, which demands better practices by all of the different agents involved in scientific research, from grant agencies to investigators and journals.
A significant part of the problem highlighted by Prof. Williamson and a team of co-authors is the systematic removal of "null results", which are those results that do not display statistical significance. As many journals in the social sciences are more likely to publish results with statistical significance and established causalities, researchers tend to omit those less successful results from their publication. This contributes to the so-called "file drawer" problem, where studies with less evident effects are systematically rejected or withdrawn from publication.
The "file drawer" problem and the omission of "null results" raise several crucial debates. Firstly, excessive emphasis on the statistical significance of results shadows what is perhaps more meaningful in scientific research, which is the emergence of more advanced and appropriate research designs that may produce null results in the earlier stages of their development. Secondly, researchers need to be aware of studies conducted in the past even if they delivered null results to improve them or perhaps even to avoid unnecessary replications. Thirdly, funding agencies would benefit greatly from larger access to unpublished results to avoid inefficient spending.
The "RARE" framework proposed by Prof. Williamson and his co-authors would largely amend these inefficiencies if various actors involved in the research process start to abide by a series of best practices As an example, scientific registries should abide by higher standards for the reporting of null results. Likewise, investigators should commit to publicly reporting results for all pre-registered hypotheses regardless of the findings. Grant agencies should also require researchers to identify registrations from previous grants and to report whether all results have been disseminated in the past, independently from their statistical significance. Finally, journals can play an extremely important role in this paradigm shift by reinforcing incentives on open reporting and by selecting research on the basis of good questions and design rather than on the significance of results.
To conclude, the article calls for a new paradigm for scientific dissemination in the social sciences and beyond. This requires larger transparency from all actors involved in research regarding the significance of results in order to overcome the "file drawer" problem and other publication biases. Abiding by the new "RARE" framework is likely to require extra costs on various actors in the short term, as they are required to change existing procedures on the reporting of null results. However, it is expected to yield large benefits in the future both in terms of efficiency and scientific reputation.
David D. Laitin et al. "Reporting All Results Efficiently: A RARE Proposal to Open up the File Drawer." Proceedings of the National Academy of Sciences Dec 2021, 118 (52) e2106178118; DOI: 10.1073/pnas.2111611118. DOI: https://dx.doi.org/10.1073/pnas.2106178118.