Four Ways To Reduce Bad Scientific Research

An experimental psychologist at the University of Oxford argues that in two decades, “we will look back on the past 60 years — particularly in biomedical science — and marvel at how much time and money has been wasted on flawed research.” [M]any researchers persist in working in a way almost guaranteed not to deliver meaningful results. They ride with what I refer to as the four horsemen of the reproducibility apocalypse: publication bias, low statistical power, P-value hacking and HARKing (hypothesizing after results are known). My generation and the one before us have done little to rein these in. In 1975, psychologist Anthony Greenwald noted that science is prejudiced against null hypotheses; we even refer to sound work supporting such conclusions as ‘failed experiments’… The problems are older than most junior faculty members, but new forces are reining in these four horsemen. First, the field of meta-science is blossoming, and with it, documentation and awareness of the issues. We can no longer dismiss concerns as purely theoretical. Second, social media enables criticisms to be raised and explored soon after publication. Third, more journals are adopting the ‘registered report’ format, in which editors evaluate the experimental question and study design before results are collected — a strategy that thwarts publication bias, P-hacking and HARKing. Finally, and most importantly, those who fund research have become more concerned, and more strict. They have introduced requirements that data and scripts be made open and methods be described fully. I anticipate that these forces will soon gain the upper hand, and the four horsemen might finally be slain.