A number of replication projects have been undertaken in recent years the purpose of which is to reproduce scientific findings. And the findings are not pretty. Many studies have failed to replicate. "In psychology, only 39% of the experiments yielded significant findings in the replication study, compared to 97% of the original experiments. In economics, 61% of 18 studies replicated, and among Nature/Science publications, 62% of 21 studies did. In addition, the relative effect sizes of findings that did replicate were only 75% of the original ones. For failed replications, they were close to 0%.
"Replication is the process of verifying studies by having independent researchers repeat them. Replication allows us to confirm that the results are consistent and can be relied on. Studies that fail to replicate are less likely to be true. Studies that do replicate successfully are more likely to be true.
Researchers from the Rady School of Management at the University of California, San Diego looked at three recent replication studies to see whether papers that failed to replicate were cited significantly more than papers that replicated. "The number of citations is a basic measure that is used to assess the scholarly impact of a published work. It is used to study intellectual history and evaluate the quality of scientific work across a variety of disciplines. In promotion decisions, for example, most academic institutions use citations as an important metric in the decision of whether to promote a faculty member."
Research that is less likely to be true is cited more often than research that is more likely to be true
The researchers found that "papers that fail to replicate are cited more than those that are replicable." "On average, papers that failed to replicate [those that are less likely to be true] are cited almost 16 times more per year." They also found that papers that were successfully replicated, or those that were more likely to be true, were cited 153 times less than those that failed! These differences remained even after the failed replication was published. "Notably, only a minority of publications after the failed replications were published acknowledge the failure."
The authors offer a number of reasons for why non-replicable papers are published and cited. One reason they offer is that when a paper is interesting (as in large effect sizes), the review team may apply lower standards. Journals, even prestigious ones, prefer to publish studies showing large effects. Studies with small or null effects are less likely to get published, and are often not even submitted for publication. This is a problem that plagues not only psychology, but also economics, general science, medicine, neuroscience, and genetics.
Research with exaggerated claims receive more media coverage
Another reason the authors offer is that "... some papers create 'hype' using exaggerated and inaccurate claims regarding their findings." Interesting findings tend to receive more media coverage and become famous. And this effect "lingers even when the study is discredited."
Hans Eysenck, one of the worst scandals in the history of science
Hans Eysenck, one of the most cited psychologists of the 20th century, researched and wrote extensively about the relationship between personality and disease. Together with a colleague, he published findings that cancer and other fatal diseases were caused primarily by personality. Their findings were the most astonishing ever to be published in the scientific literature. In one study, they found that 38.5% of subjects with cancer-prone personalities had died of cancer! His research has since been discredited.
One researcher wrote in 2019: "In my opinion, it is one of the worst scandals in the history of science, not least because the Heidelberg results have sat in the peer-reviewed literature for nearly three decades while dreadful and detailed allegations have remained uninvestigated. In the meantime, these widely cited studies have had direct and indirect influences on some people’s smoking and lifestyle choices. This means that for an unknown and unknowable number of individual men and women, this programme of research has been a contributory factor in premature illness and death."
Where you obtain your neuroscience information matters
Media likes sensational findings. Media doesn't often distinguish between correlational studies and randomized controlled trials. Correlational studies don't imply causation, only a relationship between the two variables: Variable A could have caused variable B, variable B could have caused variable A, or the two variables could have been caused by a third variable that wasn't tested. There is no way to know. Randomized controlled trials do imply causation: That one variable caused the other. Whenever you see the words "predict", "related", or "associated," the study is correlational.
Serra-Garcia, M., & Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21), eabd1705. doi: 10.1126/sciadv.abd1705