New study shows scientific reproducibility is hampered by lack of specificity of material resources

Written By Nicole Vasilevsky and Melissa Haendel

A key requirement when performing scientific experiments is the accessibility of material resources, including the reagents or model organisms, needed to address a specific hypothesis. The published scientific literature is a source of this valuable information – namely, the name and vendor of an antibody that will pick up a signal on a Western blot, or the description of a mutant zebrafish strain that shows altered development in a specific tissue and time. But frequently the published literature lacks sufficient detail to the extent that researchers are unable to identify material resources used to perform experiments. Biocurators must often chase down authors to identify these resources to support data ingest into public repositories, which is expensive, time consuming, and not sufficiently effective. To highlight to publishers, editors, reviewers, and authors the significance of the problem, we undertook a study to quantify the issue.

Our study, led by Melissa Haendel, Ph.D. and Nicole Vasilevsky, Ph.D. in OHSU Library’s Ontology Development Group, was published on Sept. 5th in the new journal PeerJ. PeerJ is an Open Access publisher with a novel, open approach to peer review and scholarly publishing. Here we demonstrate the magnitude of the problem, which negatively affects the ability of scientists to reproduce and extend published studies. The study shows that a large number of scientific resources are unidentifiable based on the information reported within the journal articles.

Our study examined nearly 240 articles from more than 80 journals spanning five disciplines: neuroscience, immunology, cell biology, developmental biology and general science. The articles were evaluated to determine if the reported material research resources could be uniquely identified based on the information that was provided in each article, its supplemental data, or prior references. Specific criteria were developed to determine if antibodies, cell lines, constructs, model organisms, and knockdown reagents were identifiable. Based on these criteria, our team also developed guidelines for reporting of research resources. These guidelines areavailable online (http://www.force11.org/node/4433 and http://biosharing.org/bsg-000532) and are being used as a new data standard by authors, reviewers, publishers, and other data contributors to aid reproducibility.

The study showed that just under 50 percent of scientific resources used in previously published articles were unidentifiable, a percentage which varied across resource types and disciplines. While this low value was not unexpected, the actual degree of unidentifiability is still shockingly low. Our study also found no increased level of identification in journals that had more stringent reporting guidelines, suggesting that the reporting guidelines are not strictly adhered to in the evaluation of the research.

The hope is that by quantifying the lack of research reproducibility stemming from inadequate resource identification, that we can highlight to the research and publishing community that there is a significant and pressing need to make material resource information more accessible. The OHSU library is well poised to help researchers with their data management and publishing needs; feel free to contact the library for assistance: library@ohsu.edu.

Link to the Published Version of the article (quote this link in your story – the link will ONLY work after the embargo lifts):  https://peerj.com/articles/148

PeerJ encourages authors to publish the peer reviews, and author rebuttals, for their article. For the purposes of due diligence by the Press, we can provide these materials as a PDF (and they will be published alongside the final article). Please contact us at press@peerj.com to request a copy of the reviews.

Citation to the article:  PeerJ 1:e148 doi: 10.7717/peerj.148

About PeerJ

PeerJ is an Open Access publisher of peer reviewed articles, which offers researchers a lifetime membership, for a single low price, giving them the ability to openly publish all future articles for free. The launch of PeerJ occurred on February 12th, 2013. PeerJ is based in San Francisco, CA and London, UK and can be accessed at https://peerj.com/.

All works published in PeerJ are Open Access and published using a Creative Commons license (CC-BY 3.0). Everything is immediately available—to read, download, redistribute, include in databases and otherwise use—without cost to anyone, anywhere, subject only to the condition that the original authors and source are properly attributed.

For more information, contact:

Nicole Vasilevsky: vasilevs@ohsu.edu

Melissa Haendel: haendel@ohsu.edu

Bookmark and Share

Comments are closed.

About the Author