Improving reproducibility in hydrology studies – ScienceDaily

In his paper, "Evaluating Data Availability and Reproducibility of Research in Hydrology and Water Resources," published Feb. 26 in Nature's Scientific Data, David Rosenberg co-authors developed an online research tool to evaluate the reproducibility of published research. The team reviewed 360 articles from six water journals published in 2017. Of 360 articles, they could only replicate the results of four articles.

"Our research tool divides the concept of scientific reproducibility into specific components of data availability, reproducibility of results and replicability of results," said Rosenberg, associate professor of civil and environmental engineering at USU. "We suggest, then, how authors, journals, funders and institutions can use the research tool to increase low reproducibility rates."

The authors say that reproducibility can be divided into three components:

• Are the data, models, codes, instructions for use and other artifacts used at work available?
• Can artifacts be used to reproduce published results?
• Can discoveries be replicated with new datasets?

The team's online research tool consists of 15 questions and provides a checklist of the essentials needed for artifact availability and reproducibility of results. Artifacts is a comprehensive term for all data, software, models, codes, instructions, and other materials needed to reproduce the results in a study.

The team found that about 70% of the articles in the sample stated that some materials were available, but only about 48% of the materials could be accessed online. Only about 6% of the sample articles made the artifacts publicly available, and only 1% of the sample articles made the artifacts available and could be fully reproduced.

The authors said that many articles were missing instructions to generate results. If authors provide guidance, they say, the number of articles that could be tested for reproducibility would double. Articles that made all available artifacts had a six-fold chance of having some or all of their results reproduced. Two journals researched by the team required articles to declare how artefacts could be accessed, and four journals encouraged statements. No journal required authors to make available all artifacts.

The research tool can help recognize and encourage authors to achieve certain levels of reproducibility. For example, authors can use the search tool to evaluate the reproducibility of their results. Rosenberg and his team also recommend a system of medals to recognize different levels of reproducibility:

• Bronze Medal: All artifacts are made available in the article or in open repositories
• Silver Medal: All artifacts are made available and the results are fully reproducible
• Gold Medal: The results are fully reproducible and the general findings can be replicated in different configurations with the same or different artifacts

Rosenberg recommends that medal icons be placed alongside online articles to recognize authors for their reproducibility work and make it easier for readers to find best reproducibility practices. Rosenberg and his team attributed four silver medals and six bronze medals from the 360 ​​articles they reviewed. The awarding of gold medals for replicability of results continues to be an important line of future work.

"We hope the research tool will help authors, journals, funders and institutions make scientific work more reproducible," Rosenberg said. "We welcome the discussion to improve the research tool and improve the reproducibility of our science."

Story Source:

Materials provided by Utah State University. Original by Lexie Richins. Note: Content can be edited by style and size.