Category: Research Integrity and Trustworthy Science Symposium

This post is part of a series on emerging research challenges and solutions. The introduction to the series is available here, and all posts in the series are available here.

By John P.A. Ioannidis, MD, DSc, C.F. Rehnborg Chair in Disease Prevention, Professor of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics, and Co-Director, Meta-Research Innovation Center at Stanford (METRICS), Stanford University

Generating reproducible research results is not an easy task. As discussions about a reproducibility crisis become more common and occasionally heated, investigators may feel intimidated or even threatened, caught in the middle of the reproducibility wars. Some feel that the mounting pressure to deliver (both quantity and quality) may be threatening the joy of doing science and even the momentum to explore bold ideas. However, this is a gross misunderstanding. The effort to understand the shortcomings of reproducibility in our work and to find ways to improve our research standards is not some sort of externally imposed police auditing. It is a grassroots movement that stems from scientists themselves who want to improve their work, including its validity, relevance, and utility.

As it has been clarified before, reproducibility of results is just one of many aspects of reproducibility. It is difficult to deal with it in isolation, without also considering reproducibility of methods and reproducibility of inferences. Reproducibility of methods is usually impossible to assess, because unfortunately the triplet of software, script/code, and complete raw data is hardly ever available in a complete functional form. Lack of reproducibility of inferences leads to debates, even when the evidence seems strong and well-rounded. Reproducibility of results, when considered in the context of these other two reproducibility components, is unevenly pursued across disciplines. Some fields like genetic epidemiology have long understood the importance of routinely incorporating replication as a sine qua non in their efforts. Others still consider replication as second-class, “me too” research. Nevertheless, it can be shown (see Ioannidis, Behavioral and Brain Sciences, in press), that in most circumstances replication has at least the same value—and often more value—than original discovery. However, this leads to the question: how do we reward and incentivize investigators to follow a reproducible research path?

Read More

This post is part of a series on emerging research challenges and solutions. The introduction to the series is available here, and all posts in the series are available here.

By CK Gunsalus, Director, National Center for Professional and Research Ethics (NCPRE), University of Illinois Urbana-Champaign

We know what it takes for institutions and scholars to produce high-quality, high-integrity research, and yet we do not always act upon that knowledge. As far back as 1988, Paul J. Friedman described both the roots of systemic shortcoming and approaches for conducting trustworthy research. Despite a clear understanding of the issues and steps that would improve our research and educational environments, the academy continues to be dogged by those same systemic issues. A recent National Academies of Sciences, Engineering and Medicine consensus study, Fostering Integrity in Research, in which I participated as a panel member, explores that same disconnect and makes recommendations. The bottom line is this: we must shift our attention and energy away from individual bad actors—though they exist and must be addressed—and toward the highly complex ecosystem within which research is conducted.

An update of an earlier appraisal published 1992, the 2017 NASEM report describes the transformation of research through advances in technology, globalization, increased interdisciplinarity, growing competition, and multiplying policy applications. It identifies six core values underlying research integrity—objectivity, openness, accountability, honesty, fairness and stewardship—and outlines best practices, including checklists, for all aspects of the research enterprise. I encourage you to read it and use these tools in your own work.

All the reports in the world won’t improve research integrity, however, if we don’t do the work in our institutions, departments, and research groups. There are many components to this effort, some of which are discussed in separate posts by my colleagues John P.A. Ioannidis and Barbara A. Spellman elsewhere in this symposium. Let’s focus here on institutional infrastructure.

Read More