Challenges for Investigators—Generating Reproducible Research Results

This post is part of a series on emerging research challenges and solutions. The introduction to the series is available here, and all posts in the series are available here.

By John P.A. Ioannidis, MD, DSc, C.F. Rehnborg Chair in Disease Prevention, Professor of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics, and Co-Director, Meta-Research Innovation Center at Stanford (METRICS), Stanford University

Generating reproducible research results is not an easy task. As discussions about a reproducibility crisis become more common and occasionally heated, investigators may feel intimidated or even threatened, caught in the middle of the reproducibility wars. Some feel that the mounting pressure to deliver (both quantity and quality) may be threatening the joy of doing science and even the momentum to explore bold ideas. However, this is a gross misunderstanding. The effort to understand the shortcomings of reproducibility in our work and to find ways to improve our research standards is not some sort of externally imposed police auditing. It is a grassroots movement that stems from scientists themselves who want to improve their work, including its validity, relevance, and utility.

As it has been clarified before, reproducibility of results is just one of many aspects of reproducibility. It is difficult to deal with it in isolation, without also considering reproducibility of methods and reproducibility of inferences. Reproducibility of methods is usually impossible to assess, because unfortunately the triplet of software, script/code, and complete raw data is hardly ever available in a complete functional form. Lack of reproducibility of inferences leads to debates, even when the evidence seems strong and well-rounded. Reproducibility of results, when considered in the context of these other two reproducibility components, is unevenly pursued across disciplines. Some fields like genetic epidemiology have long understood the importance of routinely incorporating replication as a sine qua non in their efforts. Others still consider replication as second-class, “me too” research. Nevertheless, it can be shown (see Ioannidis, Behavioral and Brain Sciences, in press), that in most circumstances replication has at least the same value—and often more value—than original discovery. However, this leads to the question: how do we reward and incentivize investigators to follow a reproducible research path?

Read More

Challenges for Journals—Encouraging Sound Science

This post is part of a series on emerging research challenges and solutions. The introduction to the series is available here, and all posts in the series are available here.

By Barbara A. Spellman, Professor of Law and Professor of Psychology, University of Virginia School of Law

Journals and scientists should be BFFs. But currently they are frenemies. Or, in adult-speak:

Journals play an important role in ensuring that the scientific enterprise is sound. Their most obvious function is to publish science—good science, science that has been peer-reviewed by experts and is of interest to a journal’s readership. But in fulfilling that mission, journals may provide incentives to scientists that undermine the quality of published science and distort the scientific record.

Journal policies certainly contributed to the replication crisis. As businesses, publishers (appropriately) want to make money; to do so they need people to buy, read, and cite their journals. To make that happen, editors seek articles that are novel, that confirm some new hypothesis, and that have clear results. Scientists know that editors want articles with these qualities. Accordingly, scientists may (knowingly or not) bias the scientific process to produce that type of result.

Read More

Systems Matter: Research Environments and Institutional Integrity

This post is part of a series on emerging research challenges and solutions. The introduction to the series is available here, and all posts in the series are available here.

By CK Gunsalus, Director, National Center for Professional and Research Ethics (NCPRE), University of Illinois Urbana-Champaign

We know what it takes for institutions and scholars to produce high-quality, high-integrity research, and yet we do not always act upon that knowledge. As far back as 1988, Paul J. Friedman described both the roots of systemic shortcoming and approaches for conducting trustworthy research. Despite a clear understanding of the issues and steps that would improve our research and educational environments, the academy continues to be dogged by those same systemic issues. A recent National Academies of Sciences, Engineering and Medicine consensus study, Fostering Integrity in Research, in which I participated as a panel member, explores that same disconnect and makes recommendations. The bottom line is this: we must shift our attention and energy away from individual bad actors—though they exist and must be addressed—and toward the highly complex ecosystem within which research is conducted.

An update of an earlier appraisal published 1992, the 2017 NASEM report describes the transformation of research through advances in technology, globalization, increased interdisciplinarity, growing competition, and multiplying policy applications. It identifies six core values underlying research integrity—objectivity, openness, accountability, honesty, fairness and stewardship—and outlines best practices, including checklists, for all aspects of the research enterprise. I encourage you to read it and use these tools in your own work.

All the reports in the world won’t improve research integrity, however, if we don’t do the work in our institutions, departments, and research groups. There are many components to this effort, some of which are discussed in separate posts by my colleagues John P.A. Ioannidis and Barbara A. Spellman elsewhere in this symposium. Let’s focus here on institutional infrastructure.

Read More

The First Human Body Transplant – Ethical and Legal Considerations

By Ana S. Iltis, PhD

brain_glowingprofileTo what lengths should we go to preserve human life? This is a question many are asking after hearing that three men plan to make medical history by conducting the first human head transplant. Or, rather, whole body transplant. Italian neurosurgeon Dr. Sergio Canavero and Chinese surgeon Dr. Xiaoping Ren plan to provide a Russian volunteer, Valery Spiridonov, a new body. During the procedure, Spiridonov’s body and head would be detached and, with the help of a crane, surgeons would move the head and attach it to the donor body.  But is this ethical? What role might law and regulation play in monitoring them or in assessing their conduct after the fact?

Critics call the plan crazy, unethical, and sure to fail. The likelihood of success is very low and the risk of Spiridinov dying is high. Spiridonov says that as soon as animal studies confirm the possibility of survival, the risks will be worth taking. He has Werdnig-Hoffmann Disease, a genetic disorder that destroys muscle and nerve cells. He is confined to a wheelchair and has lived longer than expected. Body transplantation offers him the best chance at a life worth living. Read More

Conflict of Interests Disclosures Come to PubMed

By Christopher Robertson

Scholars and policymakers have long been concerned that the biomedical science literature — and thus the practice of medicine — is biased by the companies who fund research on their own products.  Prior research has shown that industry-funded studies tend to produce results favorable to their company sponsors.  One solution is disclosure of industry funding, so that physicians and other consumers of the biomedical literature can weigh scientific findings accordingly.

My prior work with Aaron Kesselheim, Susannah Rose, and others has found that adding such disclosures to biomedical abstracts could make a big difference — physicians understand them and will rely upon them.  Nonetheless, most journals bury the disclosures at the end of articles, which are often hidden behind paywalls and not nearly as salient as the methods and findings displayed in the abstract.  For the Institutional Corruption Lab of the Edmond J. Safra Center, I worked with a team of hackers to create a browser extension that proves the feasibility of adding those disclosures into PubMed, a Federal government database of the scientific literature.

Thankfully, that browser extension is becoming obsolete, as the National Library of Medicine (part of the NIH) has begun implementing such disclosures themselves, right in PubMed.   A search reveals that nearly 80,000 abstracts now have such tags.  While a lot in absolute terms, it is a small minority of the 17 million abstracts covered by PubMed.  Commentators have suggested that as much as 70% of the funding for clinical trials comes from industry, so we should expect millions of abstracts to have such disclosures.

Thus we are still a long way from comprehensive and effective disclosure.  There are two problems. Read More

NPRM Symposium: How Should We Think About Whether To Donate Our Leftover, Non-Identified Tissue to Research?

By Michelle Meyer

Proposed changes to the federal Common Rule would ask patients for the first time to decide whether to allow their non-identified, leftover tissue to be used for research or thrown away. For that choice to be meaningful, the public needs to be aware of the nature, risks, and benefits of biospecimens research, and of what the proposed changes will—and will not—do. In my latest Forbes essay, “No, Donating Your Leftover Tissue To Research Is Not Like Letting Someone Rifle Through Your Phone,” I consider the power of analogies and other reflections on Rebecca Skloot’s recent New York Times op-ed on the NPRM.

New Developments in the Guatemala STD Experiments Case

In the late 1940s, US government scientists, in collaboration with Guatemalan counterparts, were involved in a horrible array of experiments on human subjects in which a variety of vulnerable groups in Guatemala were intentionally infected with syphilis, gonorrhea, and chancroid and left without treatment. [For more on how they ended up in Guatemala and the ethics of intentional infection studies, see my work here and here.] The experiments were done without consent and without scientific rigor, violating both contemporaneous and modern ethical standards.  They were not uncovered, however, until a few years ago when a historian discovered the files in the midst of doing archival research on one of the scientists, who had also been involved in the Tuskegee syphilis study in the US.

Since her discovery, the US and Guatemalan governments have both issued apologies and reports condemning the studies (here and here), and the US pledged a relatively small amount of money to support the Guatemalan government’s efforts to improve surveillance and control of H.I.V. and other sexually transmitted diseases in that country. However, individual compensation to the victims of the experiments and their families has not been forthcoming; the victims calls for a voluntary compensation program to be established have gone unheeded, and they have also been unable to prevail in court, for a variety of jurisdictional and technical reasons.

As Glenn Cohen and I argued following the victims’ first court loss in 2012, compensation is a moral imperative.  We expressed support for a voluntary compensation program, but in its absence, alternative mechanisms of justice are essential.  Therefore, we were heartened to hear that a petition for the victims was just filed in the Inter-American Commission on Human Rights in Washington, D.C., by the Office of Human Rights for the Archdiocese of Guatemala, represented by the UC Irvine School of Law International Human Rights Clinic and The City Project of Los Angeles.  The petition claims violations of the rights to life, health, freedom from torture, and crimes against humanity under both the American Declaration of the Rights and Duties of Man, and the American Convention on Human Rights, as well as the denial of a right to a remedy for human rights violations.

There are still a number of hurdles ahead, not the least of which is determining which individuals would actually be entitled to compensation, as the record keeping in the initial experiments was so poor and so much time has passed.  But we are heartened that advocates are still pressing forward for these victims and hope that justice, though certainly delayed, will not continue to be denied.

More information on the petition is available here.

NPRM Summary from HHS

As Michelle noted, the Notice of Proposed Rule Making (NPRM) on human subjects research is out after a long delay. For my (and many Bill of Health bloggers’) view about its predecessor ANPRM, you can check out our 2014 book, Human Subjects Research Regulation: Perspectives on the Future.

Here is HHS’s own summary of what has changed and what it thinks is most important:

The U.S. Department of Health and Human Services and fifteen other Federal Departments and Agencies have announced proposed revisions to modernize, strengthen, and make more effective the Federal Policy for the Protection of Human Subjects that was promulgated as a Common Rule in 1991.  A Notice of Proposed Rulemaking (NPRM) was put on public display on September 2, 2015 by the Office of the Federal Register.  The NPRM seeks comment on proposals to better protect human subjects involved in research, while facilitating valuable research and reducing burden, delay, and ambiguity for investigators. It is expected that the NPRM will be published in the Federal Register on September 8, 2015.  There are plans to release several webinars that will explain the changes proposed in the NPRM, and a town hall meeting is planned to be held in Washington, D.C. in October. Read More

Black Lives Matter To Human Research–Lessons From ‘On The Run’

On-the-Run-w-Goffman

By Michele Goodwin

In her recent publication, On The Run, University of Wisconsin sociology professor, Alice Goffman writes about embedded research from 2002-2007 in a “ghetto” community she names 6th Street (located in Philadelphia).  The African American residents of this community are mostly poor and tethered to the  criminal justice system as parolees, on probation, and in and out of jail.  Goffman’s human research subjects comprised the jailed, imprisoned, and minors–IRBs generally describe these populations as “vulnerable.”

On The Run is hailed as original, creative, and transgressive because of Goffman’s lengthy stay in such a descriptively chilling, dangerous, and Black neighborhood–where frequent gun battles teach kids to dive for cover, the women are teen mothers or crack addicted, and law enforcement incessantly polices the community. Indeed, she moves into the neighborhood and lives with three of the 6th Street boys.  Much could be gained from documenting the challenges in such a community, particularly given the troubling patterns of mass incarceration in the U.S.  However, the book raises questions about what represents credibility, quality, and rigor in social science research; the book lacks an index, bibliography, and meaningful citations.  I write about these concerns and more in a forthcoming Texas Law Review essay, which can be found here.

Reviewers lauded the rigor and ignored ethics of the book, agreeing with Goffman’s Princeton advisor, Professor Mitchell Duneier, and his NY Times assessment that  “[t]he level of immersion is really unusual,” because “[s]he got access to the life of the ghetto and came to understand aspects of it we don’t ever get to see.”  Yet, therein resides a significant problem. Fascination with the ghetto and perceptions that life in inner-cities is so bad that researchers can’t possibly expose those human subjects to risks and harms may have blinded the book’s many reviewers to the fact that Black lives matter, including in human research.  It might have also implied a lower standard for rigor; it is rare that an academic book lacks a bibliography and index.  Goffman also destroyed her field notes. These concerns becomes starkly relevant when she writes about her desire and collaboration with “Mike” to kill a man from the neighboring 4th Street.

Read More

The Ethics of Punking the Diet-Research Media Complex (and Millions of Readers)

Scale

By Michelle Meyer

A remarkable new “sting” of the “diet research-media complex” was just revealed. It tells us little we didn’t already know and has potentially caused a fair amount of damage, spread across millions of people. It does, however, offer an opportunity to explore the importance of prospective group review of non-consensual human subjects research—and the limits of IRBs applying the Common Rule in serving that function in contexts like this.

Journalist John Bohannon, two German reporters, a doctor and a statistician recruited 16 German subjects through Facebook into a three-week randomized controlled trial of diet and weight loss. One-third were told to follow a low-carb diet, one-third were told to cut carbs but add 1.5 ounces of dark chocolate (about 230 calories) per day, and one-third served as control subjects and were told to make no changes to their current diet. They were all given questionnaires and blood tests in advance to ensure they didn’t have diabetes, eating disorders, or other conditions that would make the study dangerous for them, and these tests were repeated after the study. They were each paid 150 Euros (~$163) for their trouble.

But it turns out that Bohannon, the good doctor (who had written a book about dietary pseudoscience), and their colleagues were not at all interested in studying diet. Instead, they wanted to show how easy it is for bad science to be published and reported by the media. The design of the diet trial was deliberately poor. It involved only a handful of subjects, had a poor balance of age and of men and women, and so on. But, through the magic of p-hacking, they managed several statistically significant results: eating chocolate accelerates weight loss and leads to healthier cholesterol levels and increased well-being. Read More