How to Deal with False Research Findings

Posted on March 1, 2007  Comments (3)

The Science of Getting It Wrong: How to Deal with False Research Findings by JR Minkel adds to our recent spate of posts on drawing faulty conclutions from data (such as: Correlation is Not Causation, Cancer Deaths – Declining Trend?, Seeing Patterns Where None Exists, Karl Popper Webcast).

In his widely read 2005 PLoS Medicine paper, Ioannidis, a clinical and molecular epidemiologist, attempted to explain why medical researchers must frequently repeal past claims. In the past few years alone, researchers have had to backtrack on the health benefits of low-fat, high-fiber diets and the value and safety of hormone replacement therapy as well as the arthritis drug Vioxx, which was pulled from the market after being found to cause heart attacks and strokes in high-risk patients.

Using simple statistics, without data about published research, Ioannidis argued that the results of large, randomized clinical trials—the gold standard of human research—were likely to be wrong 15 percent of the time and smaller, less rigorous studies are likely to fare even worse.

Among the most likely reasons for mistakes, he says: a lack of coordination by researchers and biases such as tending to only publish results that mesh with what they expected or hoped to find. Interestingly, Ioannidis predicted that more researchers in the field are not necessarily better—especially if they are overly competitive and furtive, like the fractured U.S. intelligence community, which failed to share information that might have prevented the September 11, 2001, terrorist strikes on the World Trade Center and the Pentagon.

But Ioannidis left out one twist: The odds that a finding is correct increase every time new research replicates the same result, according to a study published in the current PLoS Medicine.

3 Responses to “How to Deal with False Research Findings”

  1. CuriousCat: Contradictory Medical Studies
    July 30th, 2007 @ 11:09 pm

    “The problem is, the world of medical and health research is messier than most people realize. Black-and-white answers are rare, even when it comes to a single drug trial…”

  2. Curious Cat Science and Engineering Blog » Mistakes in Experimental Design and Interpretation
    August 29th, 2008 @ 4:13 pm

    We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies…

  3. Statistical Errors in Medical Studies » Curious Cat Science and Engineering Blog
    February 27th, 2014 @ 8:45 am

    […] for medical studies presenting misleading results (Why Most Published Research Findings Are False, How to Deal with False Research Findings, Medical Study Integrity (or Lack Thereof), Surprising New Diabetes Data). This post collects some […]

Leave a Reply