There are big problems with medical research, as we have posted about many times in the past. A very significant part of the problem is health care research is very hard. There are all sorts of interactions that make conclusive results much more difficult than other areas.
But failures in our practices also play a big role. Just poor statistical literacy is part of the problem (especially related to things like interactions, variability, correlation that isn’t evidence of causation…). Large incentives that encourage biased research results are a huge problem.
Lies, Damned Lies, and Medical Science
He discovered that the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals. The systemic failure to do adequate long term studies once we approve drugs, practices and devices are also a big problem.
This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”
Another problem is that medical research often doesn’t get the normal scientific inquiry check of confirmation research by other scientists.
Most journal editors don’t even claim to protect against the problems that plague these studies. University and government research overseers rarely step in to directly enforce research quality, and when they do, the science community goes ballistic over the outside interference. The ultimate protection against research error and bias is supposed to come from the way scientists constantly retest each other’s results—except they don’t. Only the most prominent findings are likely to be put to the test, because there’s likely to be publication payoff in firming up the proof, or contradicting it.
Related: Statistical Errors in Medical Studies – Medical Study Integrity (or Lack Thereof) – Contradictory Medical Studies (2007) – Does Diet Soda Result in Weight Gain?