Posts about statistics

Medical Study Findings too Often Fail to Provide Us Useful Knowledge

There are big problems with medical research, as we have posted about many times in the past. A very significant part of the problem is health care research is very hard. There are all sorts of interactions that make conclusive results much more difficult than other areas.

But failures in our practices also play a big role. Just poor statistical literacy is part of the problem (especially related to things like interactions, variability, correlation that isn’t evidence of causation…). Large incentives that encourage biased research results are a huge problem.

Lies, Damned Lies, and Medical Science

He discovered that the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals. The systemic failure to do adequate long term studies once we approve drugs, practices and devices are also a big problem.

This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

Another problem is that medical research often doesn’t get the normal scientific inquiry check of confirmation research by other scientists.

Most journal editors don’t even claim to protect against the problems that plague these studies. University and government research overseers rarely step in to directly enforce research quality, and when they do, the science community goes ballistic over the outside interference. The ultimate protection against research error and bias is supposed to come from the way scientists constantly retest each other’s results—except they don’t. Only the most prominent findings are likely to be put to the test, because there’s likely to be publication payoff in firming up the proof, or contradicting it.

Related: Statistical Errors in Medical StudiesMedical Study Integrity (or Lack Thereof)Contradictory Medical Studies (2007)Does Diet Soda Result in Weight Gain?

Mabel Mercer sings Experiment by Cole Porter

Mabel Mercer sings Experiment by Cole Porter:

[ Video removed 🙁 ]

Lyrics for Experiment:

Before you leave these portals to meet less fortunate mortals,
There’s just one final message I would give to you.
You all have learned reliance on the sacred teachings of science
So I hope through life you never will decline in spite of philistine defiance
To do what all good scientists do.
Make it your motto day and night.
Experiment and it will lead you to the light.
The apple on the top of the tree is never too high to achieve,
So take an example from Eve, experiment.
Be curious, though interfering friends may frown,
Get furious at each attempt to hold you down.
If this advice you only employ, the future can offer you infinite joy
And merriment.
Experiment and you’ll see.

The lyrics were included in the book by George Box, my father and Stu Hunter: Statistics for Experimenters.

Related: Scientists Singing About ScienceHere Comes Science by They Might Be GiantsThey Will Know We are Christians By Our LoveCambrian Explosion Song

Introduction to Fractional Factorial Designed Experiments

Scientific inquiry is aided by sensible application of statistical tools. I grew up around the best minds in applied statistics. My father was an eminent applied statistican, and George Box (the person in the video) was often around our house (or we were at his). Together they wrote Statistics for Experimenters (along with Stu Hunter, not related to me) the bible for design of experiments (George holds up the 1st edition in the video).

The video may be a bit confusing without at least a basic idea of factorial designed experiments. These introductory videos, by Stu Hunter, on Using Design of Experiments to Improve Results may help get you up to speed.

This video looks at using fractional factorials to reduce the number of experiments needed when doing a multifactor experiment. I grew up understanding that the best way to experiment is by varying multiple factors at the same time. You learn much quicker than One Factor At a Time (OFAT), and you learn about interactions (which are mainly lost in OFAT). I am amazed to still hear scientists and engineers talk about OFAT as a sensible method or even as the required method, but I know many do think that way.

To capture the interactions a full factorial requires an ever larger number of experimental runs to be complete. Assessing 4 factors requires 16 runs, 6 would require 64 and 8 would require 256. This can be expensive and time consuming. Obviously one method is to reduce the number of factors to experiment with. That is done (by having those knowledgable about the process include only those factors worth the effort), but if you still have, for example, 8 very important factors using a fractional factorial design can be very helpful.

And as George Box says “What you will often find is that there will be redundant factors… and don’t forget about those redundant factors. Knowing that something doesn’t matter is almost as important as knowing what does.” If you learn a factor isn’t having an affect you may be able to save money. And you can eliminate varying that factor in future experiments.

Continue reading

George Box 1919 to 2013 – A Great Friend, Scientist and Statistician

Reposted from my management blog.

I would most likely not exist if it were not for George Box. My father took a course from George while my father was a student at Princeton. George agreed to start the Statistics Department at the University of Wisconsin – Madison, and my father followed him to Madison, to be the first PhD student. Dad graduated, and the next year was a professor there, where he and George remained for the rest of their careers.

George died today, he was born in 1919. He recently completed An Accidental Statistician: The Life and Memories of George E. P. Box which is an excellent book that captures his great ability to tell stories. It is a wonderful read for anyone interested in statistics and management improvement or just great stories of an interesting life.

photo of George EP Box

George Box by Brent Nicastro.

George Box was a fantastic statistician. I am not the person to judge, but from what I have read one of the handful of most important applied statisticians of the last 100 years. His contributions are enormous. Several well know statistical methods are known by his name, including:

George was elected a member of the American Academy of Arts and Sciences in 1974 and a Fellow of the Royal Society in 1979. He also served as president of the American Statistics Association in 1978. George is also an honorary member of ASQ.

George was a very kind, caring and fun person. He was a gifted storyteller and writer. He had the ability to present ideas so they were easy to comprehend and appreciate. While his writing was great, seeing him in person added so much more. Growing up I was able to enjoy his stories often, at our house or his. The last time I was in Madison, my brother and I visited with him and again listened to his marvelous stories about Carl Pearson, Ronald Fisher and so much more. He was one those special people that made you very happy whenever you were near him.

George Box, Stuart Hunter and Bill Hunter (my father) wrote what has become a classic text for experimenters in scientific and business circles, Statistics for Experimenters. I am biased but I think this is acknowledged as one of (if not the) most important books on design of experiments.

George also wrote other classic books: Time series analysis: Forecasting and control (1979, with Gwilym Jenkins) and Bayesian inference in statistical analysis. (1973, with George C. Tiao).

George Box and Bill Hunter co-founded the Center for Quality and Productivity Improvement at the University of Wisconsin-Madison in 1984. The Center develops, advances and communicates quality improvement methods and ideas.

The Box Medal for Outstanding Contributions to Industrial Statistics recognizes development and the application of statistical methods in European business and industry in his honor.

All models are wrong but some are useful” is likely his most famous quote. More quotes By George Box

A few selected articles and reports by George Box

Related: It is not about proving a theorem it is about being curious about thingsBox on QualitySoren BisgaardLearning Design of Experiments with Paper HelicoptersPeter Scholtes

Cancer Risks From Our Food

comic showing the dangers of drawing false conclusion based on statistical significance

Randall Munroe illustrates RA Fisher’s point that you must think to draw reasonable conclusions from data. Click the image to see the full xkcd comic.

Pretty much everything you eat is associated with cancer. Don’t worry about it. by Sarah Kliff

The changes in cancer risk were all over the map: 39 percent found an increased risk, 33 percent found a decreased risk and 23 percent showed no clear evidence either way.

The vast majority of those studies, Schoenfeld and Ioannidis found, showed really weak associations between the ingredient at hand and cancer risk. A full 80 percent of the studies had shown statistical relationships that were “weak or nominally significant,” as measured by the study’s P-values. Seventy-five percent of the studies purporting to show a higher cancer risk fell into this category, as did 76 percent of those showing a lower cancer risk.

Sadly the evidence is often not very compelling but creates uncertainly in the public. Poorly communicated results and scientific illiteracy (both from publishers and the public) leads to more confusion than is necessary. Even with well done studies, good communication and a scientifically literate population nutrition and human health conclusion are more often questionable than they are clear.

Related: Researchers Find Switch That Allows Cancer Cells to SpreadGlobal Cancer Deaths to Double by 2030Physical Inactivity Leads to 5.3 Million Early Deaths a Year

Medical Studies Showing Largest Benefits Often Prove to be False

There is another study showing the results of health studies often are proven false. Medical studies with striking results often prove false

If a medical study seems too good to be true, it probably is, according to a new analysis.

In a statistical analysis of nearly 230,000 trials compiled from a variety of disciplines, study results that claimed a “very large effect” rarely held up when other research teams tried to replicate them.

The report should remind patients, physicians and policymakers not to give too much credence to small, early studies that show huge treatment effects, Ioannidis said.

The Stanford professor chose to publish this paper in a closed science publication. But previously he published openly on: Why Most Published Research Findings Are False.

Related: Majority of Clinical Trials Don’t Provide Meaningful EvidenceStatistical Errors in Medical StudiesMistakes in Experimental Design and InterpretationHow to Deal with False Research Findings

Today, Most Deaths Caused by Lifetime of Action or Inaction

Chart of the Leading Causes of Death in 1900 and 2010

Our instincts lead us to fear the unknown and immediate threats (probably so we can be ready to run – or maybe fight). But today the biggest risks to an untimely dealt are not lions, other people out to get us, or even just random infection. We have to adapt to the new risks by taking action to eat healthfully and exercise, in the same way we we have evolved to avoid becoming a meal for a hungry beast.

Today the largest causes of death are heart disease and cancer (which account for more than 60% of the deaths causes by the top 10 leading causes of death). The next leading causes are non-infectious airways diseases, cerebrovascular diseases and accidents. Alzheimer’s, diabetes, nephropathies, pneumonia or influenza and suicide make of the rest of the top 10 leading causes.

In 1900 Pneumonia or influenza and tuberculosis took as many lives (per 100,000 people) and cancer and heart disease take today. We have done well decreasing the incidents of death (fewer deaths per 100,000) by greatly reducing and nearly eliminating some causes of death (the 2 leading causes from 1900 are good examples).

Continue reading

Majority of Clinical Trials Don’t Provide Meaningful Evidence

The largest comprehensive analysis of finds that clinical trials are falling short of producing high-quality evidence needed to guide medical decision-making.

The analysis, published today in the Journal of the American Medical Association, found the majority of clinical trials is small, and there are significant differences among methodical approaches, including randomizing, blinding and the use of data monitoring committees.

This is a critical issue as medical studies continue to leave quite a bit to be desired. Even more importantly the failure to systemically study and share evidence of effectiveness once treatments are authorized leaves a great deal to be desired. On top of leaving quite a bit to be desired, the consequences are serious. If we make mistakes for example in how we date fossils it matters but it is unlikely to cause people their lives or health. Failure to adequately manage and analyze health care experiments may very well cost people their health or lives.

“Our analysis raises questions about the best methods for generating evidence, as well as the capacity of the clinical trials enterprise to supply sufficient amounts of high quality evidence to ensure confidence in guideline recommendations,” said Robert Califf, MD, first author of the paper, vice chancellor for clinical research at Duke University Medical Center, and director of the Duke Translational Medicine Institute.

The analysis was conducted by the Clinical Trials Transformation Initiative (CTTI), a public-private partnership founded by the Food and Drug Administration (FDA) and Duke. It extends the usability of the data in for research by placing the data through September 27, 2010 into a database structured to facilitate aggregate analysis.

Related: Statistical Errors in Medical StudiesHow to Deal with False Research FindingsMedical Study Integrity (or Lack Thereof)

Continue reading

Numeracy: The Educational Gift That Keeps on Giving

I like numbers. I always have. This is just luck, I think. I see, how helpful it is to have a good understanding of numbers. Failing to develop a facility with numbers results in many bad decisions, it seems to me.

A new article published in closed anti-science way, sadly (so no link), examines how people who are numerate (like literate but for number—understand) process information differently so that they ultimately make more informed decisions. Cancer risks. Investment alternatives. Calories. Numbers are everywhere in daily life, and they figure into all sorts of decisions.

People who are numerate are more comfortable thinking about numbers and are less influenced by other information, says Ellen Peters of Ohio State University (sadly Ohio State allows research by staff paid by them to be unavailable to the public – sad), the author of the new paper. For example, in one of Peters’s studies, students were asked to rate undergraduates who received what looked like different test scores. Numerate people were more likely to see a person who got 74% correct and a person who got 26% incorrect as equivalent, while people who were less numerate thought people were doing better if their score was given in terms of a percent correct.

People make decisions based on this sort of information all the time. For example, “A lot of people take medications,” Peters says. Every drug has benefits and potential risks, and those can be presented in different ways. “You can talk about the 10 percent of the population that gets the side effect or the 90 percent that does not.” How you talk about it will influence how dangerous the drug seems to be, particularly among people who are less numerate.

Other research has shown that only less numerate people respond differently to something that has a 1 in 100 chance of happening than something that has a 1 percent chance of happening. The less numerate see more risk in the 1 in 100 chance—even though these numbers are exactly the same.

“In general, people who are numerate are better able to bring consistent meaning to numbers and to make better decisions,” Peters says. “It suggests that courses in math and statistics may be the educational gift that keeps on giving.”

Related: full press releaseBigger Impact: 15 to 18 mpg or 50 to 100 mpg?Data Doesn’t Lie, But People Can be FooledUnderstanding Data: Simpson’s Paradoxapplied statistics is not about proving a theorem, it’s about being curious about thingsEncouraging Curiosity in KidsDangers of Forgetting the Proxy Nature of DataCompounding is the Most Powerful Force in the Universe

Google Prediction API

This looks very cool.

The Prediction API enables access to Google’s machine learning algorithms to analyze your historic data and predict likely future outcomes. Upload your data to Google Storage for Developers, then use the Prediction API to make real-time decisions in your applications. The Prediction API implements supervised learning algorithms as a RESTful web service to let you leverage patterns in your data, providing more relevant information to your users. Run your predictions on Google’s infrastructure and scale effortlessly as your data grows in size and complexity.

Accessible from many platforms: Google App Engine, Apps Script (Google Spreadsheets), web & desktop apps, and command line.

The Prediction API supports CSV formatted training data, up to 100M in size. Numeric or unstructured text can be sent as input features, and discrete categories (up to a few hundred different ones) can be provided as output labels.

Language identification
Customer sentiment analysis
Product recommendations & upsell opportunities
Document and email classification

Related: The Second 5,000 Days of the WebRobot Independently Applies the Scientific MethodControlled Experiments for Software SolutionsStatistical Learning as the Ultimate Agile Development Tool by Peter Norvig

Statistical Errors in Medical Studies

I have written about statistics, and various traps people often fall into when examining data before (Statistics Insights for Scientists and Engineers, Data Can’t Lie – But People Can be Fooled, Correlation is Not Causation, Simpson’s Paradox). And also have posted about reasons for systemic reasons for medical studies presenting misleading results (Why Most Published Research Findings Are False, How to Deal with False Research Findings, Medical Study Integrity (or Lack Thereof), Surprising New Diabetes Data). This post collects some discussion on the topic from several blogs and studies.

HIV Vaccines, p values, and Proof by David Rind

if vaccine were no better than placebo we would expect to see a difference as large or larger than the one seen in this trial only 4 in 100 times. This is distinctly different from saying that there is a 96% chance that this result is correct, which is how many people wrongly interpret such a p value.

So, the modestly positive result found in the trial must be weighed against our prior belief that such a vaccine would fail. Had the vaccine been dramatically protective, giving us much stronger evidence of efficacy, our prior doubts would be more likely to give way in the face of high quality evidence of benefit.

While the actual analysis the investigators decided to make primary would be completely appropriate had it been specified up front, it now suffers under the concern of showing marginal significance after three bites at the statistical apple; these three bites have to adversely affect our belief in the importance of that p value. And, it’s not so obvious why they would have reported this result rather than excluding those 7 patients from the per protocol analysis and making that the primary analysis; there might have been yet a fourth analysis that could have been reported had it shown that all important p value below 0.05.

How to Avoid Commonly Encountered Limitations of Published Clinical Trials by Sanjay Kaul, MD and and George A. Diamond, MD

Trials often employ composite end points that, although they enable assessment of nonfatal events and improve trial efficiency and statistical precision, entail a number of shortcomings that can potentially undermine the scientific validity of the conclusions drawn from these trials. Finally, clinical trials often employ extensive subgroup analysis. However, lack of attention to proper methods can lead to chance findings that might misinform research and result in suboptimal practice.

Why Most Published Research Findings Are False by John P. A. Ioannidis
Continue reading

  • Recent Comments:

    • joseph: Commendable effort, Everyone ought to be concerned of the climate
    • Priya Talwar: Hey John! I think you done a very hard work to find cougar’s really must say and keep...
    • kaan: Kudos! to those doctors at Nicklaus Children’s Hospital in Miami! Hope VR and human biotics develop...
    • Khan: thanks for the post
    • Eric LeClair: Hats off to these ocean farmers. Too bad the media doesn’t cover these guys. The real...
    • Eric LeClair: 3d printing is hands down an incredibly technology. However, I totally believe the consumer...
    • Tom: Hi, on the photo is our m-Bitbeam set. More info about this set: m Regards Tom
    • Kazuko Lally: Valuable ideas ! I am thankful for the facts ! Does someone know if my assistant could...
  • Recent Trackbacks:

  • Links