Mistakes in Experimental Design and Interpretation

Posted on June 19, 2007  Comments (5)

Mistakes in Experimental Design and Interpretation:

Humans are very good at detecting patterns, but rather poor at detecting randomness. We expect random incidents of cancer to be spread homogeneously, when in fact true randomness results in random clusters, not homogeneity. It is a mistake for an experiment to consider a pool of 47,000 possibilities, and then only report on the 7 cases that seem interesting.

A proper experiment states its hypothesis before gathering evidence and then puts the hypothesis to the test. Remember when you did your seventh grade science fair experiment: you made up a hypothesis first (“Hamsters will get fatter from eating Lucky Charms than Wheaties”) and then did the experiment to confirm or refute the hypothesis. You can’t just make up a hypothesis after the fact to fit the data.

This is an excellent article discussing very common errors in how people use data. We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies.

Related: Seeing Patterns Where None ExistsIllusions, Optical and OtherUnderstanding DataDangers of Forgetting the Proxy Nature of DataHow to Deal with False Research Findingsdescriptive “theory” and normative theory

5 Responses to “Mistakes in Experimental Design and Interpretation”

  1. Curious Cat Science and Engineering Blog » Poor Reporting and Unfounded Implications
    January 1st, 2009 @ 8:35 am

    […] Looking at random data people will find patterns. Sound scientific experimentation is how we learn, not trying to find anything that support our opinions. Statistics don’t lie but ignorant people draw faulty conclusions from data (when they are innumerate – illiteracy with mathematical concepts). […]

  2. Curious Cat Science and Engineering Blog » Data Analysts Captivated by R’s Power
    January 13th, 2009 @ 1:14 pm

    “Close to 1,600 different packages reside on just one of the many Web sites devoted to R, and the number of packages has grown exponentially. One package, called BiodiversityR, offers a graphical interface aimed at making calculations of environmental trends easier…”

  3. Curious Cat Science and Engineering Blog » Soren Bisgaard 1951-2009
    December 20th, 2009 @ 5:21 pm

    I will remember the passion he brought to his work. He reminded me of my father in his desire to improve how things are done and provide people the opportunity to lead better lives…

  4. Curious Cat Management Improvement Blog » Taxes per Person by Country
    April 7th, 2010 @ 1:00 pm

    the idea that data lies is false, and that such a notion is commonly held is a sign of sloppy thinking. You can present data in different ways to focus on different aspects of a system. And you can make faulty assumptions based on data you look at…

  5. Statistical Errors in Medical Studies » Curious Cat Science and Engineering Blog
    February 27th, 2014 @ 8:45 am

    I have written about statistics, and various traps people often fall into when examining data before (Statistics Insights for Scientists and Engineers, Data Can’t Lie – But People Can be Fooled, Correlation is Not Causation, Simpson’s Paradox)…

Leave a Reply