Mistakes in Experimental Design and Interpretation
Posted on June 19, 2007 Comments (4)
Mistakes in Experimental Design and Interpretation:
Humans are very good at detecting patterns, but rather poor at detecting randomness. We expect random incidents of cancer to be spread homogeneously, when in fact true randomness results in random clusters, not homogeneity. It is a mistake for an experiment to consider a pool of 47,000 possibilities, and then only report on the 7 cases that seem interesting.
…
A proper experiment states its hypothesis before gathering evidence and then puts the hypothesis to the test. Remember when you did your seventh grade science fair experiment: you made up a hypothesis first (“Hamsters will get fatter from eating Lucky Charms than Wheaties”) and then did the experiment to confirm or refute the hypothesis. You can’t just make up a hypothesis after the fact to fit the data.
…
A proper experiment states its hypothesis before gathering evidence and then puts the hypothesis to the test. Remember when you did your seventh grade science fair experiment: you made up a hypothesis first (“Hamsters will get fatter from eating Lucky Charms than Wheaties”) and then did the experiment to confirm or refute the hypothesis. You can’t just make up a hypothesis after the fact to fit the data.
This is an excellent article discussing very common errors in how people use data. We have tendencies that lead us to draw faulty conclusions from data. Given that it is important to understand what common mistakes are made to help us counter the natural tendencies.
Related: Seeing Patterns Where None Exists – Illusions, Optical and Other – Understanding Data – Dangers of Forgetting the Proxy Nature of Data – How to Deal with False Research Findings – descriptive “theory” and normative theory
4 Responses to “Mistakes in Experimental Design and Interpretation”
Leave a Reply
RSS Feed
January 1st, 2009 @ 8:35 am
[...] Looking at random data people will find patterns. Sound scientific experimentation is how we learn, not trying to find anything that support our opinions. Statistics don’t lie but ignorant people draw faulty conclusions from data (when they are innumerate – illiteracy with mathematical concepts). [...]
January 13th, 2009 @ 1:14 pm
[...] Mistakes in Experimental Design and Interpretation – Data Based Decision Making at Google – Freeware Math Programs – How Large Quantities of [...]
December 20th, 2009 @ 5:21 pm
[...] The Work of Peter Scholtes – Mistakes in Experimental Design and Interpretation – The Scientific Context of Quality Improvement by George Box and Soren Bisgaard, 1987 [...]
April 7th, 2010 @ 1:00 pm
[...] Fooled by Randomness – Simpson’s Paradox – Mistakes in Experimental Design and Interpretation – Government Debt as Percentage of GDP 1990-2008 by Country – Communicating with the [...]