How Large Quantities of Information Change Everything
Posted on December 2, 2008 Comments (0)
Scale: How Large Quantities of Information Change Everything
There’s another important downside to scale. When we look at large quantities of information, what we’re really doing is searching for patterns. And being the kind of creatures that we are, and given the nature of the laws of probability, we are going to find patterns. Distinguishing between a real legitimate pattern, and something random that just happens to look like a pattern can be somewhere between difficult and impossible. Using things like Bayesian methods to screen out the false positives can help, but scale means that scientists need to learn new methods – both the new ways of doing things that they couldn’t do before, and the new ways of recognizing when they’ve screwed up.
There’s the nature of scale. Tasks that were once simple have become hard or even impossible, because they can’t be done at scale. Tasks that were once impossible have become easy because scale makes them possible. Scale changes everything.
I discussed related ideas on my Curious Cat Management Improvement blog recently: Does the Data Deluge Make the Scientific Method Obsolete?
Related: Seeing Patterns Where None Exists – Mistakes in Experimental Design and Interpretation – Optical Illusions and Other Tricks on the Brain – Data Based Decision Making at Google