Posts about design of experiments

I Just Finished Statistics for Experimenters and I Cannot Praise it Enough

Guest post by Michael Betancourt.

I just finished Box, Hunter, and Hunter (Statistics for Experimenters) and I cannot praise it enough. There were multiple passages where I literally giggled. In fact I may have been a bit too enthusiastic about tagging quotes beyond “all models are wrong but some are useful” that I can’t share them all.

photo of Statistics for Experimenters with many blue bookmarks shown

I wish someone had shared this with me when I was first learning statistics instead of the usual statistics textbooks that treat model development as an irrelevant detail. So many of the elements that make this book are extremely relevant to statistics today. Some examples:

  • The perspective of learning from data only through the lens of the statistical model. The emphasis on sequential modeling, using previous fits to direct better models, and sequential experiments, using past fits to direct better targeted experiments.
  • The fixation on checking model assumptions, especially with interpretable visual diagnostics that capture not only residuals but also meaningful scales of deviation. Proto visual predictive checks as I use them today.
  • The distinction between empirical models and mechanistic models, and the treatment of empirical linear models as Taylor expansions of mechanistic models with covariates as _deviations_ around some nominal value. Those who have taken my course know how important I think this is.
  • The emphasis that every model, even mechanistic models, are approximations and should be treated as such.
  • The reframing of frequentist statistical tests as measures of signal to noise ratios.
  • The importance of process drift and autocorrelation in data when experimental configurations are not or cannot be arbitrarily randomized.
  • The diversity of examples and exercises using real data from real applications with detailed contexts, including units everywhere.

Really the only reason why I wouldn’t recommend this as an absolute must read is that the focus on linear models and use of frequentist methods does limit the relevance of the text to contemporary Bayesian applications a bit.

Texts like these make me even more frustrated by the desire to frame movements like data science as revolutions that give people the justification to ignore the accumulated knowledge of applied statisticians.

Academic statistics has no doubt largely withdrawn into theory with increasingly smaller overlap with applications, but there is so much relevant wisdom in older applied statistics texts like these that doesn’t need to be rediscovered just reframed in a contemporary context.

Oh, I forgot perhaps the best part! BHH continuously emphasizes the importance of working with domain experts in the design and through the entire analysis with lots of anecdotal examples demonstrating how powerful that collaboration can be.

I felt so much less alone every time they talked about experimental designs not being implemented properly andthe subtle effects that can have in the data, and serious effects in the resulting inferences, if not taken into account.

Michael Betancourt, PhD, Applied Statistician – long story short, I am a once and future physicist currently masquerading as a statistician in order to expose the secrets of inference that statisticians have long kept from scientists. More seriously, my research focuses on the development of robust statistical workflows, computational tools, and pedagogical resources that bridge statistical theory and practice and enable scientists to make the most out of their data.
Twitter: @betanalpha
Website: betanalpha
Patreon: Michael Betancourt

Related: Statistics for Experimenters, Second EditionStatistics for Experimenters in SpanishStatistics for Experimenters ReviewCorrelation is Not Causation

Mabel Mercer sings Experiment by Cole Porter

Mabel Mercer sings Experiment by Cole Porter:

[ Video removed 🙁 ]

Lyrics for Experiment:

Before you leave these portals to meet less fortunate mortals,
There’s just one final message I would give to you.
You all have learned reliance on the sacred teachings of science
So I hope through life you never will decline in spite of philistine defiance
To do what all good scientists do.
Experiment.
Make it your motto day and night.
Experiment and it will lead you to the light.
The apple on the top of the tree is never too high to achieve,
So take an example from Eve, experiment.
Be curious, though interfering friends may frown,
Get furious at each attempt to hold you down.
If this advice you only employ, the future can offer you infinite joy
And merriment.
Experiment and you’ll see.

The lyrics were included in the book by George Box, my father and Stu Hunter: Statistics for Experimenters.

Related: Scientists Singing About ScienceHere Comes Science by They Might Be GiantsThey Will Know We are Christians By Our LoveCambrian Explosion Song

Introduction to Fractional Factorial Designed Experiments

Scientific inquiry is aided by sensible application of statistical tools. I grew up around the best minds in applied statistics. My father was an eminent applied statistican, and George Box (the person in the video) was often around our house (or we were at his). Together they wrote Statistics for Experimenters (along with Stu Hunter, not related to me) the bible for design of experiments (George holds up the 1st edition in the video).

The video may be a bit confusing without at least a basic idea of factorial designed experiments. These introductory videos, by Stu Hunter, on Using Design of Experiments to Improve Results may help get you up to speed.

[the video has been removed from the internet]

This video looks at using fractional factorials to reduce the number of experiments needed when doing a multifactor experiment. I grew up understanding that the best way to experiment is by varying multiple factors at the same time. You learn much quicker than One Factor At a Time (OFAT), and you learn about interactions (which are mainly lost in OFAT). I am amazed to still hear scientists and engineers talk about OFAT as a sensible method or even as the required method, but I know many do think that way.

To capture the interactions a full factorial requires an ever larger number of experimental runs to be complete. Assessing 4 factors requires 16 runs, 6 would require 64 and 8 would require 256. This can be expensive and time consuming. Obviously one method is to reduce the number of factors to experiment with. That is done (by having those knowledgable about the process include only those factors worth the effort), but if you still have, for example, 8 very important factors using a fractional factorial design can be very helpful.

And as George Box says “What you will often find is that there will be redundant factors… and don’t forget about those redundant factors. Knowing that something doesn’t matter is almost as important as knowing what does.” If you learn a factor isn’t having an affect you may be able to save money. And you can eliminate varying that factor in future experiments.

Continue reading

George Box 1919 to 2013 – A Great Friend, Scientist and Statistician

Reposted from my management blog.

I would most likely not exist if it were not for George Box. My father took a course from George while my father was a student at Princeton. George agreed to start the Statistics Department at the University of Wisconsin – Madison, and my father followed him to Madison, to be the first PhD student. Dad graduated, and the next year was a professor there, where he and George remained for the rest of their careers.

George died today, he was born in 1919. He recently completed An Accidental Statistician: The Life and Memories of George E. P. Box which is an excellent book that captures his great ability to tell stories. It is a wonderful read for anyone interested in statistics and management improvement or just great stories of an interesting life.

photo of George EP Box

George Box by Brent Nicastro.

George Box was a fantastic statistician. I am not the person to judge, but from what I have read one of the handful of most important applied statisticians of the last 100 years. His contributions are enormous. Several well know statistical methods are known by his name, including:

George was elected a member of the American Academy of Arts and Sciences in 1974 and a Fellow of the Royal Society in 1979. He also served as president of the American Statistics Association in 1978. George is also an honorary member of ASQ.

George was a very kind, caring and fun person. He was a gifted storyteller and writer. He had the ability to present ideas so they were easy to comprehend and appreciate. While his writing was great, seeing him in person added so much more. Growing up I was able to enjoy his stories often, at our house or his. The last time I was in Madison, my brother and I visited with him and again listened to his marvelous stories about Carl Pearson, Ronald Fisher and so much more. He was one those special people that made you very happy whenever you were near him.

George Box, Stuart Hunter and Bill Hunter (my father) wrote what has become a classic text for experimenters in scientific and business circles, Statistics for Experimenters. I am biased but I think this is acknowledged as one of (if not the) most important books on design of experiments.

George also wrote other classic books: Time series analysis: Forecasting and control (1979, with Gwilym Jenkins) and Bayesian inference in statistical analysis. (1973, with George C. Tiao).

George Box and Bill Hunter co-founded the Center for Quality and Productivity Improvement at the University of Wisconsin-Madison in 1984. The Center develops, advances and communicates quality improvement methods and ideas.

The Box Medal for Outstanding Contributions to Industrial Statistics recognizes development and the application of statistical methods in European business and industry in his honor.

All models are wrong but some are useful” is likely his most famous quote. More quotes By George Box

A few selected articles and reports by George Box

Related: It is not about proving a theorem it is about being curious about thingsBox on QualitySoren BisgaardLearning Design of Experiments with Paper HelicoptersPeter Scholtes

11 Year Old Using Design of Experiments

This reminds me of great times I had experimenting with my father when I was a kid. Though, to be honest, Sarah is much more impressive than I was.

Catapulting to Success with Design of Experiments

photo of Sarah and her trebuchet

Sarah Flexman with her trebuchet at the Storm the Castle science challenge in North Carolina.

At the end of 2010, Sarah had decided to take part in Storm the Castle, one of the events offered in the statewide science Olympiad competition. This particular challenge was to design, build and launch a model trebuchet, which is a medieval-style catapult for hurling heavy stones…

Here’s Sarah’s whole process: She built the trebuchet, tested it, used JMP for DOE during optimization, changed the hook angle and sling to improve performance, did more tests, entered this new data, reran the model, and made her final prediction graphs. The variables in her DOE were string length, counterweight and projectile weight, and she optimized for distance – that is, how far the projectile would go.

“Rather than doing 125 tests because we have three variables with five levels each, DOE found a way for us to perform only 26 tests and get approximately the same results. I typed in the results, ran the model and used the JMP Profiler. I understood how the variables predicted the outcome and found several patterns,” she explained.

“I hadn’t done any building like that. The whole day was fun. It was a very open learning environment. You were experimenting with things you had never done before. I would definitely do it again,” Sarah said. And she will – next year.

I have collected quite a few design of experiments resources, for those who are interested in learning more. Here is a nice webcast by brother: Combinatorial Testing – The Quadrant of Massive Efficiency Gains, discussing the incredibly efficiency designed combinatorial testing (very similar ideas to design of experiments) can provide.

Related: Learning Design of Experiments with Paper HelicoptersPlaying Dice and Children’s NumeracyStatistics Insights for Scientists and EngineersSarah (a different one), aged 3, Learns About SoapStatistics for ExperimentersMulti-factor designed experimentsCombinatorial Testing for SoftwareWhat Else Can Software Development and Testing Learn from Manufacturing? Don’t Forget Design of Experiments (DoE)Letting Children Learn

Designed Experiments

One-Factor-at-a-Time Versus Designed Experiments by Veronica Czitrom:

The advantages of designed experiments over [One Factor at a Time] OFAT experiments are illustrated using three real engineering OFAT experiments, and showing how in each case a designed experiment would have been better. This topic is important because many scientists and engineers continue to perform OFAT experiments.

I still remember, as a child, asking what my father was going to be teaching the company he was going to consult with for a few days. He said he was going to teach them about using designed factorial experiments. I said, but you explained that to me and I am just a kid? How can you be teaching adults that? Didn’t they learn it in school? The paper provides some examples showing why OFAT experimentation is not as effective as designed multi-factor experiments.

Related: Design of Experiments articlesStatistics for Experimenters (2nd Edition)Design of Experiments blog posts