Posts about data
animals appropriate technology bacteria biology Career Cats cool curiouscat economy Energy Engineering engineering education engineers evolution experiment food fun Funding government green Health Care human health innovation K-12 kids Life Science medical research nature ocean physics Products Research Robots Science science education science explained science facts science webcasts scientific inquiry Students Technology university research USA water webcasts

I Just Finished Statistics for Experimenters and I Cannot Praise it Enough

Posted on April 15, 2020  Comments (1)

Guest post by Michael Betancourt.

I just finished Box, Hunter, and Hunter (Statistics for Experimenters) and I cannot praise it enough. There were multiple passages where I literally giggled. In fact I may have been a bit too enthusiastic about tagging quotes beyond “all models are wrong but some are useful” that I can’t share them all.

photo of Statistics for Experimenters with many blue bookmarks shown

I wish someone had shared this with me when I was first learning statistics instead of the usual statistics textbooks that treat model development as an irrelevant detail. So many of the elements that make this book are extremely relevant to statistics today. Some examples:

  • The perspective of learning from data only through the lens of the statistical model. The emphasis on sequential modeling, using previous fits to direct better models, and sequential experiments, using past fits to direct better targeted experiments.
  • The fixation on checking model assumptions, especially with interpretable visual diagnostics that capture not only residuals but also meaningful scales of deviation. Proto visual predictive checks as I use them today.
  • The distinction between empirical models and mechanistic models, and the treatment of empirical linear models as Taylor expansions of mechanistic models with covariates as _deviations_ around some nominal value. Those who have taken my course know how important I think this is.
  • The emphasis that every model, even mechanistic models, are approximations and should be treated as such.
  • The reframing of frequentist statistical tests as measures of signal to noise ratios.
  • The importance of process drift and autocorrelation in data when experimental configurations are not or cannot be arbitrarily randomized.
  • The diversity of examples and exercises using real data from real applications with detailed contexts, including units everywhere.

Really the only reason why I wouldn’t recommend this as an absolute must read is that the focus on linear models and use of frequentist methods does limit the relevance of the text to contemporary Bayesian applications a bit.

Texts like these make me even more frustrated by the desire to frame movements like data science as revolutions that give people the justification to ignore the accumulated knowledge of applied statisticians.

Academic statistics has no doubt largely withdrawn into theory with increasingly smaller overlap with applications, but there is so much relevant wisdom in older applied statistics texts like these that doesn’t need to be rediscovered just reframed in a contemporary context.

Oh, I forgot perhaps the best part! BHH continuously emphasizes the importance of working with domain experts in the design and through the entire analysis with lots of anecdotal examples demonstrating how powerful that collaboration can be.

I felt so much less alone every time they talked about experimental designs not being implemented properly andthe subtle effects that can have in the data, and serious effects in the resulting inferences, if not taken into account.

Michael Betancourt, PhD, Applied Statistician – long story short, I am a once and future physicist currently masquerading as a statistician in order to expose the secrets of inference that statisticians have long kept from scientists. More seriously, my research focuses on the development of robust statistical workflows, computational tools, and pedagogical resources that bridge statistical theory and practice and enable scientists to make the most out of their data.
Twitter: @betanalpha
Website: betanalpha
Patreon: Michael Betancourt

Related: Statistics for Experimenters, Second Edition – Statistics for Experimenters in Spanish – Statistics for Experimenters Review – Correlation is Not Causation

Ranking Countries by Scientific Publication Citations: USA, UK, Germany…

Posted on January 24, 2015  Comments (1)

The SCImago Journal and Country Rank provides journal and country scientific indicators developed from the information contained in the Scopus database. I posted about this previously (in 2014, 2011 and 2008).

The data in the post is based on their data from 1996 through 2013. The web site also lets you look at these ranking by very specific categories. For example biotechnology #1 USA, #2 Germany, #3 UK, #4 Japan, #12 China or human computer interaction #1 USA, #2 Germany, #3 UK #4 Japan, #13 China).

I like looking at data and country comparisons but in doing so it is wise to remember this is the results of a calculation that is interesting but hardly definative. We don’t have the ability to measure the true scientific research output by country.

The table shows the top 6 countries by h-index and then some others I chose to list.

Country h-index 2010
h-index 		2007
h-index 		% of World
Population		 % of World GDP total cites
USA 1,518 1,139 793     4.5%   22.2% 152,984,430
United Kingdom 918 689 465  0.9  3.5 37,450,384
Germany 815 607 408  1.1  5.0  30,644,118
France 742 554 376  0.9  3.8  21,193,343
Canada 725 536 370  0.5  2.4 18,826,873
Japan 635 527 372  1.8  7.8 23,633,462
Additional countries of interest (with 2013 country rank)
16) China 436 279 161  19.2  11.7  14,752,062
19) South Korea 375 258 161    .7  1.7  5,770,844
22) Brazil 342 239 148  2.8  3.0 4,164,813
23) India 341 227 146  17.5  2.6 5,666,045

Continue reading →

Country H-index Ranking for Science Publications

Posted on January 9, 2014  Comments (2)

The SCImago Journal and Country Rank provides journal and country scientific indicators developed from the information contained in the Scopus database (this site also lets you look at these ranking by very specific categories (I think 313 categories), for example biotechnology #1 USA, #2 Germany, #3 UK, #4 Japan, #9 China or Theoretical Computer Science #1 USA, #2 UK, #3 Canada, #6 China). I posted about this previously (in 2008 and 2011) and take a look at the updated picture in this post.

I like looking at data and country comparisons but in doing so it is wise to remember this is the results of a calculation that is interesting but hardly definative. We don’t have the ability to have exact numbers on haw the true scientific knowledge output by countries are. I think you can draw the conclusion that the USA is very influential, and along with other data make the case even that the USA is the leading scientific publication center.

The table shows the top 6 countries by h-index and then some others I chose to list.

Country h-index 2007
h-index 		% of World
Population		 % of World GDP total cites
USA 1,389 793     4.4%   22.4% 129,540,193
United Kingdom 851 465  0.9  3.4 31,393,290
Germany 740 408  1.2  4.7  25,848,738
France 681 376  0.9  3.6  5,795,531
Canada 658 370  0.5  2.5 15,696,168
Japan 635 372  1.8  8.2 20,343,377
Additional countries of interest
16) China 385 161  19.2  11.3  11,253,119
19) South Korea 343 161    .7  1.8  4,640,390
22) Brazil 305 148  2.8  3.1 3,362,480
24) India 301 146  17.6  2.5 4,528,302

Continue reading →

Medical Study Findings too Often Fail to Provide Us Useful Knowledge

Posted on July 24, 2013  Comments (1)

There are big problems with medical research, as we have posted about many times in the past. A very significant part of the problem is health care research is very hard. There are all sorts of interactions that make conclusive results much more difficult than other areas.

But failures in our practices also play a big role. Just poor statistical literacy is part of the problem (especially related to things like interactions, variability, correlation that isn’t evidence of causation…). Large incentives that encourage biased research results are a huge problem.

Lies, Damned Lies, and Medical Science

He discovered that the range of errors being committed was astonishing: from what questions researchers posed, to how they set up the studies, to which patients they recruited for the studies, to which measurements they took, to how they analyzed the data, to how they presented their results, to how particular studies came to be published in medical journals. The systemic failure to do adequate long term studies once we approve drugs, practices and devices are also a big problem.

This array suggested a bigger, underlying dysfunction, and Ioannidis thought he knew what it was. “The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

Another problem is that medical research often doesn’t get the normal scientific inquiry check of confirmation research by other scientists.

Most journal editors don’t even claim to protect against the problems that plague these studies. University and government research overseers rarely step in to directly enforce research quality, and when they do, the science community goes ballistic over the outside interference. The ultimate protection against research error and bias is supposed to come from the way scientists constantly retest each other’s results—except they don’t. Only the most prominent findings are likely to be put to the test, because there’s likely to be publication payoff in firming up the proof, or contradicting it.

Related: Statistical Errors in Medical Studies – Medical Study Integrity (or Lack Thereof) – Contradictory Medical Studies (2007) – Does Diet Soda Result in Weight Gain?

Introduction to Fractional Factorial Designed Experiments

Posted on June 10, 2013  Comments (6)

Scientific inquiry is aided by sensible application of statistical tools. I grew up around the best minds in applied statistics. My father was an eminent applied statistican, and George Box (the person in the video) was often around our house (or we were at his). Together they wrote Statistics for Experimenters (along with Stu Hunter, not related to me) the bible for design of experiments (George holds up the 1st edition in the video).

The video may be a bit confusing without at least a basic idea of factorial designed experiments. These introductory videos, by Stu Hunter, on Using Design of Experiments to Improve Results may help get you up to speed.

[the video has been removed from the internet]

This video looks at using fractional factorials to reduce the number of experiments needed when doing a multifactor experiment. I grew up understanding that the best way to experiment is by varying multiple factors at the same time. You learn much quicker than One Factor At a Time (OFAT), and you learn about interactions (which are mainly lost in OFAT). I am amazed to still hear scientists and engineers talk about OFAT as a sensible method or even as the required method, but I know many do think that way.

To capture the interactions a full factorial requires an ever larger number of experimental runs to be complete. Assessing 4 factors requires 16 runs, 6 would require 64 and 8 would require 256. This can be expensive and time consuming. Obviously one method is to reduce the number of factors to experiment with. That is done (by having those knowledgable about the process include only those factors worth the effort), but if you still have, for example, 8 very important factors using a fractional factorial design can be very helpful.

And as George Box says “What you will often find is that there will be redundant factors… and don’t forget about those redundant factors. Knowing that something doesn’t matter is almost as important as knowing what does.” If you learn a factor isn’t having an affect you may be able to save money. And you can eliminate varying that factor in future experiments.

Continue reading →

George Box 1919 to 2013 – A Great Friend, Scientist and Statistician

Posted on March 30, 2013  Comments (2)

Reposted from my management blog.

I would most likely not exist if it were not for George Box. My father took a course from George while my father was a student at Princeton. George agreed to start the Statistics Department at the University of Wisconsin – Madison, and my father followed him to Madison, to be the first PhD student. Dad graduated, and the next year was a professor there, where he and George remained for the rest of their careers.

George died today, he was born in 1919. He recently completed An Accidental Statistician: The Life and Memories of George E. P. Box which is an excellent book that captures his great ability to tell stories. It is a wonderful read for anyone interested in statistics and management improvement or just great stories of an interesting life.

photo of George EP Box

George Box by Brent Nicastro.

George Box was a fantastic statistician. I am not the person to judge, but from what I have read one of the handful of most important applied statisticians of the last 100 years. His contributions are enormous. Several well know statistical methods are known by his name, including:

George was elected a member of the American Academy of Arts and Sciences in 1974 and a Fellow of the Royal Society in 1979. He also served as president of the American Statistics Association in 1978. George is also an honorary member of ASQ.

George was a very kind, caring and fun person. He was a gifted storyteller and writer. He had the ability to present ideas so they were easy to comprehend and appreciate. While his writing was great, seeing him in person added so much more. Growing up I was able to enjoy his stories often, at our house or his. The last time I was in Madison, my brother and I visited with him and again listened to his marvelous stories about Carl Pearson, Ronald Fisher and so much more. He was one those special people that made you very happy whenever you were near him.

George Box, Stuart Hunter and Bill Hunter (my father) wrote what has become a classic text for experimenters in scientific and business circles, Statistics for Experimenters. I am biased but I think this is acknowledged as one of (if not the) most important books on design of experiments.

George also wrote other classic books: Time series analysis: Forecasting and control (1979, with Gwilym Jenkins) and Bayesian inference in statistical analysis. (1973, with George C. Tiao).

George Box and Bill Hunter co-founded the Center for Quality and Productivity Improvement at the University of Wisconsin-Madison in 1984. The Center develops, advances and communicates quality improvement methods and ideas.

The Box Medal for Outstanding Contributions to Industrial Statistics recognizes development and the application of statistical methods in European business and industry in his honor.

“All models are wrong but some are useful” is likely his most famous quote. More quotes By George Box

A few selected articles and reports by George Box

Related: It is not about proving a theorem it is about being curious about things – Box on Quality – Soren Bisgaard – Learning Design of Experiments with Paper Helicopters – Peter Scholtes

Chart of Wind Power Generation Capacity Globally 2005-2012

Posted on January 28, 2013  Comments (10)

Chart of installed wind energy capacity by country from 2005 to 2012

Chart of installed wind energy capacity by country from 2005 to 2012 by Curious Cat Science and Engineering Blog using data from the Wind Energy Association. 2012 data is for the capacity on June 30, 2012. Chart may be used with attribution as specified.

Wind power generation capacity continues to grow faster than the increase in electricity use. The rate of growth has slowed a bit overall, though China’s growth continues to be large.

From 2005-2012 globally wind power generation capacity increased 330%; lead by China with an increase of 5,250%. Of the leading countries Germany grew the least – just 63%. The percent of global capacity of the 8 countries listed in the chart (the 8 countries with the highest capacity in 2012) has been amazingly consistent given the huge growth: from a low of 79% in 2006 to a high of 82.4% in 2011 (2012 was 82%).

Global growth in wind energy capacity was 66% in 2008-2010. In 2010 to 2012 the increase was 28%. The second period is just 18 months (since the 2012 data is for the first half of the year). Extending the current (2010-2012) rate to the end of 2012 would yield an increase of 37%, which still shows there has been a slowdown compared to the 66% rate in the previous 2 year period. The decrease in government subsidies and incentives is responsible for the slowing of added capacity, though obviously the growth is still strong.

From 2005 to 2012 China’s share of global wind energy capacity increased from 2% to 27%, the USA 15% to 20%, Germany fell from 31% to 12%, India fell from 7.5% to 6.8% (while growing capacity 292%).

Hydro power is by far the largest source of green electricity generation (approximately 5 times the capacity of wind power – but hydro capacity is growing very slowly). And installed solar electricity generation capacity is about 1/5 of wind power capacity.

Related: Global Wind Energy Capacity Exceeds 2.5% of Global Electricity Needs (2010) – Wind Power Capacity Up 170% Worldwide from 2005-2009 – Wind Power Provided Over 1% of Global Electricity in 2007

Majority of Clinical Trials Don’t Provide Meaningful Evidence

Posted on May 3, 2012  Comments (2)

The largest comprehensive analysis of ClinicalTrials.gov finds that clinical trials are falling short of producing high-quality evidence needed to guide medical decision-making.

The analysis, published today in the Journal of the American Medical Association, found the majority of clinical trials is small, and there are significant differences among methodical approaches, including randomizing, blinding and the use of data monitoring committees.

This is a critical issue as medical studies continue to leave quite a bit to be desired. Even more importantly the failure to systemically study and share evidence of effectiveness once treatments are authorized leaves a great deal to be desired. On top of leaving quite a bit to be desired, the consequences are serious. If we make mistakes for example in how we date fossils it matters but it is unlikely to cause people their lives or health. Failure to adequately manage and analyze health care experiments may very well cost people their health or lives.

“Our analysis raises questions about the best methods for generating evidence, as well as the capacity of the clinical trials enterprise to supply sufficient amounts of high quality evidence to ensure confidence in guideline recommendations,” said Robert Califf, MD, first author of the paper, vice chancellor for clinical research at Duke University Medical Center, and director of the Duke Translational Medicine Institute.

The analysis was conducted by the Clinical Trials Transformation Initiative (CTTI), a public-private partnership founded by the Food and Drug Administration (FDA) and Duke. It extends the usability of the data in ClinicalTrials.gov for research by placing the data through September 27, 2010 into a database structured to facilitate aggregate analysis.

Related: Statistical Errors in Medical Studies – How to Deal with False Research Findings – Medical Study Integrity (or Lack Thereof)

Continue reading →

Citizen Science

Posted on April 1, 2012  Comments (1)

Citizen science enters a new era

Another online program, Phylo, is advancing scientists’ knowledge of genetics by making a game out of DNA matching. If areas of genetic sequence are roughly similar between species, it suggests strongly that they could have an important function. Finding them has been beyond the scope of computer algorithms. But earlier this month, researchers published a study where gamers outsmarted the best computers – they made the best possible DNA sequence match between up to eight species at a time.

The potential for regular people contributing to science is great. This has a long history. For most of human history science was done by non-scientists since there were no scientists. Calling is science might be a stretch but to me it was (passing on what health cures worked for various sicknesses, how to use various tools, how to grow crops…). As scientists came into being they were primarily unprofessional – that is they practiced science but were doing so as a hobby, they were not paid and had no requirements to get a PhD or anything.

Today regular people help by collecting data (counting birds, documenting plant growth [time of year], migration data, weather data…) sharing knowledge with scientists who ask, sharing their computer to be used to analyze data, analyzing data (for example, in astronomy hobbyists often make new discoveries) and the latest way people help is through games (that essentially tap human brainpower to analyze data – such as Foldit, which I have posted about previously).

I like the contributions people can make to science but I think the biggest value is the scientific understanding people gain while participating. As Neil Degrasse Tyson says the scientifically literate see a different world.

Cornell University provides an online tool to find opporunities participate in scientific research.

And we shouldn’t forget the amazing science done by students like those honored with Intel Talent Search, though the work those winning the awards do I would lump with science by “real scientists” (I believe now most of those who win are working on projects with university scientists).

Related: Backyard Scientists Aid Research – 8-10 Year Olds Research Published in Royal Society Journal – Teen diagnoses her own disease in science class

Numeracy: The Educational Gift That Keeps on Giving

Posted on February 19, 2012  Comments (5)

I like numbers. I always have. This is just luck, I think. I see, how helpful it is to have a good understanding of numbers. Failing to develop a facility with numbers results in many bad decisions, it seems to me.

A new article published in closed anti-science way, sadly (so no link), examines how people who are numerate (like literate but for number—understand) process information differently so that they ultimately make more informed decisions. Cancer risks. Investment alternatives. Calories. Numbers are everywhere in daily life, and they figure into all sorts of decisions.

People who are numerate are more comfortable thinking about numbers and are less influenced by other information, says Ellen Peters of Ohio State University (sadly Ohio State allows research by staff paid by them to be unavailable to the public – sad), the author of the new paper. For example, in one of Peters’s studies, students were asked to rate undergraduates who received what looked like different test scores. Numerate people were more likely to see a person who got 74% correct and a person who got 26% incorrect as equivalent, while people who were less numerate thought people were doing better if their score was given in terms of a percent correct.

People make decisions based on this sort of information all the time. For example, “A lot of people take medications,” Peters says. Every drug has benefits and potential risks, and those can be presented in different ways. “You can talk about the 10 percent of the population that gets the side effect or the 90 percent that does not.” How you talk about it will influence how dangerous the drug seems to be, particularly among people who are less numerate.

Other research has shown that only less numerate people respond differently to something that has a 1 in 100 chance of happening than something that has a 1 percent chance of happening. The less numerate see more risk in the 1 in 100 chance—even though these numbers are exactly the same.

“In general, people who are numerate are better able to bring consistent meaning to numbers and to make better decisions,” Peters says. “It suggests that courses in math and statistics may be the educational gift that keeps on giving.”

Related: full press release – Bigger Impact: 15 to 18 mpg or 50 to 100 mpg? – Data Doesn’t Lie, But People Can be Fooled – Understanding Data: Simpson’s Paradox – applied statistics is not about proving a theorem, it’s about being curious about things – Encouraging Curiosity in Kids – Dangers of Forgetting the Proxy Nature of Data – Compounding is the Most Powerful Force in the Universe

Ritalin Doesn’t Show Long Term Effectiveness for ADHD

Posted on February 2, 2012  Comments (1)

From the New York Times opinion piece, Ritalin Gone Wrong, by L. Alan Sroufe is a professor emeritus of psychology at the University of Minnesota’s Institute of Child Development:

Attention-deficit drugs increase concentration in the short term, which is why they work so well for college students cramming for exams. But when given to children over long periods of time, they neither improve school achievement nor reduce behavior problems. The drugs can also have serious side effects, including stunting growth.
…
To date, no study has found any long-term benefit of attention-deficit medication on academic performance, peer relationships or behavior problems, the very things we would most want to improve. Until recently, most studies of these drugs had not been properly randomized, and some of them had other methodological flaws.

But in 2009, findings were published from a well-controlled study that had been going on for more than a decade, and the results were very clear… At first this study suggested that medication, or medication plus therapy, produced the best results. However, after three years, these effects had faded, and by eight years there was no evidence that medication produced any academic or behavioral benefits.

As I have written before I am skeptical of the amount of drug use our health care system encourages: Lifestyle Drugs and Risk.

Related: Long Term ADHD Drug Benefits Questioned (2009) – Nearly 1 million Children Potentially Misdiagnosed with ADHD in the USA – Diet May Help ADHD Kids More Than Drugs – Over-reliance on Prescription Drugs to Aid Children’s Sleep? – Epidemic of Diagnoses