Limitations of statistical design of experiments approaches in engineering testing

A hypothetical experiment and Monte Carlo simulations were used to examine the effectiveness of statistical design of experiments methods in identifying from the experimental data the correct terms in postulated regression models for a variety of experimental conditions. Two analysis of variance techniques (components of variance and pooled mean square error) combined with F-test statistics were investigated with first-order and second-order regression models. It was concluded that there are experimental conditions for which one or the other of the procedures results in model identification with high confidence, but there are also other conditions in which neither procedure is successful. The ability of the statistical approaches to identify the correct models varies so drastically, depending on experimental conditions, that it seems unlikely that arbitrarily choosing a method and applying it will lend to identification of the effects that are significant with a reasonable degree of confidence. It is concluded that before designing and conducting an experiment, one should use simulations of the proposed experiment with postulated truths in order to determine which statistical design of experiments approach, if any will identify the correct model from the experimental data with an acceptable degree of confidence. In addition, no significant change in the effectiveness of the methods in identifying the correct model was observed when systematic uncertainties of up to 10 percent in the independent variables and in the response were introduced into the simulations. An explanation is that the systematic errors in the simulation data caused a shift of the whole response surface up or down from the true value, without a significant change in shape. [S0098-2202(00)03102-3].