Вы находитесь на странице: 1из 15

See discussions, stats, and author profiles for this publication at: https://www.researchgate.


On the Use of Statistics in Design and the Implications for Deterministic

Computer Experiments

Article · February 1997

Source: NTRS

140 1,409

4 authors, including:

Timothy W. Simpson Patrick N Koch

Pennsylvania State University SAS Institute


Janet Allen
University of Oklahoma


Some of the authors of this publication are also working on these related projects:

The Goal-oriented Inverse Design (GoID) Method for Systems-based Design Under Uncertainty View project

Investigating the Effectiveness of Machine Learning Paradigms for Supporting Engineering Designers in Rapidly Evolving Digital Manufacturing View project

All content following this page was uploaded by Patrick N Koch on 05 June 2014.

The user has requested enhancement of the downloaded file.

Proceedings of DETC’97
1997 ASME Design Engineering Technical Conferences
September 14-17, 1997, Sacramento, California




Timothy W. Simpson, Jesse D. Peplinski, Patrick N. Koch, and Janet K. Allen

Systems Realization Laboratory
Woodruff School of Mechanical Engineering
Georgia Institute of Technology
Atlanta, Georgia 30332-0405, USA

ABSTRACT element codes can take from minutes to hours, if not longer.
Perhaps the most prevalent use of statistics in engineering What's more, this mode of query-and-response can often lead to
design is through Taguchi's parameter and robust design - using a trial and error approach to design, where a designer may never
orthogonal arrays to compute signal to noise ratios in a process uncover the functional relationship between x and y and
of design improvement. In our view, however, there is an therefore may never identify the best settings for input values.
equally exciting use of statistics in design that could become Statistical techniques are widely used in engineering design
just as prevalent: it is the concept of metamodeling, whereby to address these concerns. The basic approach is to construct
statistical models are built to approximate detailed computer approximations of the analysis codes that are much more effi-
analysis codes. Although computers continue to get faster, our cient to run and that yield insight into the functional
analysis codes always seem to keep pace, so that their computa- relationship between x and y. If the true nature of a computer
tional time remains non-trivial. Through metamodeling, analysis code is represented as
approximations of these codes are built that are orders of magni- y = f(x),
tude cheaper to run. These metamodels can then be linked to
optimization routines for fast analysis, or they can serve as a then a "model of the model" or metamodel (Kleijnen, 1987) of
bridge for integrating codes across different domains. the analysis code is taken to be
In this paper we first review metamodeling techniques that y = g(x), and so y = y + ε
encompass the Design of Experiments, Response Surface
Methodology, Taguchi methods, neural networks, inductive where ε represents both the error of approximation and
learning, and kriging. We discuss their existing applications in measurement or random errors. The most common meta-
engineering design and then address the dangers of applying modeling approach is to apply the Design of Experiments
traditional statistical techniques to approximate deterministic (DOE) to identify an efficient set of computer runs (x 1, x 2, ...,
computer analysis codes. We conclude with recommendations x n) and then employ regression analysis to create a polynomial
for the appropriate use of metamodeling techniques in given approximation of the computer analysis code. These approxi-
situations and how common pitfalls can be avoided. mations then can replace the existing code while providing:
KEYWORDS: Design of Experiments, Response • a better understanding of the relationship between x and y
Surface Methodology, Metamodeling, Kriging, Neural Net- (we typically lose this understanding when running computer
works, Rule Induction, D-Optimal, Taguchi Robust Design codes "blindly"),
• facilitated integration of domain dependent computer codes
(we no longer work with the codes themselves but rather
1 INTRODUCTION simple approximations of them), and
Much of today's engineering analysis work consists of run- • fast analysis tools for optimization and exploration of the
ning complex computer analysis codes -- supplying a vector of design space (we become more efficient because we are using
design variables (inputs) x and receiving a vector of responses the approximations rather than the computationally expensive
(outputs) y. Despite continual advances in computing power codes themselves).
and speed, the expense of running many analysis codes remains However, we have found that many published applications
non-trivial. Single evaluations of aerodynamic or finite- (including our own) utilizing these statistical methods for

1 Copyright © 1997 by ASME

computer-oriented design are statistically questionable, if not focus on experimental designs with particular emphasis on
incorrect. The root cause is that many of our analysis codes are (fractional) factorial designs, central composite designs, and
deterministic, which means that the error of approximation is orthogonal arrays. We also discuss measures of merit for the
not due to random effects. This calls into question most subse- evaluation of experimental designs. In Section 2.2, we discuss
quent statistical analyses of model significance. Consequently, model choice and model fitting, focusing on response surfaces,
our purpose in this paper is to highlight the potential statistical neural networks, inductive learning and kriging. We conclude
pitfalls in metamodeling and to provide some general recom- with an overview of two of the more prevalent metamodeling
mendations for the proper use of statistical techniques in design. techniques, namely, Response Surface Methodology and
In Section 2, we present a review of metamodeling techniques Taguchi's robust design.
that encompasses regression, neural networks, inductive learn-
ing, and the more advanced approach of kriging. We conclude
Section 2 with an introduction to the general statistical 2.1 Experimental Design
approaches of Response Surface Methodology (RSM) and Properly designed experiments are essential for effective
Taguchi's robust design. In Section 3, we describe the engi- computer utilization. The traditional approach in engineering is
neering design context for statistical applications, review to vary one parameter at a time within a computer analysis code
existing applications, methods and tools, and close by taking a and observe the effects, or to randomly assign different combi-
closer look at metamodeling techniques for deterministic com- nations of factor settings to be used as alternative analyses for
puter experiments. In Section 4 we present some general rec- comparison. Experimental design techniques developed for
ommendations for avoiding some important pitfalls in the effective physical experiments, however, are being applied for
application of metamodeling to computer-oriented design, and the design of engineering computer experiments to increase the
in Section 5 we conclude by discussing briefly some of the efficiency (computer time) and effectiveness of these analysis
more advanced issues that continue to make metamodeling an codes. In this section an overview of different types of experi-
active research area. ment designs is provided, along with measures of merit for
selecting or comparing different designs.

2 REVIEW OF METAMODELING TECHNIQUES 2 . 1 . 1 A Survey of Experimental Designs. A n

In its most basic sense, metamodeling involves (a) experimental design formally represents a sequence of experi-
choosing an experimental design for generating data, (b) ments to be performed, expressed in terms of factors (design
choosing a model to represent the data, and then (c) fitting the variables) set at specified levels (predefined values). An
model to the observed data. There are a variety of options for experimental design is represented mathematically by a matrix
each of these steps as shown in Figure 1, and we have X where the rows denote experimental runs and the columns
attempted to highlight a few of the more prevalent ones. For denote the particular factor setting for each factor for each run.
example, Response Surface Methodology usually employs Factorial Designs: The most basic experimental design is a
central composite designs, second order polynomials and least full factorial design. The number of design points dictated by a
squares regression analysis. full factorial design is the product of the number of levels for
SAMPLE each factor. The most common designs are the 2k (for evaluat-
EXPERIMENTAL MODEL MODEL METAMODELING ing main effects and interactions) and 3k designs (for evaluating
main and quadratic effects and interactions) for k factors at 2 and
(Fractional) Factorial Polynomial
(linear, quadratic, etc.)
Least Squares
Response Surface
3 levels, respectively. A 23 full factorial design is shown
Central Composite graphically in Figure 2(a).
Box-Behnken Splines Weighted Least The size of a full factorial experiment increases exponen-
(linear, cubic) Squares Regression
tially with the number of factors which may lead to an unman-
Best Linear ageable number of experiments, e.g., 10 factors each with 2
Frequency-Domain Unbiased Predictor Kriging
(BLUP) levels requires 210 or 1024 experiments. Fractional factorial
Orthogonal Array
Best Linear
designs can be used when experiments are costly and the
Plackett-Burman Kernel Smoothing Predictor (BLP) number of design points for a full factorial design is large. A
Radial Basis Functions Log-Likelihood
fractional factorial design consists of a fraction of a full factorial
Hybrid design; the most common fractional factorial designs are 2(k-p)
Latin Hypercube Network of Neurons Backpropagation Neural
Networks designs where a fraction is equal to 1/2(p). A half fraction of the
Select By Hand 23 full factorial design is shown in Figure 2(b).
Rulebase or Entropy Inductive
Random Selection Decision Tree (information-theoretic) Learning The reduction of the number of design points associated
with a fractional factorial design does not come without a price.
Design resolution and aliasing of effects are two critical (and
Figure 1. Techniques for Metamodeling related) issues which are important when selecting a fractional
factorial design. The 23 full factorial design shown in Figure
In the remainder of this section we provide a brief overview 2(a) allows estimation of all main effects (x1, x 2, x 3), all two
of several of the options listed in Figure 1. In Section 2.1, we factor interactions (x1x2, x 1x3 and x 2x3), as well as the three

2 Copyright © 1997 by ASME

factor interaction (x1x2x3). For the 23-1 fractional factorial indi- lead to an unmanageable number of design points, e.g., 7
cated by the solid dots in Figure 2(b), the main effects are alised factors at three levels requires 37 or 2187 experiments. The
(or biased) with the two factor interactions. Aliased effects can- most common second-order designs, configured to reduce the
not be estimated independently unless all but one of the aliased number of design points, are central composite and Box-
effects are known or assumed not to exist. The resolution of an Behnken designs.
experiment design is defined by the aliasing of effects: a design A central composite design (CCD) is a two level (2k or 2 (k-
of resolution R is one in which no p-factor effect is aliased with ) factorial design, augmented by n0 center points and two "star"
any other effect containing less than R-p factors (Box, et al., points positioned at ±α for each factor. This design, shown for
1978). The 23-1 design in Figure 2(b) can be at most a Resolu- three factors in Figure 2(c) consists of 2(k-p) + 2 k + n 0 total de-
tion III design, denoted23-1III. sign points to estimate 2k + k(k-1)/2 + 1 coefficients. Setting
Often 2k and 2(k-p) designs are used for identifying or screen- α=1 locates the star points on the centers of the faces of the
ing important factors when the number of factors is too large to cube (for three factors), giving a face-centered central composite
evaluate higher order effects. In these cases the sparsity of (CCF) design. It should be noted that for values of α other than
effects principle (Montgomery, 1997) is invoked, in which the 1, each factor is evaluated at five levels.
system is assumed to be dominated by the main effects and their Often it is desirable to employ the smallest number of fac-
low-order interactions. Based on this principle, two level frac- tors levels (e.g., three levels to fit a quadratic model) when
tional factorial designs can be used to "screen" the factors to creating an experimental design. One common, economical
determine which have the largest effect. The sparsity of effects class of such designs is the Box-Behnken designs. These
principle may not always hold true, however. As noted by designs are formed by combining 2k factorials with incomplete
Hunter (1970), every design provides aliased estimates (quadratic block designs (see Box and Draper, 1987). These designs do
and cubic effects, if present, will bias the estimates of the mean not contain any points at the vertices of the hypercube defined
and main effects respectively when a two level factorial design by the upper and lower limits for each factor. This is desirable
is used). "Any phenomenon omitted from a fitted model will if these extreme points are expensive or impossible to test.
confound [i.e., alias] certain estimated parameters in the model More information about central composite and Box-Behnken
regardless of the design used. Good fractional factorial designs design can be found in (Montgomery, 1997).
are carefully arranged so that estimates of effects thought to be Orthogonal Arrays: The experiment designs used by
important are confounded by effects thought to be unimpor- Taguchi, called orthogonal arrays, are most often simply
tant." fractional factorial designs in two or three levels (2(k-p) and 3(k-p)
One specific family of fractional factorial designs com- designs). These arrays are constructed to reduce the number of
monly used for screening are the two level Plackett-Burman (P- design points necessary to evaluate the required effects; the two-
B) designs. These resolution III designs are constructed to study level L4, L 12, and L16 arrays, for example, allow 3, 11, and 15
k=N-1 factors in N=4m design points. The class of P-B designs factors/effects to be evaluated in 4, 12, and 16 design points
in which N is a power of two are called geometric designs are respectively. In many cases, these designs are identical to those
are identical to the usual 2k-p fractional factorials. When N is given by Plackett and Burman (Lucas, 1994). The definition of
strictly a multiple of four, the P-B designs are referred to as orthogonality, for these arrays and other experiment designs, is
non-geometric designs and have very messy alias structures. given in Section 2.1.2. An overview of Taguchi's approach to
Their use in practical problems is often problematic particularly parameter design, within which the orthogonal arrays are
when the design is saturated, i.e., the number of factors is implemented, is given in Section 2.3.
exactly N-1. If all interactions are negligible, however, these
designs allow unbiased estimation of all main effects, using 2 . 1 . 2 Measures of Merit for Evaluating Experi-
only one more design point than the number of factors, with mental Designs. Selecting the appropriate design is
smallest possible variance (Box, et al., 1978). Minimum vari-
essential for effective experimentation. Experimenters must
ance and minimum size designs will be discussed in Section
balance the desire to gain as much information as possible
2.1.2. For a more complete discussion of factorial designs,
about the response-factor relationships with the cost of
confounding, and design resolution (see Myers and
experimentation and need for efficiency (measured in number of
Montgomery, 1995).
runs). Several measures of merit are available and useful for
X2 X2 evaluating and comparing experimental designs to ensure the
appropriate experiment is designed.
Orthogonality, Rotatability, Minimum Variance, and
X1 X1 X1
Minimum Bias: Four desirable characteristics of an
X3 X3
X3 experimental design, to facilitate efficient estimates of the
parameters, are orthogonality, rotatability, minimum variance,
(a) 23 Full Factorial (b) 23-1 Fractional Factorial (c) Composite Design and minimum bias. A design is orthogonal if for every pair of
Figure 2. Basic Three-Factor Designs factors xi and x j the sum of the crossproducts for the N design
points is zero. For a first order model, the estimates of all
Central Composite and Box-Behnken Designs: For estimat- coefficients will have minimum variance if the design is
ing quadratic effects the 3k or 3 k-p designs can be used but often configured so that

3 Copyright © 1997 by ASME

N D-optimal determinant value, normalized by the size of the
∑ x2iu = N matrix for comparing different size designs.
D-efficiency = (|X'X|design /|X'X|D-optimum)1/p (1)
the variance of predictions y will also have constant variance at
The D-efficiency statistic is the primary criteria with which
a fixed distance from the center of the design, and the design
designs have been compared. Other statistics for comparing
will be rotatable.
designs such as G-efficiency and A-optimality have also been
For second order modeling, Hunter (1985) suggests that
formulated (see Myers and Montgomery, 1995). Having
orthogonality loses its importance. "If the objective of the
completed our review of experimental design alternatives and
experimenter is to forecast a response at either present or future
measures of merit, we can now turn to the issues of model
settings of x, then an unbiased minimum variance estimate of
choice and model fitting.
the forecast y is required. In the late 1950's Box and his co-
workers demonstrated that rotatability....and the minimization
of bias from higher order terms....were the essential criteria for 2.2 Model Choice and Model Fitting
good forecasting." After selecting an appropriate experimental design and
Unsaturated/Saturated and Supersaturated Designs: In many performing the necessary computer runs, the next step involves
cases the primary concern in the design of an experiment is the choosing an approximating model and fitting method. Many
size (number of runs) of the resulting design. Most experiment alternative models and methods exist, but for practicality we
designs are unsaturated in that they contain at least two more review in this section the four that we have found to be most
design points than the number of factors. An experiment prevalent in the literature: response surfaces, neural networks,
design is said to be saturated if the number of design points is inductive learning and kriging.
equal to one more than the number of factor effects to be
estimated. Saturated fractional factorial designs allow unbiased
2 . 2 . 1 Response Surfaces. Given a response of
estimation of all main effects with smallest size and smallest
possible variance (Box, et al., 1978). In a supersaturated interest, y, and a vector of independent factors x thought to
design, then, the number of design points is less than or equal influence y, the relationship between y and x can be written as
to the number of factors. The most common examples of follows:
saturated designs are the Plackett-Burman two level designs y = f(x) + ε , (3)
discussed in the previous section, and the related orthogonal
where ε represents random error which is assumed to be
arrays of Taguchi. Additional aspects of supersaturated designs
normally distributed with mean zero and standard deviation σ.
can be found in (Draper and Lin, 1990a; Draper and Lin,
Since the true response surface function f(x) is usually
unknown, a response surface g(x) is created to approximate f(x).
For estimating second order effects, small composite
designs have be developed to reduce the number of design points Predicted values are then obtained using y = g(x).
necessary for a composite design. A small composite design The most widely used response surface approximating
becomes saturated if the number of design points is 2k + k(k- functions are simple low-order polynomials. If little curvature
1)/2 + 1 (the number of coefficients to be estimated for a full appears to exist, the first-order polynomial given in Equation 4
quadratic model). Myers and Montgomery (1995) note that can be employed. If significant curvature exists, the second-
recent work has suggested that these designs may not be good order polynomial in Equation 5, including all two-factor
choices in all applications; additional discussion on small interactions, can be used.
composite designs can be found in (Box and Draper, 1987; k
Lucas, 1974). y = b0 + ∑ bixi (4)
D-Optimal, D-Efficiency, and G-Efficiency: It is most
desirable to use unsaturated designs, when building predictive k k k k
models, unless running the necessary experiments is y = b0 + ∑ bixi + ∑ biix2i + ∑ ∑ bijxixj
prohibitively expensive. For comparing experiments based on i=1 i=1 i=1 j=1
number of design points and information obtained, many The parameters of the polynomials in Equations 4 and 5 are
measures of merit exist. A common measure is D-efficiency. usually determined using a least squares regression analysis to
A design is said to be D-Optimal if for the the model the fit these response surface approximations to existing data.
volume of the joint confidence region for the vector of These approximations are normally used for prediction within
unknown parameters is minimum. As a consequence, the Response Surface Methodology. A more complete discussion
global maximum of the determinant of X'X is achieved. Here of response surfaces and least squares fitting can be found in
X is the expanded design matrix, including N rows (one for (Myers and Montgomery, 1995). An overview of RSM is
each design point or run), and p columns (one for each given in Section 2.3.
coefficient to be estimated, including the overall mean). The D-
efficiency statistic for comparing designs, given in Equation 1,
compares the determinant value achieved for the design with the 2 . 2 . 2 Neural Networks. A neural network is composed
of neurons (single-unit perceptrons) which are multiple linear

4 Copyright © 1997 by ASME

regression models with a nonlinear (typically sigmoidal) inductive learning have been in process control and diagnostic
transformation on y. If the inputs to each neuron are denoted systems, and inductive learning approaches can be used to
x1, x 2, ..., x n, and the regression coefficients are denoted by the automate the knowledge-acquisition process of building expert
weights, wi, then the output, y, might be given by systems. Excellent examples can be found in (Evans and
Fisher, 1994; Langley and Simon, 1995; Leech, 1986).
y= 1
The training data collected are in the form of {(x 1,y 1),
1 + e -η/T (x 2,y 2), ..., (x n,y n)} where each x i represents a vector of
where η = Σwi xi + β. A neural network is then created by attribute values (such as processing parameters and
assembling the neurons into an architecture; the most prevalent environmental conditions), and each yi represents a
is the multi-layer feedforward architecture. corresponding observed output value. Although attributes and
There are two main issues in building a network: 1) outputs can be real-valued, the method is better suited to
specifying the architecture, and 2) training the network to discrete-valued data. Real values must often be transformed into
perform well with reference to a training set. "To a statistician, discrete representations (Evans and Fisher, 1994). Once the data
this is equivalent to (i) specifying a regression model, and (ii) set has been collected, training algorithms build a decision tree
estimating the parameters of the model given a set of data" by selecting the "best" divisive attribute and then recursively
(Cheng and Titterington, 1994). If the architecture is made calling the resulting data subsets. Although trees can be built
large enough, a neural network can be a nearly universal by selecting attributes randomly, a more efficient approach is to
approximator (Rumelhart, et al., 1994). select the attribute that minimizes the amount of information
"Training" a neural net refers to determining the proper needed for category membership. The mathematics of such an
values of all the weights (wi ) in the architecture, and is information-theoretic approach are given in (Evans and Fisher,
accomplished most commonly through backpropagation 1994).
(Rumelhart, et al., 1994). First, a set of p training data points Although decision trees seem best-suited for applications
is assembled {(x1,y 1}, (x2,y 2), ..., (x p,y p)}. For a network with with discrete input and output values, there are also applications
a single output y, the performance of the network is then with continuous variables that have met with greater success
defined as than standard statistical analysis. For example, Leech (1986)
E = ∑ (yp - yp )2 reports a process-control application where "Standard statistical
p analysis methods were employed with limited success. Some
of the data were non-numerical, the dependencies between
where yp is the output that results from the network given variables were not well understood, and it was necessary to
input x p and E is defined as the total error of the system. The simultaneously control several characteristics of the final
weights are then adjusted in proportion to product while working within system constraints. The results
∂E ∂y . of the statistical analysis, a set of correlations for each output
of interest, were difficult for people responsible for the day-to-
∂y ∂wij day operation to interpret and use."
Neural networks are best suited to approximate
deterministic functions in regression-type applications. Cheng 2 . 2 . 4 Kriging. Since many computer analysis coedes are
and Titterington (1994) note that "In most applications of deterministic and are therefore not subject to measurement error,
neural networks that generate regression-like output, there is no the usual measures of uncertainty derived from least-squares
explicit mention of randomness. Instead, the aim is function residuals have no obvious statistical meaning (Sacks, et al.,
approximation." Typical applications are speech recognition 1989b). Consequently, some statisticians (e.g., (Sacks, et al.,
and handwritten character recognition; very often the data is 1989a; Welch, et al., 1992)) have suggested modeling the
complex and of high dimensionality . Networks with tens of response as a combination of a polynomial model (usually
thousands of parameters are not unheard of, and the amount of linear) plus departure, i.e.,
training data is similar. Gathering the training data and k
determining the model parameters is a process that can be very y(x) = ∑ βjfj(x) + Z(x)
computationally expensive. j=1

where Z(x) is the systematic departure from the assumed model.

2 . 2 . 3 Rule Induction. Rule induction is one of five Z(•) represents the realization of a stochastic process and is
main paradigms of machine learning that also include neural assumed to have mean zero and covariance
networks, case-based learning, genetic algorithms, and analytic
learning (Langley and Simon, 1995). Of the five, inductive V(w,x) = σ2R(w,x)
learning is most akin to regression and metamodeling and is between Z(w) and Z(x) where σ2 is the process variance and
therefore our focus here. An inductive learning system induces R(w,x) is the correlation. The covariance structure of Z relates
rules from examples, so the fundamental modeling constructs to the smoothness of the approximating surface.
are condition-action rules. These rules in essence partition the One method of analysis for such models is kriging
data into discrete categories, and can be combined into decision (Matheron, 1963) which works as follows (Sacks, et al.,
trees for ease of interpretation. Many of the applications of 1989b). Given a design S = {s 1, s 2, ..., s n} and data ys = {y(s1),

5 Copyright © 1997 by ASME

..., y(s n)}', we are concerned with estimating the y(x) at an affect it." Finally, Biles (1984) defines RSM as the, "body of
untried x using the linear (or polynomial) predictor techniques by which one experimentally seeks an optimum set
of system conditions".
y(x) = c'(x)ys. RSM then encompasses and incorporates the design of
If we consider y(x) as random, y s becomes Y s = [Y(s 1), ..., experiments (Section 2.1), response surface model building
(Section 2.2.1), and 'model exploitation' for exploring a factor
Y(sn)]', and we can compute the mean squared error of the space and seeking optimum factor settings. The general RSM
predictor averaged over the random process. Thus, the best approach includes all or a subset of the following steps:
linear unbiased predictor (BLUP) is found by picking the n x 1 i) screening: when the number of factors is too large for a
vector c(x) to minimize comprehensive exploration and/or when experimentation is
MSE[y(x)] = E[c'(x)Ys - Y(x)]2 expensive, screening experiments are used to reduce the set
of factors to those that are most important to the
subject to the constraint for unbiasedness response(s) being investigated;
E[c'(x)Ys] = E[Y(x)] (6) ii) first-order experimentation: when the starting point is far
from an optimum (or in general when knowledge about the
R(w,x) must first be specified in order to compute Equation 6 space being investigated is sought), first order-models and
and should reflect the characteristics of the computer code. For an approach such as steepest-ascent are employed to
a smooth response, a covariance function with some derivatives "rapidly and economically move to the vicinity of the
might be preferred, whereas an irregular response might call for optimum" (Montgomery and Evans, 1975);
a function with no derivatives. Sacks, et al. (1989b) discuss iii) second-order experimentation: after the best solution using
several possible correlation functions which result in a linear first-order methods is obtained, a second-order model is fit
interpolation or cubic spline interpolation of the data (see in the region of the first-order solution to evaluate
(Mitchell, et al., 1988) for more details). Consequently, curvature effects and attempt to improve the solution.
kriging is extremely flexible due to the wide range of A more detailed description of RSM techniques and tools can be
correlation functions R(w,x) which may be chosen. Depending found in (Box and Draper, 1987), and a comprehensive review
on the choice of the correlation function, kriging can either of RSM developments and applications from 1966-1988 is
"honor the data", providing an exact interpolation of the data, or given in (Myers, et al., 1989).
"smooth the data", providing a nonexact interpolation of the
data (Cressie, 1988). More information about kriging and the
notion of modeling deterministic computer experiments as the 2 . 3 . 2 Taguchi's Robust Design. Genichi Taguchi
realization of a stochastic process can be found in (Cressie, developed an approach for industrial product design that is built
1989; Journel, 1986; Laslett, 1994; Matheron, 1963; Sacks, et on the foundation of statistically designed experiments.
al., 1989b). Cressie (1988) deals specifically with predicting Taguchi's robust design for quality engineering includes three
noiseless versions of Z, as is the case for deterministic steps: system design, parameter design, and tolerance design
computer experiments. As a final note, keep in mind that (Byrne and Taguchi, 1987). The key element of the approach is
kriging and splines (nonparametric regression models) are not the parameter design step, within which statistical
the same. In fact, in several comparative studies, kriging has experimentation is incorporated.
been seen to outperform splines and never performs worse than Rather than simply improving or optimizing a response
them (Laslett, 1994). value, the focus in parameter design is to identify the factor
settings that minimize variation in performance as well as
adjust the mean performance to a desired target. Factors
2.3 Strategies for Experimentation and included in experimentation include control factors and noise
Metamodeling factors. Control factors are those which can be set and held at
Two widely used methods incorporating experimental specific values, while noise factors are those which cannot be
design, model building, and prediction are Response Surface controlled, e.g., temperature on a shop floor. The evaluation of
Methodology and Taguchi's robust design or parameter design. mean performance and performance variation is accomplished
A brief overview of these two approaches is provided. through "crossing" two orthogonal arrays (Section 2.1.1).
Control factors are varied according to an inner array, or
"control" array, and for each run of the control array, the noise
2 . 3 . 1 Response Surface Methodology. Different
factors are varied according to an outer array, or "noise" array.
authors describe Response Surface Methodology differently. For each control factor experiment, a response value is obtained
Myers, et al. (1989) define RSM as "a collection of tools in for each noise factor design point, and the mean and variance of
design or data analysis that enhance the exploration of a region the response (measured across the noise design points) can be
of design variables in one or more responses." Box and Draper calculated. The performance characteristic used by Taguchi is a
(1987) state that, "Response surface methodology comprises a signal-to-noise (S/N) ratio defined in terms of the mean and
group of statistical techniques for empirical model building and variance of the response. Several alternate S/N ratios are
model exploitation. By careful design and analysis of available based on whether lower, higher, or nominal response
experiments, it seeks to relate a response, or output variable to
the levels of a number of predictors, or input variables, that

6 Copyright © 1997 by ASME

values are desired; specific S/N ratio formulations can be found terms, the design variables are factors, and the design objectives
in (Ross, 1988). are responses.) An intelligent search for these solutions most
The Taguchi approach does not explicitly include model commonly relies on some choice of optimization techniques,
building and optimization. Analysis of experimental results is which generate and evaluate many potential solutions in the
used to identify factor effects, plan additional experiments, and path towards improvement. Regardless of the optimization
to set factor values. Comprehensive presentation and technique employed a large number of function evaluations are
application of the Taguchi approach is given in (Phadke, 1989; required, thereby making fast analysis modules a near
Ross, 1988). Issues and techniques presented by Taguchi have imperative.
been used extensively in engineering design, and in many cases When are metamodels useful or appropriate? In the later
incorporated within the traditional RSM for efficient, effective, stages of design, when detailed information about a specific
and robust design, e.g., (Myers and Montgomery, 1995). These solution is available, highly accurate solutions are essential for
applications and associated implications for engineering design measuring system performance. In the early stages of design,
are discussed in the next section. however, the focus is on generating, evaluating, and comparing
potential conceptual configurations. In this stage, a conceptual
design problem is characterized by a large amount of
3 METAMODELING IN THE CONTEXT OF information, often uncertain, that must be managed. To ensure
ENGINEERING DESIGN the identification of a "good" system level conceptual or
How are the metamodeling techniques of the previous preliminary configuration, a comprehensive search is necessary.
section employed in engineering design? They can all be used In this case, the tradeoff between accuracy and efficiency
to produce approximations of existing computer analysis codes, becomes appropriate. The creation of metamodels allows fast
thereby producing fast analysis modules for more efficient analysis, which facilitates both comprehensive and efficient
computation. These metamodeling techniques can also yield design space search, at the expense of a (hopefully slight) loss
insight to the functional relationship between input and output of accuracy.
parameters. Having established this baseline engineering design
"context", in the next section we present a review of several
Given: Factors, responses statistical applications in engineering design, highlighting
general trends and observations. In Section 3.2 we discuss a
few of the more general statistical methods or "tools" which
Large # of YES
Run Screening have been developed for engineering applications, and we
Factors? Experiment conclude with Section 3.3 in which we take a closer look at the
some of the statistical pitfalls associated with the application of
NO Screening statistical techniques to deterministic computer experiments.
Run Modeling Reduce # This then paves the way for Section 4, where we present our
Experiment(s) Factors guidelines and recommendations for the appropriate use of
statistics in computer-oriented design.
Build Predictive
Model ( y )
Building 3.1 Existing Applications in Engineering
Noise YES Build Robustness In this section we provide a survey of several engineering
Factors? Model ( y , y ) applications of Design of Experiments, Response Surface
Methodology, and Taguchi's robust design method. Most of
our examples come from aerospace and mechanical engineering
design applications presented at conferences in recent years; for
Search Design brevity's sake, we summarize our findings in Table 1. Our
Model Space observations follow.
• In terms of experimental design, central composite designs
Solutions and D-Optimal designs seem to be the preferred choice among
(improved or robust) aerospace engineers while orthogonal arrays (OAs) are
Figure 3. A Principal Use of Statistics in preferred by many mechanical engineers; very few people
Computer-Oriented Design utilize grid searches and random point generation since they
Where would such models be useful? There is an implicit are less efficient.
process that is common to most of the engineering design • Optimization seems to be the principal driver for aerospace
applications, and this process is illustrated in Figure 3. A applications of DOE and RSM; these types of design
designer's ultimate goal is most often to arrive at improved or applications typically involve the use of computer intensive
robust solutions, which are in essence the values of design analysis and optimization routines, and RSM is a logical
variables that best meet the design objectives. (In statistical choice for increased efficiency.

7 Copyright © 1997 by ASME

• Mechanical engineers are more inclined to use OAs and
Taguchi for robust design instead, working with the signal-
to-noise ratio for parameter and tolerance design. 3.2 Existing Methods and Tools in
• Very few designers actually model Taguchi's loss function Engineering Design
directly (e.g., Beard and Sutherland, 1993); many prefer to In this section we present a few methods and tools
model the response instead. developed specifically for engineering design that are build
• Most applications employ second order response surface around or incorporate the statistical techniques and approaches
models; there are only a few instances were higher order reviewed in Section 2. As evident in Table 1 in the previous
models (e.g., Venter, et al., 1996) and mixed polynomial section, the Taguchi approach and RSM have been widely
models (e.g., Roux, et al., 1996) are employed. applied in engineering design; a review of literature comparing
• When orthogonal arrays are used, special care must be taken these approaches is given first, followed by an overview of
to avoid confounding main effects with interactions, Robust Design Simulation (RDS), Robust Concept
providing the interactions are not assumed to be insignificant Exploration Method (RCEM), NORMAN/DEBORA,
based on prior observations or experience. Some authors take DOE/Opt, and HIDER.
this into account while others do not.
• Most applications utilize a least squares regression analysis 3 . 2 . 1 Taguchi Approach vs. RSM. Both Taguchi
when fitting a model; only a few use stepwise regression, and techniques and Response Surface Methodology have been
this is usually because the model is not second order. applied extensively in engineering design. Many authors have
compared these techniques for different problems. It appears to
Table 1. Cited Engineering Applications of be commonly accepted that the principles associated with
DOE, RSM, and Taguchi Taguchi approach are not only useful but very appropriate for
Surface Method
industrial product design; Ramberg, et al. (1991) for example
suggest that "the loss function and the associated robust design
Consider Interactions

philosophy provide fresh insight into the process of optimizing

S/N, Loss, or Resp.
Fractional Factorial

Fit Using:

or improving the simulation's performance". However, two

Robust Design
Order of Eqn.

specific aspects of the Taguchi approach are commonly

Least Sqr.


criticized: the experimental design choice (orthogonal arrays,



inner and outer) and the use of the loss function or signal-to-
(Balabanov, et al., 1996) x x 2nd x x noise ratio. It has been argued and demonstrated that the use of
(Bauer and Krebs, 1996) x x x 2nd x x S/N
(Beard and Sutherland, 1993) x x x L
a single experiment combining control and noise factors, a
(Chen, et al., 1996) x x 2nd x R
more traditional RSM approach rather than inner "control" and
(Chi and Bloebaum, 1995) x x R outer "noise" arrays, is more efficient in that it requires fewer
(Engelund, et al., 1993) x x 2nd x experiments (Shoemaker, et al., 1991; Unal, et al., 1994a;
(Gadallah and ElMaraghy, 1994) x x x x x S/N Welch, et al., 1990). With respect to the loss function or
(Giunta, et al., 1996) x x 2nd x signal-to-noise ratio, the drawbacks of combining response
(Giunta, et al., 1994) x x x x 2nd x
(Healy, et al., 1975) x x 2nd x x
mean and variance into a single measure, a signal-to-noise ratio,
(Hong, et al., 1994) x x 2nd x x S/N
are well-documented in the literature and many authors advocate
(Koch, et al., 1996) x x 2nd x measuring a response directly rather than loss, and separately
(Korngold and Gabriele, 1995) x 2nd x tracking mean and variance (Chen, et al., 1995; Ramberg, et
(Li, et al., 1994) x x x S/N al., 1991; Welch, et al., 1990). Shoemaker et al. (1991),
(Mavris, et al., 1996) x x 2nd x R however, warn that "A potential drawback of the response-
(Mavris, et al., 1995) x x 2nd x R
(Roux, et al., 1996) x x Mix x
model approach that is depends more critically than the loss-
(Rowell, et al., 1996) x x x 2nd x x x model approach on how well the model fits."
(Stanley, et al., 1992) x x x R Given the general acceptance of the Taguchi principles and
(Sundaresan, et al., 1993) x x x R of robust design, with these criticisms of these techniques,
(Unal, et al., 1996) x x x many advocate a combined Taguchi-RSM approach, or simply
(Unal, et al., 1992) x x x R applying more traditional RSM techniques within the
(Unal, et al., 1994a) x x 2nd x
framework defined by Taguchi (Lucas, 1994; Myers, et al.,
(Unal, et al., 1994b) x x x x x R
(Venter, et al., 1996) x x 4th x 1989; Myers and Montgomery, 1995; Ramberg, et al., 1991).
(Yu and Ishii, 1993) x x 2nd x x R It is the belief of the authors of this paper that both orthogonal,
inner and outer arrays, and single, composite experiments each
have advantages and disadvantages and appropriate uses, and that
In addition to these specific engineering design applications
separate observation of mean and variance leads to useful
of DOE, RSM, and Taguchi, many methods and "tools" have
problem insight. Regardless, the core principles of both
been developed for more general design application. We
Taguchi and RSM have provided a foundation for the specific
examine several of these methods in the next section,
design methods discussed in Section 3.2.2.
highlighting various applications of each.

8 Copyright © 1997 by ASME

algorithms. HIDER and AIMS have been applied to the design
3 . 2 . 2 An Overview of Existing Methods. Robust of a cutting process (Lu and Tcheng, 1991), a diesel engine
(Yerramareddy, et al., 1992), and a wheel loader (Reddy, 1996).
Design Simulation (RDS), combines the design of experiments
The Concurrent SubSpace Optimization (CSSO)
and response surface metamodeling aspects of RSM with Monte
methodology developed in (Renaud and Gabriele, 1993; Renaud
Carlo simulation to facilitate robust design (Mavris, et al.,
and Gabriele, 1994; Renaud and Gabrielle, 1991) utilizes design
1995). The general implementation of this approach consists
data generated during concurrent subspace optimizations to
of three sequential experimentation stages: screening, second-
develop response surface approximations of the design space.
order experimentation for model building, and a combined
Optimization of these response surface representations of the
experimentation/Monte Carlo simulation stage for
design space form the basis of the coordination procedure
incorporating noise. The objective of RDS is to achieve robust
developed in (Renaud and Gabriele, 1993; Renaud and Gabriele,
system designs: those that minimize performance variation due
1994; Renaud and Gabrielle, 1991). The data generated by the
to uncontrollable noise, while achieving as closely as possible
subspace optimizers is not uniformly centered about the current
performance target values. RDS is presented and applied to the
design as in CCD or other sampling strategies, but instead
design of a High Speed Civil Transport aircraft in (Mavris, et
follows the descent path of the subspace optimizers. In
al., 1996; Mavris, et al., 1995).
(Renaud and Gabriele, 1993; Renaud and Gabriele, 1994;
The Robust Concept Exploration Method (RCEM) has
Renaud and Gabrielle, 1991) interpolating polynomial response
been developed to facilitate quick evaluation of different design
surfaces are constructed which have either a first order or second
alternatives and generation of top-level design specifications
order basis for use in the CSSO coordination procedure. In
with quality considerations in the early stages of design (Chen,
(Wujek, et al., 1996; Wujek and Renaud, 1997) a modified
et al., 1996a). Foundational to the RCEM is the integration of
decomposition strategy is employed in developing quadratic
robust design principles, design of experiments and response
response surfaces for use in the CSSO coordination procedure.
surface methodology, and the compromise Decision Support
Finally, in (Sellar and Batill, 1996; Sellar, et al., 1996a; Sellar,
Problem (DSP) (a multiobjective decision model which is a
et al., 1996b) artificial neural network response surfaces are
hybrid formulation based on Mathematical Programming and
developed for use in the CSSO coordination procedure.
Goal Programming). This RCEM is applied to a High Speed
Additional approaches incorporating statistical techniques
Civil Transport (HSCT) in (Chen, et al., 1996a) The authors of
in design exist; only a few have been included in this section.
this paper have also applied RCEM to the design of a General
Our focus is not on the methods, but on the appropriateness of
Aviation Aircraft (Simpson, et al., 1996), to a manufacturing
the statistical techniques: many of the examples to which these
simulation problem (Peplinski, et al., 1996), and to the design
methods have been applied employ deterministic computer
of a turbine lift engine (Koch, et al., 1996).
experiments, to which the application of statistical techniques
NORMAN/DEBORA is a TCAD (Technology Computer
is questionable. The associated issues are discussed in the next
Aided Design) system created by a Belgian company, IMEC,
incorporating advanced sequential design of experiments (DOE)
and Response Surface Methodology (RSM) techniques to aid in
engineering optimization and robust design problems. 3.3 A Closer Look at Experimental Design for
Advanced DOE/RSM techniques incorporated within this Deterministic Computer Experiments
system include a novel design of experiments concept, Target Since engineering design commonly involves exercising
Oriented Design (TOD), a unique parameter transformation deterministic computer codes, the use of statistical
technique, RATIOFIND, and a non-linearly constrained experimentation in creating metamodels of these codes warrants
optimizer, DEBORA (Booth, et al., 1991). The a closer look. Given a response of interest, y, and a vector of
NORMAN/DEBORA system is demonstrated for semiconductor independent factors x thought to influence y, the relationship
integrated circuit (IC) technology design and optimization in between y and x (Equation 3) includes the random error term ε .
(Cartuyvels and Dupas, 1993). To apply least squares regression, the error values for each data
DOE/Opt is a prototype computer system for performing point are assumed to have identical and independent normal
design of experiments, response surface modeling, and distributions with means of zero and standard deviations of σ, or
optimization (Boning and Mozumder, 1994). Applications ε i i.i.d. N(0, σ2). This scenario is shown in Figure 4(a). The
described for DOE/Opt are in the semiconductor process and least squares estimator then minimizes the sum of the squared
device design area and include process/device optimization, differences between the actual data points and the values
simulator tuning, process control recipe generation, and design predicted by the model. It is acceptable if no data point actually
for manufacturability. lies on the predicted model, because it is assumed that the
Hierarchical and Interactive Decision Refinement (HIDER) model "smoothes out" the random error. Of course, it is likely
is a methodology for concept exploration in the early stages of that the regression model itself is only an approximation of the
design that integrates simulation, optimization and machine true behavior between x and y, so that the final relationship is
learning to support design decision making (Reddy, 1996).
HIDER uses the Adaptive Interactive Modeling System (AIMS) y = g(x) + ε bias + ε random (7)
(Lu and Tcheng, 1991) to decompose the design space using where ε bias represents the error of approximation. However, for
distance-based, population-based, and hyperplane-based deterministic computer experiments as illustrated in Figure

9 Copyright © 1997 by ASME

4(b), ε random has mean zero and variance zero, so after model Unal, et al., 1994b; Venter, et al., 1996; Welch, et al., 1990).
fitting we have the relationship These measures have no statistical meaning since they assume
the observations include an error term which has mean of zero
y = g(x) + ε bias (8)
and a non-zero standard deviation. Consequently, the use of
The deterministic case of Equation 7 conflicts sharply with stepwise regression for polynomial model fitting is not
the methods of least squares regression. First, unless ε bias is appropriate since it utilizes F-statistic values when
i.i.d. N(0,σ2) then the assumptions for statistical inference from adding/removing model parameters.
least squares regression are violated. Even further, because there R-Squared (when defined as the model sum of squares
is no random error there is little justification for smoothing divided by the total sum of squares) is really the only measure
across data points; instead the model should hit each point for verifying model adequacy for deterministic computer
exactly and interpolate between them as shown in Figure 4(b). experiments, and often this measure not sufficient (a high R-
Finally, most standard tests for model and parameter Squared value can be deceiving). Consequently, confirmation
significance are based on computations using ε random (the mean testing of model validity through use of additional (different)
squared error) and are therefore impossible to compute. These data points becomes essential. Residual plots may also be
observations are supported by literature in the statistics extremely helpful when verifying model adequacy for
community; as Sacks, et al. (1989b) carefully point out, identifying trends in data, examining outliers, etc.
because deterministic computer experiments lack random error: Some researchers (e.g., Giunta, et al., 1996; Giunta, et al.,
• response surface model adequacy is determined solely by 1994; Venter, et al., 1996) have also employed metamodeling
systematic bias, techniques such as RSM for modeling deterministic computer
• the usual measures of uncertainty derived from least-squares experiments which contain numerical noise. This numerical
residuals have no obvious statistical meaning (deterministic noise is used as a surrogate for random error, thus allowing the
measures of uncertainty exist, e.g., max |y(x) - y(x)| over x, standard least-squares approach to be applied. However, the
but they may be very difficult to compute), and assumption of equating numerical noise to random error is
• the classical notions of experimental blocking, replication questionable, and the appropriateness of their approach warrants
and randomization are irrelevant. further investigation.
Similarly, according to Welch and his co-authors (1990), Given the potential problems of applying least-squares
current methods for the design and analysis of physical regression to deterministic applications, the trade-off then
experiments (e.g., Box and Draper, 1987; Box, et al., 1978) are becomes one of appropriateness vs. practicality. If a response
not ideal for complex, deterministic computer models. "In the surface is created to model data from a deterministic computer
presence of systematic error rather than random error, statistical code using experimental design and least-squares fitting, and if
testing is inappropriate" (Welch, et al., 1990). Finally, a it provides very good agreement between predicted and actual
discussion of how the model should interpolate the observations values (a situation that often occurs when the computer model
can be found in (Sacks, et al., 1989a). is "well-behaved"), then there is no practical reason to discard it.
It should be used, albeit with caution. However, it is important
y y ε=0 to be aware of the fundamental assumptions of the statistical
techniques employed, so that we avoid incorrect or misleading
statements about model significance. In the next section we
ε ~ N(0,σ2 ) provide several guidelines pertaining to the appropriate use of
(yi - yi ) statistical methods and metamodeling of deterministic computer

L.S.E.=∑ (y i - y i)
2 ∑ (y i - y i) =0
How can a design engineer efficiently apply the
y = g(x) = second order
metamodeling tools of Section 2 while avoiding the pitfalls
least squares fit y = g(x) = spline fit described in Section 3.3? There are two ways to answer this
question: from the bottom up (tools applications) and from
x x
the top down (motives tools). The bottom-up approach is
(a) Non-Deterministic Case (b) Deterministic Case
presented in Section 4.1, the top-down in Section 4.2.
Figure 4. Deterministic and Non-Deterministic
4 . 1 Evaluation of Metamodeling Techniques
Curve Fitting
There are two components to this section. The first is an
evaluation of the four metamodeling techniques described in
So where can these methods go wrong? Unfortunately it is
Section 2.2. The second component is choosing an
very easy to unthinkingly classify the ε bias term from a
experimental design which has more direct applicability to
deterministic model fit as ε random and then proceed with standard
response surface methods but also applies to the remaining
statistical testing. Several authors have reported statistical
metamodeling techniques. The explorations of which designs
measures such as the F-statistics and root MSE for verification
are most appropriate are, as yet, open research areas.
of model adequacy (e.g., Healy, et al., 1975; Koch, et al., 1996;

10 Copyright © 1997 by ASME

Jensen and Myers, 1989). For comparisons of different design
4 . 1 . 1 Evaluation of Model Choice and Model types, refer to the following.
• Lucas (1976) compares CCD, Box-Behnken, uniform shell,
Fitting Alternatives. In Section 2.2 we described four
Hoke, Pesotchinsky, and Box-Draper designs, using the D-
metamodeling techniques; here we present some brief guidelines efficiency and G-efficiency statistics.
for their evaluation. • Myers and Montgomery (1995) provide a comprehensive
Response Surfaces: primarily intended for applications review of many experimental designs for fitting second order
with random error, though deterministic applications are not response surfaces. They conclude that hybrid designs are a
necessarily incorrect. Building higher-order models with greater good idea, if the unusual levels for the design variables can be
than ten factors becomes difficult. (Building regression models tolerated. For computer experiments, this is unlikely to be a
is tractable for fifteen to twenty factors, but actually obtaining problem.
the data is the limiting factor.) It is the most well-established • Montgomery and Evans (1975) compare six second-order
metamodeling technique, and is probably the easiest to use. designs: a) 3^2 factorial, b) rotatable orthogonal CCD, c)
Neural Networks: nonlinear regression approach best rotatable uniform precision CCD, d) rotatable minimum bias
suited for deterministic (non-noisy) applications. They can be a CCD, e) rotatable orthogonal hexagon, and f) rotatable
nearly universal approximator, and so are able to handle highly uniform precision hexagon. The criteria used for comparison
nonlinear or extremely large (~ 10,000 parameters) problems, as are average response achievement and distance from true
long as enough data points are provided and enough computer optimum.
time is allocated to fit the model. Neural networks therefore • Lucas (1974) compares symmetric and asymmetric composite
lend themselves to large problems, so that "Now the procedure and smallest composite designs for different numbers of
is to toss the data directly into the NN software, use tens of factors using the |X'X| criterion, and the D-efficiency and G-
thousands of parameters in the fit, let the workstation run 2-3 efficiency statistics.
weeks grinding away doing the gradient descent, and voilá, out • Giovannitti-Jensen and Myers (1989) discuss several first and
comes the result" (Cheng and Titterington, 1994). second order designs, observing that the performance of
Inductive Learning: modeling technique most appropriate rotatable CCD and Box-Behnken designs are nearly identical.
where the input and output factors are primarily discrete-valued They note that "hybrid designs appear to be very promising."
or can be grouped into categories. The predictive model, in the
form of condition/action rules or a decision tree, may perhaps
lack the mathematical insight desired for engineering design 4.2 Initial Recommendations for Metamodeling
applications, but it is ideal for applications such as process Uses
control and diagnosis. The majority of metamodeling applications are built around
Kriging: interpolation method created to handle the creation of low-order polynomial metamodels using central
deterministic data. It is not based on the same assumptions as composite designs and least squares regression. The nearly
standard regression analysis and appears to be better suited for universal popularity of this approach is due, at least in part, to
deterministic computer experiments, as long as the number of the maturity and well-established nature of response surface
factors is small (≤10, according to (Welch, et al., 1992)). methodology, the ease and simplicity of the method, and easily
Kriging is extremely flexible due to the wide scope of accessible software support tools. However, the RSM approach
correlation functions R(w,x) which may be chosen (Laslett, starts to break down when there are a large (> 10) number of
1994); it can provide an exact interpolation of the data or factors to be modeled, or when the relationship to be modeled is
"smooth the data" (Cressie, 1988). Additional advantages highly nonlinear. And as we have shown in Section 3.3, there
include (Welch, et al., 1992): are also dangers of applying the RSM approach blindly to
• it provides a basis, via the likelihood, for a stepwise deterministic applications. The alternative approaches to
algorithm to determine the important factors; metamodeling described in section 4.1.1 address these
• it is flexible, allowing nonlinear and interactions effects to limitations, each in their own way. Our recommendations are
emerge without explicitly modeling such effects; summarized as follows:
• the same data can be used for screening and building the • If a large number of factors must be modeled in a
predictor, so expensive runs are efficiently used. deterministic application, then neural networks may be the
The complexity of the method, however, coupled with a lack of best metamodeling choice despite their tendency to be
computer software support, may make the learning curve of this computationally expensive to create.
technique prohibitive in the near term. • If the underlying function to be modeled is deterministic
4.1.2 Evaluation of Experimental Designs. and highly nonlinear in a small (≤ 10) number of factors,
There are many voices in the discussion of the relative merits of then kriging may be the best choice despite its statistical
different experimental designs, and it is therefore unlikely that complexity.
we have captured them all. However, broad trends appear, and • However, in deterministic applications that have a small
in general we have found that central composite designs number of factors and are fairly well behaved, we present
consistently perform and compare favorably. Other second order another option for exploration: applying the standard RSM
designs worth investigating are the hexagonal (Montgomery and approach, augmented with a Taguchi outer array.
Evans, 1975) and Box-Behnken and hybrid designs (Giovannitti-

11 Copyright © 1997 by ASME

R S M / O A approach: The fundamental problem with space. We hope to have preliminary results in this area in the
applying least-squares regression to deterministic applications is near future.
the absence of ε random in Equation 8. However, if some of the
input parameters for the computer code can be classified as
noise factors, and if these noise factors are varied across an outer ACKNOWLEDGMENTS
array for each setting of the control factors, then in essence a Timothy Simpson and Jesse Peplinski have been supported
series of replications are generated that can be used to by NSF Graduate Research Fellowships, and Patrick Koch has
approximate ε random . This approximation can be justified if it is been supported by a Gwaltney manufacturing fellowship. In
reasonable to assume that, were the experiments performed on addition, financial support from NSF Grants DMI-96-12365 and
an actual physical system, that the random error observed would DMI-96-12327, and NASA Grant NGT-51102 is gratefully
have been due to fluctuations in the same noise factors that are acknowledged.
varied in the computer code. If this assumption is reasonable,
then statistical testing of model and parameter significance can
be performed, and models of both mean response and variability REFERENCES
Balabanov, V., Kaufman, M., Knill, D. L., Haim, D., Golovidov,
can be created from the same experimental design. We are O. and Giunta, A. A., 1996, "Dependence of Optimal Structural
currently investigating this approach in our ongoing work. Weight on Aerodynamic Shape for a High Speed Civil
Transport," 6th AIAA/USAF/NASA/ISSMO Symposium on
Multidisciplinary Analysis and Optimization, Bellevue, WA,
5 SUMMARY AND CONCLUSIONS AIAA, Inc., Vol. 1, pp. 599-612.
In this paper we have surveyed some applications of Bauer, D. and Krebs, R., 1996, "Optimization of Deep Drawing
statistics to engineering design and have developed the idea of Processes Using Statistical Design of Experiments," 22nd
metamodeling in the engineering design context, as discussed in Design Automation Conference (McCarthy, J. M., ed.), Irvine,
Section 1 and as shown in Figure 3. Metamodeling is an active CA, ASME, Paper No. 96-DETC/DAC-1446.
research area in more than just the statistics community, to Beard, J. E. and Sutherland, J. W., 1993, "Robust Suspension
which our reference list is mute testimonial. However, System Design," Advances in Design Automation (Gilmore, B.
applying these techniques to deterministic applications in J., Hoeltzel, D. A., et al., eds.), Albuquerque, NM, ASME, Vol.
65-1, pp. 387-395.
engineering design can lead to problems, as pointed out in Biles, W. E., 1984, "Design of Simulation Experiments," 1984
Section 3.1 and discussed in Section 3.3. We present the set of Winter Simulation Conference (WSC) (Sheppard, S., Pooch, U.,
our recommendations for applying metamodeling techniques in et al., eds.), Dallas, TX, IEEE, pp. 99-104.
Section 4, but this set is by no means complete. Boning, D. S. and Mozumder, P. K., 1994, "DOE/Opt: A System for
Comprehensive comparisons of these techniques have yet to be Design of Experiments, Response Surface Modeling, and
performed, and the difficulties of large problem size and Optiminzation Using Process and Device Simulation," IEEE
nonlinearity are ever-present. Trans. on Semiconductor Manufacturing, Vol. 7(2), pp. 233-244.
In particular, an issue of interest to us in our continuing Booth, R., Dupas, L., Cartuyvels, R. and De Meyer, K., 1991, "New
research is the problem of size, or scaling. As the size of a Technique to Improve Response Surface Method Model Fitting
problem (number of factors) increases, the cost associated with Accuracy for Process Technology and Device Design
Optimization," INternational Workshop on VLSI Process and
creating metamodels can begin to outweigh the associated Device Modeling (VPAD), Oiso Kanagawa, Japan.
gains. In addition, often the reduction in factors resulting from Box, G. E. P. and Draper, N. R., 1987, Empirical Model-Building
screening is still not sufficient to bring the problem to a and Response Surfaces, John Wiley & Sons, New York.
manageable size. This problem is compounded by the multiple Box, G. E. P., Hunter, W. G. and Hunter, J. S., 1978, Statistics for
response problem: complex engineering design problems Experimenters, John Wiley & Sons, Inc., New York.
invariably include multiple measure of performance, or Byrne, D. M. and Taguchi, S., 1987, "The Taguchi Approach to
responses, that must be modeled. The screening process also Parameter Design," 40th Annual Quality Congress Transactions,
breaks down when attempting to select the most important Milwaukee, Wisconsin, American Society for Quality Control.
factors for more than one response, as each response often Cartuyvels, R. and Dupas, L., 1993, "NORMAN/DEBORA: a
requires different important factors. The general question that Powerful CAD-Integrating Automatic Sequencing System
Including Advanced DOE/RSM Techniques for Various
arises from these problems, then, is how can these Engineering Optimization Problems," JSPE-IFIP WG 5.3
experimentation and metamodeling techniques be used DIISM'93 Workshop (Design of Information Infrastructure
efficiently for larger problems (problems with greater than 10 Systems for Manufacturing), Tokyo, Japan.
factors after screening)? One approach that we are pursuing is Chen, W., Allen, J. K., Mavris, D. and Mistree, F., 1996a, "A
problem partitioning or decomposition. A significant literature Concept Exploration Method for Determining Robust Top-Level
base exist in techniques for breaking a problem into smaller Specifications," Engr. Opt., Vol. 26, pp. 137-158.
problems; a good review of such methods can be found in Chen, W., Allen, J. K., Mistree, F. and Tsui, K.-L., 1995,
(Lewis and Mistree, 1997). Using these techniques, a complex "Integration of Response Surface Methods with the Compromise
problem may be broken down into smaller problems that allow Decision Support Problem in Developing a General Robust
efficient experimentation and metamodeling, which again will Design Procedure," ASME Design Automation Conference
(Azarm, S., et al., ed.), Boston, Massachusetts, ASME, New
lead to comprehensive and efficient exploration of a design York, DE-Vol. 82-2, pp. 485-492.

12 Copyright © 1997 by ASME

Chen, W., Simpson, T. W., Allen, J. K. and Mistree, F., 1996b, Kleijnen, J. P. C., 1987, Statistical Tools for Simulation
"Use of Design Capability Indices to Satisfy a Ranged Set of Practitioners, Marcel Dekker, New York.
Design Requirements," 1996 ASME Design Automation Koch, P. N., Allen, J. K. and Mistree, F., 1996, "Robust Concept
Conference (Dutta, D., ed.), Irvine, CA, ASME, New York, Paper Exploration for Configuring Turbine Propulsion Systems," 1996
No. 96-DETC/DAC-1090. ASME Design and Automation Conference (Dutta, D., ed.),
Cheng, B. and Titterington, D. M., 1994, "Neural Networks: A Irvine, CA, Paper No. 96-DETC/DAC-1285.
Review from a Statistical Perspective," Statistical Science, Vol. Korngold, J. C. and Gabriele, G. A., 1995, "Multidisciplinary
9(1), pp. 2-54. Analysis and Optimization of Discrete Problems Using Response
Chi, H.-W. and Bloebaum, C. L., 1995, "Mixed Variable Surface Methods," 21st Annual Advances in Design Automation
Optimization Using Taguchi's Orthogonal Arrays," 21st Annual Conference, Boston, MA, ASME, Vol. 82-1, pp. 173-180.
Advances in Design Automation Conference (Azarm, S., Dutta, Langley, P. and Simon, H. A., 1995, "Applications of Machine
D., et al., eds.), Boston, MA, ASME, Vol. 68-1, pp. 501-508. Learning and Rule Induction," Communications of the ACM,
Cressie, N., 1988, "Spatial Prediction and Ordinary Kriging," Vol. 38(11), pp. 55-64.
Mathematical Geology, Vol. 20(4), pp. 405-421. Laslett, G. M., 1994, "Kriging and Splines: An Empirical
Cressie, N., 1989, "Geostatistics," The American Statistician, Vol. Comparison of Their Predictive Performance in Some
43(4), pp. 197-202. Applications," Journal of the American Statistical Association,
Draper, N. R. and Lin, D. K. J., 1990a, "Connections Between Two- Vol. 89(42), pp. 391-400.
Level Designs of Resolutions III and V," Technometrics, Vol. Leech, W. J., 1986, "A Rule Based Process Control Method With
32(3), pp. 283-288. Feedback," Advances In Instrumentation, Vol. 41, pp. 169-175.
Draper, N. R. and Lin, D. K. J., 1990b, "Small Response-Surface Lewis, K. and Mistree, F., 1997, "The Other Side of
Designs," Technometrics, Vol. 32(2), pp. 187-194. Multidisciplinary Design Optimization: Accomodating a
Engelund, W. C., Douglas, O. S., Lepsch, R. A., McMillian, M. M. Multiobjective, Uncertain and Non-Deterministic World,"
and Unal, R., 1993, "Aerodynamic Configuration Design Using Engineering Optimization, in press.
Resopnse Surface Methodology Analysis," AIAA Aircraft Li, C.-C., Sheu, T.-S. and Lee, Y.-C., 1994, "Robustness
Design, Systems and Operations Meeting, Monterey, CA, Paper Consideration in Modelling a Prototype by Quality
No. AIAA 93-3967. Engineering," Design Theory and Methodology Conference
Evans, B. and Fisher, D., 1994, "Overcoming Process Delays with (Hight, T. K. and Mistree, F., eds.), Minneapolis, MN, ASME,
Decision Tree Induction," IEEE Expert, Vol. 9, pp. 60-66. Vol. 68, pp. 55-62.
Gadallah, M. H. and ElMaraghy, H. A., 1994, "The Tolerance Lu, S. C.-Y. and Tcheng, D. K., 1991, "Building Layered Models to
Optimization Problem Using a System of Experimental Design," Support Engineering Decision Making: A Machine Learning
Advances in Design Automation Conference, Minneapolis, MN, Approach," Journal of Engr. for Industry, Vol. 113, pp. 1-9.
ASME, Vol. 69-1, pp. 251-265. Lucas, J. M., 1974, "Optimum Composite Designs,"
Giovannitti-Jensen, A. and Myers, R. H., 1989, "Graphical Technometrics, Vol. 16(4), pp. 561-567.
Assessment of the Prediction Capability of Response Surface Lucas, J. M., 1976, "Which Response Surface Design is Best,"
Designs," Technometrics, Vol. 31(2), pp. 159-171. Technometrics, Vol. 18(4), pp. 411-417.
Giunta, A. A., Balabanov, V., Haim, D., Grossman, B., Mason, W. Lucas, J. M., 1994, "Using Response Surface Methodology to
H. and Watson, L. T., 1996, "Wing Design for a High-Speed Achieve a Robust Process," Journal of Quality Technology, Vol.
Civil Transport Using a Design of Experiments Methodology," 26(4), pp. 248-260.
6th AIAA/USAF/NASA/ISSMO Symposium on Multidisci- Matheron, G., 1963, "Principles of Geostatistics," Economic
plinary Analysis and Optimization, Bellevue, WA, AIAA, Inc., Geology, Vol. 58, pp. 1246-1266.
Vol. 1, pp. 168-183. Mavris, D., Bandte, O. and Schrage, D., 1996, "Application of
Giunta, A. A., Dudley, J. M., Narducci, R., Grossman, B., Haftka, Probabilistic Methods for the Determination of an Economically
R. T., Mason, W. H. and Watson, L. T., 1994, "Noisy Robust HSCT Configuration," 6th AIAA/USAF/NASA/ISSMO
Aerodynamic Response and Smooth Approximations in HSCT Symposium on Multidisciplinary Analysis and Optimization,
Design," 5th AIAA/USAF/NASA/ISSMOSymposium on Multi- Bellevue, WA, AIAA, Inc., Vol. 2, pp. 968-978.
disciplinary Analysis and Optimization, Panama City, FL, Vol. Mavris, D. N., Bandte, O. and Schrage, D. P., 1995, "Economic
2, pp. 1117-1128. Uncertainty Assessment of an HSCT Using a Combined Design
Healy, M. J., Kowalik, J. S. and Ramsay, J. W., 1975, "Airplane of Experiments/Monte Carlo Simulation Approach," 17th
Engine Selection by Optimization of Surface Fit Annual Conference of International Society of Parametric
Approximations," Journal of Aircraft, Vol. 12(7), pp. 593-599. Analysts, San Diego, CA.
Hong, J., Dagli, C. H. and Ragsdell, K. M., 1994, "Artificial Neural Mitchell, T., Morris, M. and Ylvisaker, D., 1988, "Existence of
Network and the Taguchi Method Application for Robust Smoothed Stationary Processes on an Interval," Working Paper.
Wheatstone Bridge Design," Advances in Design Automation Montgomery, D. C., 1997, Design and Analysis of Experiments,
(Gilmore, B. J., Hoeltzel, D. A., et al., eds.), Minneapolis, MN, Fourth Edition, John Wiley & Sons, New York.
ASME, Vol. 69-2, 37-41. Montgomery, D. C. and Evans, D. M., Jr., 1975, "Second-Order
Hunter, J. S., 1985, "Statistical Design Applied to Product Response Surface Designs in Computer Simulation," Simulation,
Design," Jrnl. of Quality Technology, Vol. 17(4), pp. 210-221. Vol. 29(6), pp. 169-178.
Hunter, J. S. and Naylor, T. H., 1970, "Experimental Designs for Myers, R. H., Khuri, A. I. and Carter, W. H., 1989, "Response
Computer Simulation Experiments," Management Science, Vol. Surface Methodology: 1966-1988," Technometrics, Vol. 31(2),
16(7), pp. 422-434. pp. 137-157.
Journel, A. G., 1986, "Geostatistics: Models and Tools for the Myers, R. H. and Montgomery, D. C., 1995, Response Surface
Earth Sciences," Math. Geol., Vol. 18(1), pp. 119-140. Methodology: Process and Product Optimization Using
Designed Experiments, John Wiley & Sons, New York.

13 Copyright © 1997 by ASME

Peplinski, J. D., Allen, J. K. and Mistree, F., 1996, "Integrating Shoemaker, A. C., Tsui, K. L. and Wu, J., 1991, "Economical
Product Design with Manufacturing Process Design Using the Experimentation Methods for Robust Design," Technometrics,
Robust Concept Exploration Method," 1996 ASME Design Vol. 33(4), pp. 415-427.
Theory and Methodology Conference (Wood, K., ed.), Irvine, Stanley, D. O., Unal, R. and Joyner, C. R., 1992, "Application of
CA, Paper No. 96-DETC/DTM-1502. Taguchi Methods to Propulsion System Optimization for SSTO
Phadke, M. S., 1989, Quality Engineering using Robust Design, Vehicles," Spacecraft and Rockets, Vol. 29(4), pp. 453-459.
Prentice Hall, Englewood Cliffs, New Jersey. Sundaresan, S., Ishii, K. and Houser, D. R., 1993, "A Robust
Ramberg, J. S., Sanchez, S. M., Sanchez, P. J. and Hollick, L. J., Optimization Procedure with Variations on Design Variables and
1991, "Designing Simulation Experiments: Taguchi Methods Constraints," Advances in Design Automation, Albuquerque,
and Response Suface Metamodels," Proceedings of the 1991 NM, ASME, Vol. 65-1, pp. 379-386.
Winter Simulation Conference (WSC) (Nelson, B. L., Kelton, W. Unal, R., Lepsch, R. A., Engelund, W. and Stanley, D. O., 1996,
D., et al., eds.), Phoenix, AZ, IEEE, pp. 167-176. "Approximation Model Building and Multidisciplinary Design
Reddy, S. Y., 1996, "HIDER: A Methodology for Early-Stage Optimization Using Response Surface Methods," 6th
Exploration of Design Space," 1996 ASME Design Automation AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary
Conference (Dutta, D., ed.), Irvine, CA, ASME, New York, Paper Analysis and Optimization, Bellevue, WA, AIAA, Inc., Vol. 1,
No. 96-DETC/DAC-1089. pp. 592-597.
Renaud, J. E. and Gabriele, G. A., 1993, "Improved Coordination in Unal, R., Stanley, D. O., Engelund, W. and Lepsch, R., 1994a,
Non-Hierarchic System Optimization," AIAA Journal, Vol. "Design for Quality using Response Surface Methods: An
31(12), pp. 2367-2373. Alternative to Taguchi's Parameter Design Approach,"
Renaud, J. E. and Gabriele, G. A., 1994, "Approximation in Engineering Management Journal, Vol. 6(3), pp. 40-48.
Nonhierarchic System Optimization," AIAA Journal, Vol. 32(1), Unal, R., Stanley, D. O. and Joyner, C. R., 1992, "The Role of
pp. 198-205. Statistically Designed Experiments in Conceptual Aerospace
Renaud, J. E. and Gabrielle, G. A., 1991, "Sequential Global Vehicle Design," 1992 IEEE International Engineering
Approximation in Non-Hierarchic System Decomposition and Management Conference, Eatontown, NJ, ASEM, pp. 211-214.
Optimization," Advances in Design Automation, Design Unal, R., Stanley, D. O. and Joyner, C. R., 1994b, "Parametric
Automation and Design Optimization, Miami, FL, ASME, Vol. Model Building and Design Optimization Using Reponse Surface
32-1, pp. 191-200. Methods," Journal of Parametrics, Vol. 14(1), pp. 81-96.
Ross, P. J., 1988, Taguchi Techniques for Qualtiy Engineering, Unal, R., Wu, K. C. and Stanley, D. O., 1994c, "A Study of Design
McGraw-Hill, New York, NY. of Experiments Methods for Design Optimization and Quality
Roux, W. J., Stander, N. and Haftka, R. T., 1996, "Response Improvement," 15th Annual Conference of the ASEM, ASEM.
Surface Approximations for Structural Optimization," 6th Venter, G., Haftka, R. T. and Starnes, J. H., Jr., 1996,
AIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary Analysis "Construction of Response Sufaces for Design Optimization
and Optimization, Bellevue, WA, AIAA, Inc., Vol. 1, pp. 565- Applications," 6th AIAA/USAF/NASA/ISSMO Symposium on
578. Multidisciplinary Analysis and Optimization, Bellevue, WA,
Rowell, L. F., Braun, R. D., Olds, J. R. and Unal, R., 1996, "Recent AIAA, Inc., Vol. 1, pp. 548-564.
Experiences in Multidisciplinary Conceptual Design Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J.
Optimization for Launch Vehicles," 6th AIAA/USAF/NASA/ and Morris, M. D., 1992, "Screening, Predicting, and Computer
ISSMO Symposium on Multidisciplinary Analysis and Experiments," Technometrics, Vol. 34(1), pp. 15-25.
Optimization, Bellevue, WA, AIAA, Inc., Vol. 1, pp. 642-654. Welch, W. J., Yu, W. K., Kang, S. M. and Sacks, J., 1990,
Rumelhart, D. E., Widrow, B. and Lehr, M. A., 1994, "The Basic "Computer Experiments for Quality Control by Parameter
Ideas in Neural Networks," Communications of the ACM, Vol. Design," Journal of Quality Technology, Vol. 22(1), pp. 15-22.
37(3), pp. 87-92. Wujek, B., Renaud, J. E., Batill, S. M. and Brockman, J. B., 1996,
Sacks, J., Schiller, S. B. and Welch, W. J., 1989a, "Designs for "Concurrent Subspace Optimization Using Design Variable
Computer Experiments," Technometrics, Vol. 31(1), pp. 41-47. Sharing in a Distributed Computing Environment," Concurrent
Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P., 1989b, Engineering: Research and Appl., Vol. 4(4), pp. 361-378.
"Design and Analysis of Computer Experiments," Statistical Wujek, B. A. and Renaud, J. E., 1997, "A Concurrent Engineering
Science, Vol. 4(4), pp. 409-435. Approach for Multidisciplinary Design in a Distributed
Sellar, R. S. and Batill, S. M., 1996, "Concurrent Subspace Computing Environment," Proceedings in Applied Mathematics,
Optimization Using Gradient-Enhanced Neural Network Philadelphia, PA, SIAM, Vol. 80, pp. 189-208.
Approximations," 6th AIAA/NASA/USAF Multidisciplinary Yerramareddy, S., Tcheng, D. K., Lu, S. C.-Y. and Assanis, D. N.,
Analysis & Optimization Symposium, Bellevue, WA, AIAA, 1992, "Creating and Using Models for Engineering Design,"
Vol. 1(of 2), pp. 319-330. IEEE Expert, Vol. 7(3), pp. 52-59.
Sellar, R. S., Batill, S. M. and Renaud, J. E., 1996a, "A Neual Yu, J.-C. and Ishii, K., 1993, "A Robust Optimization Method for
Network-Based, Concurrent Subspace Optimization Approach to Systems with Significant Nonlinear Effects," Advances in
Multidisciplinary Design Optimization," 34th AIAA Aerospace Design Automation, Albuquerque, NM, ASME, Vol. 65-1, pp.
Sciences Meeting, Reno, NV, AIAA, AIAA 96-0714. 371-378.
Sellar, R. S., Stelmack, M. A., Batill, S. M. and Renaud, J. E.,
1996b, "Multidisciplinary Design and Analysis of an Integrated
Aeroelastic/Propulsive System," 37th AIAA/ASME/ASCE/
AHS/ASC Structures, Structural Dynamics, and Materials
Conference, Salt Lake City, UT, AIAA, pp. 583-593.

14 Copyright © 1997 by ASME

View publication stats