See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/23884611

On the Use of Statistics in Design and the Implications for Deterministic

Computer Experiments

Article · February 1997

Source: NTRS

CITATIONS READS
140 1,409

4 authors, including:

Timothy W. Simpson Patrick N Koch

Pennsylvania State University SAS Institute
363 PUBLICATIONS   13,329 CITATIONS    41 PUBLICATIONS   2,404 CITATIONS   

SEE PROFILE SEE PROFILE

Janet Allen
University of Oklahoma
355 PUBLICATIONS   6,092 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

The Goal-oriented Inverse Design (GoID) Method for Systems-based Design Under Uncertainty View project

Investigating the Effectiveness of Machine Learning Paradigms for Supporting Engineering Designers in Rapidly Evolving Digital Manufacturing View project

All content following this page was uploaded by Patrick N Koch on 05 June 2014.

The user has requested enhancement of the downloaded file.

 Proceedings of DETC’97
1997 ASME Design Engineering Technical Conferences
September 14-17, 1997, Sacramento, California

DETC97DTM3881

ON THE USE OF STATISTICS IN DESIGN AND THE IMPLICATIONS FOR

DETERMINISTIC COMPUTER EXPERIMENTS

Timothy W. Simpson, Jesse D. Peplinski, Patrick N. Koch, and Janet K. Allen

Systems Realization Laboratory
Woodruff School of Mechanical Engineering
Georgia Institute of Technology
Atlanta, Georgia 30332-0405, USA

ABSTRACT
Perhaps the most prevalent use of statistics in engineering
design is through Taguchi's parameter and robust design - using
orthogonal arrays to compute signal to noise ratios in a process
of design improvement. In our view, however, there is an
equally exciting use of statistics in design that could become
just as prevalent: it is the concept of metamodeling, whereby
statistical models are built to approximate detailed computer
analysis codes. Although computers continue to get faster, our
analysis codes always seem to keep pace, so that their computa-
tional time remains non-trivial. Through metamodeling,
approximations of these codes are built that are orders of magni-
tude cheaper to run. These metamodels can then be linked to
optimization routines for fast analysis, or they can serve as a
bridge for integrating codes across different domains.
In this paper we first review metamodeling techniques that
encompass the Design of Experiments, Response Surface
Methodology, Taguchi methods, neural networks, inductive
learning, and kriging. We discuss their existing applications in
engineering design and then address the dangers of applying
traditional statistical techniques to approximate deterministic
computer analysis codes. We conclude with recommendations
for the appropriate use of metamodeling techniques in given
situations and how common pitfalls can be avoided.
KEYWORDS: Design of Experiments, Response
Surface Methodology, Metamodeling, Kriging, Neural Net-
works, Rule Induction, D-Optimal, Taguchi Robust Design
1 INTRODUCTION
Much of today's engineering analysis work consists of run-
ning complex computer analysis codes -- supplying a vector of
design variables (inputs) x and receiving a vector of responses
(outputs) y. Despite continual advances in computing power
and speed, the expense of running many analysis codes remains
non-trivial. Single evaluations of aerodynamic or finite-
element codes can take from minutes to hours, if not longer.
What's more, this mode of query-and-response can often lead to
a trial and error approach to design, where a designer may never
uncover the functional relationship between x and y and
therefore may never identify the best settings for input values.
Statistical techniques are widely used in engineering design
to address these concerns. The basic approach is to construct
approximations of the analysis codes that are much more effi-
cient to run and that yield insight into the functional
relationship between x and y. If the true nature of a computer
analysis code is represented as
y = f(x),
then a "model of the model" or metamodel (Kleijnen, 1987) of
the analysis code is taken to be
y = g(x), and so y = y + ε
where ε represents both the error of approximation and
measurement or random errors. The most common meta-
modeling approach is to apply the Design of Experiments
(DOE) to identify an efficient set of computer runs (x 1, x 2, ...,
x n) and then employ regression analysis to create a polynomial
approximation of the computer analysis code. These approxi-
mations then can replace the existing code while providing:
• a better understanding of the relationship between x and y
(we typically lose this understanding when running computer
codes "blindly"),
• facilitated integration of domain dependent computer codes
(we no longer work with the codes themselves but rather
simple approximations of them), and
• fast analysis tools for optimization and exploration of the
design space (we become more efficient because we are using
the approximations rather than the computationally expensive
codes themselves).
However, we have found that many published applications
(including our own) utilizing these statistical methods for

1 Copyright © 1997 by ASME

computer-oriented design are statistically questionable, if not
incorrect. The root cause is that many of our analysis codes are
deterministic, which means that the error of approximation is
not due to random effects. This calls into question most subse-
quent statistical analyses of model significance. Consequently,
our purpose in this paper is to highlight the potential statistical
pitfalls in metamodeling and to provide some general recom-
mendations for the proper use of statistical techniques in design.
In Section 2, we present a review of metamodeling techniques
that encompasses regression, neural networks, inductive learn-
ing, and the more advanced approach of kriging. We conclude
Section 2 with an introduction to the general statistical
approaches of Response Surface Methodology (RSM) and
Taguchi's robust design. In Section 3, we describe the engi-
neering design context for statistical applications, review
existing applications, methods and tools, and close by taking a
closer look at metamodeling techniques for deterministic com-
puter experiments. In Section 4 we present some general rec-
ommendations for avoiding some important pitfalls in the
application of metamodeling to computer-oriented design, and
in Section 5 we conclude by discussing briefly some of the
more advanced issues that continue to make metamodeling an
active research area.
focus on experimental designs with particular emphasis on
(fractional) factorial designs, central composite designs, and
orthogonal arrays. We also discuss measures of merit for the
evaluation of experimental designs. In Section 2.2, we discuss
model choice and model fitting, focusing on response surfaces,
neural networks, inductive learning and kriging. We conclude
with an overview of two of the more prevalent metamodeling
techniques, namely, Response Surface Methodology and
Taguchi's robust design.
2.1 Experimental Design
Properly designed experiments are essential for effective
computer utilization. The traditional approach in engineering is
to vary one parameter at a time within a computer analysis code
and observe the effects, or to randomly assign different combi-
nations of factor settings to be used as alternative analyses for
comparison. Experimental design techniques developed for
effective physical experiments, however, are being applied for
the design of engineering computer experiments to increase the
efficiency (computer time) and effectiveness of these analysis
codes. In this section an overview of different types of experi-
ment designs is provided, along with measures of merit for
selecting or comparing different designs.

2 REVIEW OF METAMODELING TECHNIQUES 2 . 1 . 1 A Survey of Experimental Designs. A n

In its most basic sense, metamodeling involves (a) experimental design formally represents a sequence of experi-
choosing an experimental design for generating data, (b) ments to be performed, expressed in terms of factors (design
choosing a model to represent the data, and then (c) fitting the variables) set at specified levels (predefined values). An
model to the observed data. There are a variety of options for experimental design is represented mathematically by a matrix
each of these steps as shown in Figure 1, and we have X where the rows denote experimental runs and the columns
attempted to highlight a few of the more prevalent ones. For denote the particular factor setting for each factor for each run.
example, Response Surface Methodology usually employs Factorial Designs: The most basic experimental design is a
central composite designs, second order polynomials and least full factorial design. The number of design points dictated by a
squares regression analysis. full factorial design is the product of the number of levels for
SAMPLE each factor. The most common designs are the 2k (for evaluat-
EXPERIMENTAL MODEL MODEL METAMODELING ing main effects and interactions) and 3k designs (for evaluating
DESIGN CHOICE FITTING TECHNIQUES
main and quadratic effects and interactions) for k factors at 2 and
(Fractional) Factorial Polynomial
(linear, quadratic, etc.)
Least Squares
Regression
Response Surface
Methodology
3 levels, respectively. A 23 full factorial design is shown
Central Composite graphically in Figure 2(a).
Box-Behnken Splines Weighted Least The size of a full factorial experiment increases exponen-
(linear, cubic) Squares Regression
D-Optimal
tially with the number of factors which may lead to an unman-
G-Optimal
Best Linear ageable number of experiments, e.g., 10 factors each with 2
Frequency-Domain Unbiased Predictor Kriging
(BLUP) levels requires 210 or 1024 experiments. Fractional factorial
Orthogonal Array
Best Linear
designs can be used when experiments are costly and the
Plackett-Burman Kernel Smoothing Predictor (BLP) number of design points for a full factorial design is large. A
Hexagon
Radial Basis Functions Log-Likelihood
fractional factorial design consists of a fraction of a full factorial
Hybrid design; the most common fractional factorial designs are 2(k-p)
Latin Hypercube Network of Neurons Backpropagation Neural
Networks designs where a fraction is equal to 1/2(p). A half fraction of the
Select By Hand 23 full factorial design is shown in Figure 2(b).
Rulebase or Entropy Inductive
Random Selection Decision Tree (information-theoretic) Learning The reduction of the number of design points associated
with a fractional factorial design does not come without a price.
Design resolution and aliasing of effects are two critical (and
Figure 1. Techniques for Metamodeling related) issues which are important when selecting a fractional
factorial design. The 23 full factorial design shown in Figure
In the remainder of this section we provide a brief overview 2(a) allows estimation of all main effects (x1, x 2, x 3), all two
of several of the options listed in Figure 1. In Section 2.1, we factor interactions (x1x2, x 1x3 and x 2x3), as well as the three

2 Copyright © 1997 by ASME

factor interaction (x1x2x3). For the 23-1 fractional factorial indi-
cated by the solid dots in Figure 2(b), the main effects are alised
(or biased) with the two factor interactions. Aliased effects can-
not be estimated independently unless all but one of the aliased
effects are known or assumed not to exist. The resolution of an
experiment design is defined by the aliasing of effects: a design
of resolution R is one in which no p-factor effect is aliased with
any other effect containing less than R-p factors (Box, et al.,
1978). The 23-1 design in Figure 2(b) can be at most a Resolu-
tion III design, denoted23-1III.
Often 2k and 2(k-p) designs are used for identifying or screen-
ing important factors when the number of factors is too large to
evaluate higher order effects. In these cases the sparsity of
effects principle (Montgomery, 1997) is invoked, in which the
system is assumed to be dominated by the main effects and their
low-order interactions. Based on this principle, two level frac-
tional factorial designs can be used to "screen" the factors to
determine which have the largest effect. The sparsity of effects
principle may not always hold true, however. As noted by
Hunter (1970), every design provides aliased estimates (quadratic
and cubic effects, if present, will bias the estimates of the mean
and main effects respectively when a two level factorial design
is used). "Any phenomenon omitted from a fitted model will
confound [i.e., alias] certain estimated parameters in the model
regardless of the design used. Good fractional factorial designs
are carefully arranged so that estimates of effects thought to be
important are confounded by effects thought to be unimpor-
tant."
One specific family of fractional factorial designs com-
monly used for screening are the two level Plackett-Burman (P-
B) designs. These resolution III designs are constructed to study
k=N-1 factors in N=4m design points. The class of P-B designs
in which N is a power of two are called geometric designs are
are identical to the usual 2k-p fractional factorials. When N is
strictly a multiple of four, the P-B designs are referred to as
non-geometric designs and have very messy alias structures.
Their use in practical problems is often problematic particularly
when the design is saturated, i.e., the number of factors is
exactly N-1. If all interactions are negligible, however, these
designs allow unbiased estimation of all main effects, using
only one more design point than the number of factors, with
smallest possible variance (Box, et al., 1978). Minimum vari-
ance and minimum size designs will be discussed in Section
2.1.2. For a more complete discussion of factorial designs,
confounding, and design resolution (see Myers and
Montgomery, 1995).
X2
X2 X2
X1 X1
X3 X3
X3
(a) 23 Full Factorial (b) 23-1 Fractional Factorial (c) Composite Design
Figure 2. Basic Three-Factor Designs
Central Composite and Box-Behnken Designs: For estimat-
ing quadratic effects the 3k or 3 k-p designs can be used but often
lead to an unmanageable number of design points, e.g., 7
factors at three levels requires 37 or 2187 experiments. The
most common second-order designs, configured to reduce the
number of design points, are central composite and Box-
Behnken designs.
A central composite design (CCD) is a two level (2k or 2 (k-
p)
) factorial design, augmented by n0 center points and two "star"
points positioned at ±α for each factor. This design, shown for
three factors in Figure 2(c) consists of 2(k-p) + 2 k + n 0 total de-
sign points to estimate 2k + k(k-1)/2 + 1 coefficients. Setting
α=1 locates the star points on the centers of the faces of the
cube (for three factors), giving a face-centered central composite
(CCF) design. It should be noted that for values of α other than
1, each factor is evaluated at five levels.
Often it is desirable to employ the smallest number of fac-
tors levels (e.g., three levels to fit a quadratic model) when
creating an experimental design. One common, economical
class of such designs is the Box-Behnken designs. These
designs are formed by combining 2k factorials with incomplete
block designs (see Box and Draper, 1987). These designs do
not contain any points at the vertices of the hypercube defined
by the upper and lower limits for each factor. This is desirable
if these extreme points are expensive or impossible to test.
More information about central composite and Box-Behnken
design can be found in (Montgomery, 1997).
Orthogonal Arrays: The experiment designs used by
Taguchi, called orthogonal arrays, are most often simply
fractional factorial designs in two or three levels (2(k-p) and 3(k-p)
designs). These arrays are constructed to reduce the number of
design points necessary to evaluate the required effects; the two-
level L4, L 12, and L16 arrays, for example, allow 3, 11, and 15
factors/effects to be evaluated in 4, 12, and 16 design points
respectively. In many cases, these designs are identical to those
given by Plackett and Burman (Lucas, 1994). The definition of
orthogonality, for these arrays and other experiment designs, is
given in Section 2.1.2. An overview of Taguchi's approach to
parameter design, within which the orthogonal arrays are
implemented, is given in Section 2.3.
2 . 1 . 2 Measures of Merit for Evaluating Experi-
mental Designs. Selecting the appropriate design is
essential for effective experimentation. Experimenters must
balance the desire to gain as much information as possible
about the response-factor relationships with the cost of
experimentation and need for efficiency (measured in number of
runs). Several measures of merit are available and useful for
evaluating and comparing experimental designs to ensure the
appropriate experiment is designed.
Orthogonality, Rotatability, Minimum Variance, and
X1
Minimum Bias: Four desirable characteristics of an
experimental design, to facilitate efficient estimates of the
parameters, are orthogonality, rotatability, minimum variance,
and minimum bias. A design is orthogonal if for every pair of
factors xi and x j the sum of the crossproducts for the N design
points is zero. For a first order model, the estimates of all
coefficients will have minimum variance if the design is
configured so that
3 Copyright © 1997 by ASME
 N D-optimal determinant value, normalized by the size of the
∑ x2iu = N
u=1
the variance of predictions y will also have constant variance at
a fixed distance from the center of the design, and the design
will be rotatable.
For second order modeling, Hunter (1985) suggests that
orthogonality loses its importance. "If the objective of the
experimenter is to forecast a response at either present or future
settings of x, then an unbiased minimum variance estimate of
the forecast y is required. In the late 1950's Box and his co-
workers demonstrated that rotatability....and the minimization
of bias from higher order terms....were the essential criteria for
good forecasting."
Unsaturated/Saturated and Supersaturated Designs: In many
cases the primary concern in the design of an experiment is the
size (number of runs) of the resulting design. Most experiment
designs are unsaturated in that they contain at least two more
design points than the number of factors. An experiment
design is said to be saturated if the number of design points is
equal to one more than the number of factor effects to be
estimated. Saturated fractional factorial designs allow unbiased
estimation of all main effects with smallest size and smallest
possible variance (Box, et al., 1978). In a supersaturated
design, then, the number of design points is less than or equal
to the number of factors. The most common examples of
saturated designs are the Plackett-Burman two level designs
discussed in the previous section, and the related orthogonal
arrays of Taguchi. Additional aspects of supersaturated designs
can be found in (Draper and Lin, 1990a; Draper and Lin,
1990b).
For estimating second order effects, small composite
designs have be developed to reduce the number of design points
necessary for a composite design. A small composite design
becomes saturated if the number of design points is 2k + k(k-
1)/2 + 1 (the number of coefficients to be estimated for a full
quadratic model). Myers and Montgomery (1995) note that
recent work has suggested that these designs may not be good
choices in all applications; additional discussion on small
composite designs can be found in (Box and Draper, 1987;
Lucas, 1974).
D-Optimal, D-Efficiency, and G-Efficiency: It is most
desirable to use unsaturated designs, when building predictive
models, unless running the necessary experiments is
prohibitively expensive. For comparing experiments based on
number of design points and information obtained, many
measures of merit exist. A common measure is D-efficiency.
A design is said to be D-Optimal if for the the model the
volume of the joint confidence region for the vector of
unknown parameters is minimum. As a consequence, the
global maximum of the determinant of X'X is achieved. Here
X is the expanded design matrix, including N rows (one for
each design point or run), and p columns (one for each
coefficient to be estimated, including the overall mean). The D-
efficiency statistic for comparing designs, given in Equation 1,
compares the determinant value achieved for the design with the
4 Copyright © 1997 by ASME
matrix for comparing different size designs.
D-efficiency = (|X'X|design /|X'X|D-optimum)1/p (1)
The D-efficiency statistic is the primary criteria with which
designs have been compared. Other statistics for comparing
designs such as G-efficiency and A-optimality have also been
formulated (see Myers and Montgomery, 1995). Having
completed our review of experimental design alternatives and
measures of merit, we can now turn to the issues of model
choice and model fitting.
2.2 Model Choice and Model Fitting
After selecting an appropriate experimental design and
performing the necessary computer runs, the next step involves
choosing an approximating model and fitting method. Many
alternative models and methods exist, but for practicality we
review in this section the four that we have found to be most
prevalent in the literature: response surfaces, neural networks,
inductive learning and kriging.
2 . 2 . 1 Response Surfaces. Given a response of
interest, y, and a vector of independent factors x thought to
influence y, the relationship between y and x can be written as
follows:
y = f(x) + ε , (3)
where ε represents random error which is assumed to be
normally distributed with mean zero and standard deviation σ.
Since the true response surface function f(x) is usually
unknown, a response surface g(x) is created to approximate f(x).
Predicted values are then obtained using y = g(x).
The most widely used response surface approximating
functions are simple low-order polynomials. If little curvature
appears to exist, the first-order polynomial given in Equation 4
can be employed. If significant curvature exists, the second-
order polynomial in Equation 5, including all two-factor
interactions, can be used.
k
y = b0 + ∑ bixi (4)
i=1
k k k k
y = b0 + ∑ bixi + ∑ biix2i + ∑ ∑ bijxixj
i<j
(5)
i=1 i=1 i=1 j=1
The parameters of the polynomials in Equations 4 and 5 are
usually determined using a least squares regression analysis to
fit these response surface approximations to existing data.
These approximations are normally used for prediction within
Response Surface Methodology. A more complete discussion
of response surfaces and least squares fitting can be found in
(Myers and Montgomery, 1995). An overview of RSM is
given in Section 2.3.
2 . 2 . 2 Neural Networks. A neural network is composed
of neurons (single-unit perceptrons) which are multiple linear
regression models with a nonlinear (typically sigmoidal)
transformation on y. If the inputs to each neuron are denoted
x1, x 2, ..., x n, and the regression coefficients are denoted by the
weights, wi, then the output, y, might be given by
y= 1
1 + e -η/T
where η = Σwi xi + β. A neural network is then created by
assembling the neurons into an architecture; the most prevalent
is the multi-layer feedforward architecture.
There are two main issues in building a network: 1)
specifying the architecture, and 2) training the network to
perform well with reference to a training set. "To a statistician,
this is equivalent to (i) specifying a regression model, and (ii)
estimating the parameters of the model given a set of data"
(Cheng and Titterington, 1994). If the architecture is made
large enough, a neural network can be a nearly universal
approximator (Rumelhart, et al., 1994).
"Training" a neural net refers to determining the proper
values of all the weights (wi ) in the architecture, and is
accomplished most commonly through backpropagation
(Rumelhart, et al., 1994). First, a set of p training data points
is assembled {(x1,y 1}, (x2,y 2), ..., (x p,y p)}. For a network with
a single output y, the performance of the network is then
defined as
E = ∑ (yp - yp )2
p analysis methods were employed with limited success. Some
where yp is the output that results from the network given
input x p and E is defined as the total error of the system. The
weights are then adjusted in proportion to
∂E ∂y .
∂y ∂wij
Neural networks are best suited to approximate
deterministic functions in regression-type applications. Cheng
and Titterington (1994) note that "In most applications of
neural networks that generate regression-like output, there is no
explicit mention of randomness. Instead, the aim is function
approximation." Typical applications are speech recognition
and handwritten character recognition; very often the data is
complex and of high dimensionality . Networks with tens of
thousands of parameters are not unheard of, and the amount of
training data is similar. Gathering the training data and
determining the model parameters is a process that can be very
computationally expensive.
2 . 2 . 3 Rule Induction. Rule induction is one of five
main paradigms of machine learning that also include neural
networks, case-based learning, genetic algorithms, and analytic
learning (Langley and Simon, 1995). Of the five, inductive
learning is most akin to regression and metamodeling and is
therefore our focus here. An inductive learning system induces
rules from examples, so the fundamental modeling constructs
are condition-action rules. These rules in essence partition the
data into discrete categories, and can be combined into decision
trees for ease of interpretation. Many of the applications of
5 Copyright © 1997 by ASME
inductive learning have been in process control and diagnostic
systems, and inductive learning approaches can be used to
automate the knowledge-acquisition process of building expert
systems. Excellent examples can be found in (Evans and
Fisher, 1994; Langley and Simon, 1995; Leech, 1986).
The training data collected are in the form of {(x 1,y 1),
(x 2,y 2), ..., (x n,y n)} where each x i represents a vector of
attribute values (such as processing parameters and
environmental conditions), and each yi represents a
corresponding observed output value. Although attributes and
outputs can be real-valued, the method is better suited to
discrete-valued data. Real values must often be transformed into
discrete representations (Evans and Fisher, 1994). Once the data
set has been collected, training algorithms build a decision tree
by selecting the "best" divisive attribute and then recursively
calling the resulting data subsets. Although trees can be built
by selecting attributes randomly, a more efficient approach is to
select the attribute that minimizes the amount of information
needed for category membership. The mathematics of such an
information-theoretic approach are given in (Evans and Fisher,
1994).
Although decision trees seem best-suited for applications
with discrete input and output values, there are also applications
with continuous variables that have met with greater success
than standard statistical analysis. For example, Leech (1986)
reports a process-control application where "Standard statistical
of the data were non-numerical, the dependencies between
variables were not well understood, and it was necessary to
simultaneously control several characteristics of the final
product while working within system constraints. The results
of the statistical analysis, a set of correlations for each output
of interest, were difficult for people responsible for the day-to-
day operation to interpret and use."
2 . 2 . 4 Kriging. Since many computer analysis coedes are
deterministic and are therefore not subject to measurement error,
the usual measures of uncertainty derived from least-squares
residuals have no obvious statistical meaning (Sacks, et al.,
1989b). Consequently, some statisticians (e.g., (Sacks, et al.,
1989a; Welch, et al., 1992)) have suggested modeling the
response as a combination of a polynomial model (usually
linear) plus departure, i.e.,
k
y(x) = ∑ βjfj(x) + Z(x)
j=1
where Z(x) is the systematic departure from the assumed model.
Z(•) represents the realization of a stochastic process and is
assumed to have mean zero and covariance
V(w,x) = σ2R(w,x)
between Z(w) and Z(x) where σ2 is the process variance and
R(w,x) is the correlation. The covariance structure of Z relates
to the smoothness of the approximating surface.
One method of analysis for such models is kriging
(Matheron, 1963) which works as follows (Sacks, et al.,
1989b). Given a design S = {s 1, s 2, ..., s n} and data ys = {y(s1),
..., y(s n)}', we are concerned with estimating the y(x) at an
untried x using the linear (or polynomial) predictor
y(x) = c'(x)ys.
If we consider y(x) as random, y s becomes Y s = [Y(s 1), ...,
Y(sn)]', and we can compute the mean squared error of the
predictor averaged over the random process. Thus, the best
linear unbiased predictor (BLUP) is found by picking the n x 1
vector c(x) to minimize
MSE[y(x)] = E[c'(x)Ys - Y(x)]2
subject to the constraint for unbiasedness
E[c'(x)Ys] = E[Y(x)] (6)
R(w,x) must first be specified in order to compute Equation 6
and should reflect the characteristics of the computer code. For
a smooth response, a covariance function with some derivatives
might be preferred, whereas an irregular response might call for
a function with no derivatives. Sacks, et al. (1989b) discuss
several possible correlation functions which result in a linear
interpolation or cubic spline interpolation of the data (see
(Mitchell, et al., 1988) for more details). Consequently,
kriging is extremely flexible due to the wide range of
correlation functions R(w,x) which may be chosen. Depending
on the choice of the correlation function, kriging can either
"honor the data", providing an exact interpolation of the data, or
"smooth the data", providing a nonexact interpolation of the
data (Cressie, 1988). More information about kriging and the
notion of modeling deterministic computer experiments as the
realization of a stochastic process can be found in (Cressie,
1989; Journel, 1986; Laslett, 1994; Matheron, 1963; Sacks, et
al., 1989b). Cressie (1988) deals specifically with predicting
noiseless versions of Z, as is the case for deterministic
computer experiments. As a final note, keep in mind that
kriging and splines (nonparametric regression models) are not
the same. In fact, in several comparative studies, kriging has
been seen to outperform splines and never performs worse than
them (Laslett, 1994).
2.3 Strategies for Experimentation and
Metamodeling
Two widely used methods incorporating experimental
design, model building, and prediction are Response Surface
Methodology and Taguchi's robust design or parameter design.
A brief overview of these two approaches is provided.
2 . 3 . 1 Response Surface Methodology. Different
authors describe Response Surface Methodology differently.
Myers, et al. (1989) define RSM as "a collection of tools in
design or data analysis that enhance the exploration of a region
of design variables in one or more responses." Box and Draper
(1987) state that, "Response surface methodology comprises a
group of statistical techniques for empirical model building and
model exploitation. By careful design and analysis of
experiments, it seeks to relate a response, or output variable to
the levels of a number of predictors, or input variables, that
6 Copyright © 1997 by ASME
affect it." Finally, Biles (1984) defines RSM as the, "body of
techniques by which one experimentally seeks an optimum set
of system conditions".
RSM then encompasses and incorporates the design of
experiments (Section 2.1), response surface model building
(Section 2.2.1), and 'model exploitation' for exploring a factor
space and seeking optimum factor settings. The general RSM
approach includes all or a subset of the following steps:
i) screening: when the number of factors is too large for a
comprehensive exploration and/or when experimentation is
expensive, screening experiments are used to reduce the set
of factors to those that are most important to the
response(s) being investigated;
ii) first-order experimentation: when the starting point is far
from an optimum (or in general when knowledge about the
space being investigated is sought), first order-models and
an approach such as steepest-ascent are employed to
"rapidly and economically move to the vicinity of the
optimum" (Montgomery and Evans, 1975);
iii) second-order experimentation: after the best solution using
first-order methods is obtained, a second-order model is fit
in the region of the first-order solution to evaluate
curvature effects and attempt to improve the solution.
A more detailed description of RSM techniques and tools can be
found in (Box and Draper, 1987), and a comprehensive review
of RSM developments and applications from 1966-1988 is
given in (Myers, et al., 1989).
2 . 3 . 2 Taguchi's Robust Design. Genichi Taguchi
developed an approach for industrial product design that is built
on the foundation of statistically designed experiments.
Taguchi's robust design for quality engineering includes three
steps: system design, parameter design, and tolerance design
(Byrne and Taguchi, 1987). The key element of the approach is
the parameter design step, within which statistical
experimentation is incorporated.
Rather than simply improving or optimizing a response
value, the focus in parameter design is to identify the factor
settings that minimize variation in performance as well as
adjust the mean performance to a desired target. Factors
included in experimentation include control factors and noise
factors. Control factors are those which can be set and held at
specific values, while noise factors are those which cannot be
controlled, e.g., temperature on a shop floor. The evaluation of
mean performance and performance variation is accomplished
through "crossing" two orthogonal arrays (Section 2.1.1).
Control factors are varied according to an inner array, or
"control" array, and for each run of the control array, the noise
factors are varied according to an outer array, or "noise" array.
For each control factor experiment, a response value is obtained
for each noise factor design point, and the mean and variance of
the response (measured across the noise design points) can be
calculated. The performance characteristic used by Taguchi is a
signal-to-noise (S/N) ratio defined in terms of the mean and
variance of the response. Several alternate S/N ratios are
available based on whether lower, higher, or nominal response
values are desired; specific S/N ratio formulations can be found terms, the design variables are factors, and the design objectives
in (Ross, 1988). are responses.) An intelligent search for these solutions most
The Taguchi approach does not explicitly include model commonly relies on some choice of optimization techniques,
building and optimization. Analysis of experimental results is which generate and evaluate many potential solutions in the
used to identify factor effects, plan additional experiments, and path towards improvement. Regardless of the optimization
to set factor values. Comprehensive presentation and technique employed a large number of function evaluations are
application of the Taguchi approach is given in (Phadke, 1989; required, thereby making fast analysis modules a near
Ross, 1988). Issues and techniques presented by Taguchi have imperative.
been used extensively in engineering design, and in many cases When are metamodels useful or appropriate? In the later
incorporated within the traditional RSM for efficient, effective, stages of design, when detailed information about a specific
and robust design, e.g., (Myers and Montgomery, 1995). These solution is available, highly accurate solutions are essential for
applications and associated implications for engineering design measuring system performance. In the early stages of design,
are discussed in the next section. however, the focus is on generating, evaluating, and comparing
potential conceptual configurations. In this stage, a conceptual
design problem is characterized by a large amount of
3 METAMODELING IN THE CONTEXT OF information, often uncertain, that must be managed. To ensure
ENGINEERING DESIGN the identification of a "good" system level conceptual or
How are the metamodeling techniques of the previous preliminary configuration, a comprehensive search is necessary.
section employed in engineering design? They can all be used In this case, the tradeoff between accuracy and efficiency
to produce approximations of existing computer analysis codes, becomes appropriate. The creation of metamodels allows fast
thereby producing fast analysis modules for more efficient analysis, which facilitates both comprehensive and efficient
computation. These metamodeling techniques can also yield design space search, at the expense of a (hopefully slight) loss
insight to the functional relationship between input and output of accuracy.
parameters. Having established this baseline engineering design
"context", in the next section we present a review of several
Given: Factors, responses statistical applications in engineering design, highlighting
general trends and observations. In Section 3.2 we discuss a
few of the more general statistical methods or "tools" which
Large # of YES
Run Screening have been developed for engineering applications, and we
Factors? Experiment conclude with Section 3.3 in which we take a closer look at the
some of the statistical pitfalls associated with the application of
NO Screening statistical techniques to deterministic computer experiments.
Run Modeling Reduce # This then paves the way for Section 4, where we present our
Experiment(s) Factors guidelines and recommendations for the appropriate use of
statistics in computer-oriented design.
Build Predictive
Model ( y )
Model
Building 3.1 Existing Applications in Engineering
Design
Noise YES Build Robustness In this section we provide a survey of several engineering
Factors? Model ( y , y ) applications of Design of Experiments, Response Surface
Methodology, and Taguchi's robust design method. Most of
our examples come from aerospace and mechanical engineering
NO
design applications presented at conferences in recent years; for
Search Design brevity's sake, we summarize our findings in Table 1. Our
Model Space observations follow.
Exercising
• In terms of experimental design, central composite designs
Solutions and D-Optimal designs seem to be the preferred choice among
(improved or robust) aerospace engineers while orthogonal arrays (OAs) are
Figure 3. A Principal Use of Statistics in preferred by many mechanical engineers; very few people
Computer-Oriented Design utilize grid searches and random point generation since they
Where would such models be useful? There is an implicit are less efficient.
process that is common to most of the engineering design • Optimization seems to be the principal driver for aerospace
applications, and this process is illustrated in Figure 3. A applications of DOE and RSM; these types of design
designer's ultimate goal is most often to arrive at improved or applications typically involve the use of computer intensive
robust solutions, which are in essence the values of design analysis and optimization routines, and RSM is a logical
variables that best meet the design objectives. (In statistical choice for increased efficiency.

7 Copyright © 1997 by ASME

• Mechanical engineers are more inclined to use OAs and
Taguchi for robust design instead, working with the signal-
to-noise ratio for parameter and tolerance design. 3.2 Existing Methods and Tools in
• Very few designers actually model Taguchi's loss function Engineering Design
directly (e.g., Beard and Sutherland, 1993); many prefer to In this section we present a few methods and tools
model the response instead. developed specifically for engineering design that are build
• Most applications employ second order response surface around or incorporate the statistical techniques and approaches
models; there are only a few instances were higher order reviewed in Section 2. As evident in Table 1 in the previous
models (e.g., Venter, et al., 1996) and mixed polynomial section, the Taguchi approach and RSM have been widely
models (e.g., Roux, et al., 1996) are employed. applied in engineering design; a review of literature comparing
• When orthogonal arrays are used, special care must be taken these approaches is given first, followed by an overview of
to avoid confounding main effects with interactions, Robust Design Simulation (RDS), Robust Concept
providing the interactions are not assumed to be insignificant Exploration Method (RCEM), NORMAN/DEBORA,
based on prior observations or experience. Some authors take DOE/Opt, and HIDER.
this into account while others do not.
• Most applications utilize a least squares regression analysis 3 . 2 . 1 Taguchi Approach vs. RSM. Both Taguchi
when fitting a model; only a few use stepwise regression, and techniques and Response Surface Methodology have been
this is usually because the model is not second order. applied extensively in engineering design. Many authors have
compared these techniques for different problems. It appears to
Table 1. Cited Engineering Applications of be commonly accepted that the principles associated with
DOE, RSM, and Taguchi Taguchi approach are not only useful but very appropriate for
Experimental
Design
Response
Surface Method
Taguchi
Method
industrial product design; Ramberg, et al. (1991) for example
suggest that "the loss function and the associated robust design
Consider Interactions

philosophy provide fresh insight into the process of optimizing

S/N, Loss, or Resp.
Fractional Factorial

Fit Using:
Optimize/Improve

or improving the simulation's performance". However, two

Robust Design
Order of Eqn.
Grid/Random

specific aspects of the Taguchi approach are commonly

Least Sqr.
D-Optimal

Stepwise

criticized: the experimental design choice (orthogonal arrays,

CCD

OAs

Paper
inner and outer) and the use of the loss function or signal-to-
(Balabanov, et al., 1996) x x 2nd x x noise ratio. It has been argued and demonstrated that the use of
(Bauer and Krebs, 1996) x x x 2nd x x S/N
(Beard and Sutherland, 1993) x x x L
a single experiment combining control and noise factors, a
(Chen, et al., 1996) x x 2nd x R
more traditional RSM approach rather than inner "control" and
(Chi and Bloebaum, 1995) x x R outer "noise" arrays, is more efficient in that it requires fewer
(Engelund, et al., 1993) x x 2nd x experiments (Shoemaker, et al., 1991; Unal, et al., 1994a;
2nd
(Gadallah and ElMaraghy, 1994) x x x x x S/N Welch, et al., 1990). With respect to the loss function or
(Giunta, et al., 1996) x x 2nd x signal-to-noise ratio, the drawbacks of combining response
(Giunta, et al., 1994) x x x x 2nd x
(Healy, et al., 1975) x x 2nd x x
mean and variance into a single measure, a signal-to-noise ratio,
(Hong, et al., 1994) x x 2nd x x S/N
are well-documented in the literature and many authors advocate
(Koch, et al., 1996) x x 2nd x measuring a response directly rather than loss, and separately
(Korngold and Gabriele, 1995) x 2nd x tracking mean and variance (Chen, et al., 1995; Ramberg, et
2nd
(Li, et al., 1994) x x x S/N al., 1991; Welch, et al., 1990). Shoemaker et al. (1991),
(Mavris, et al., 1996) x x 2nd x R however, warn that "A potential drawback of the response-
(Mavris, et al., 1995) x x 2nd x R
(Roux, et al., 1996) x x Mix x
model approach that is depends more critically than the loss-
(Rowell, et al., 1996) x x x 2nd x x x model approach on how well the model fits."
(Stanley, et al., 1992) x x x R Given the general acceptance of the Taguchi principles and
(Sundaresan, et al., 1993) x x x R of robust design, with these criticisms of these techniques,
2nd
(Unal, et al., 1996) x x x many advocate a combined Taguchi-RSM approach, or simply
(Unal, et al., 1992) x x x R applying more traditional RSM techniques within the
(Unal, et al., 1994a) x x 2nd x
2nd
framework defined by Taguchi (Lucas, 1994; Myers, et al.,
(Unal, et al., 1994b) x x x x x R
(Venter, et al., 1996) x x 4th x 1989; Myers and Montgomery, 1995; Ramberg, et al., 1991).
(Yu and Ishii, 1993) x x 2nd x x R It is the belief of the authors of this paper that both orthogonal,
inner and outer arrays, and single, composite experiments each
have advantages and disadvantages and appropriate uses, and that
In addition to these specific engineering design applications
separate observation of mean and variance leads to useful
of DOE, RSM, and Taguchi, many methods and "tools" have
problem insight. Regardless, the core principles of both
been developed for more general design application. We
Taguchi and RSM have provided a foundation for the specific
examine several of these methods in the next section,
design methods discussed in Section 3.2.2.
highlighting various applications of each.

8 Copyright © 1997 by ASME


3 . 2 . 2 An Overview of Existing Methods. Robust
Design Simulation (RDS), combines the design of experiments
and response surface metamodeling aspects of RSM with Monte
Carlo simulation to facilitate robust design (Mavris, et al.,
1995). The general implementation of this approach consists
of three sequential experimentation stages: screening, second-
order experimentation for model building, and a combined
experimentation/Monte Carlo simulation stage for
incorporating noise. The objective of RDS is to achieve robust
system designs: those that minimize performance variation due
to uncontrollable noise, while achieving as closely as possible
performance target values. RDS is presented and applied to the
design of a High Speed Civil Transport aircraft in (Mavris, et
al., 1996; Mavris, et al., 1995).
The Robust Concept Exploration Method (RCEM) has
been developed to facilitate quick evaluation of different design
alternatives and generation of top-level design specifications
with quality considerations in the early stages of design (Chen,
et al., 1996a). Foundational to the RCEM is the integration of
robust design principles, design of experiments and response
surface methodology, and the compromise Decision Support
Problem (DSP) (a multiobjective decision model which is a
hybrid formulation based on Mathematical Programming and
Goal Programming). This RCEM is applied to a High Speed
Civil Transport (HSCT) in (Chen, et al., 1996a) The authors of
this paper have also applied RCEM to the design of a General
Aviation Aircraft (Simpson, et al., 1996), to a manufacturing
simulation problem (Peplinski, et al., 1996), and to the design
of a turbine lift engine (Koch, et al., 1996).
NORMAN/DEBORA is a TCAD (Technology Computer
Aided Design) system created by a Belgian company, IMEC,
incorporating advanced sequential design of experiments (DOE)
and Response Surface Methodology (RSM) techniques to aid in
engineering optimization and robust design problems.
Advanced DOE/RSM techniques incorporated within this
system include a novel design of experiments concept, Target
Oriented Design (TOD), a unique parameter transformation
technique, RATIOFIND, and a non-linearly constrained
optimizer, DEBORA (Booth, et al., 1991). The
NORMAN/DEBORA system is demonstrated for semiconductor
integrated circuit (IC) technology design and optimization in
(Cartuyvels and Dupas, 1993).
DOE/Opt is a prototype computer system for performing
design of experiments, response surface modeling, and
optimization (Boning and Mozumder, 1994). Applications
described for DOE/Opt are in the semiconductor process and
device design area and include process/device optimization,
simulator tuning, process control recipe generation, and design
for manufacturability.
Hierarchical and Interactive Decision Refinement (HIDER)
is a methodology for concept exploration in the early stages of
design that integrates simulation, optimization and machine
learning to support design decision making (Reddy, 1996).
HIDER uses the Adaptive Interactive Modeling System (AIMS)
(Lu and Tcheng, 1991) to decompose the design space using
distance-based, population-based, and hyperplane-based
9 Copyright © 1997 by ASME
algorithms. HIDER and AIMS have been applied to the design
of a cutting process (Lu and Tcheng, 1991), a diesel engine
(Yerramareddy, et al., 1992), and a wheel loader (Reddy, 1996).
The Concurrent SubSpace Optimization (CSSO)
methodology developed in (Renaud and Gabriele, 1993; Renaud
and Gabriele, 1994; Renaud and Gabrielle, 1991) utilizes design
data generated during concurrent subspace optimizations to
develop response surface approximations of the design space.
Optimization of these response surface representations of the
design space form the basis of the coordination procedure
developed in (Renaud and Gabriele, 1993; Renaud and Gabriele,
1994; Renaud and Gabrielle, 1991). The data generated by the
subspace optimizers is not uniformly centered about the current
design as in CCD or other sampling strategies, but instead
follows the descent path of the subspace optimizers. In
(Renaud and Gabriele, 1993; Renaud and Gabriele, 1994;
Renaud and Gabrielle, 1991) interpolating polynomial response
surfaces are constructed which have either a first order or second
order basis for use in the CSSO coordination procedure. In
(Wujek, et al., 1996; Wujek and Renaud, 1997) a modified
decomposition strategy is employed in developing quadratic
response surfaces for use in the CSSO coordination procedure.
Finally, in (Sellar and Batill, 1996; Sellar, et al., 1996a; Sellar,
et al., 1996b) artificial neural network response surfaces are
developed for use in the CSSO coordination procedure.
Additional approaches incorporating statistical techniques
in design exist; only a few have been included in this section.
Our focus is not on the methods, but on the appropriateness of
the statistical techniques: many of the examples to which these
methods have been applied employ deterministic computer
experiments, to which the application of statistical techniques
is questionable. The associated issues are discussed in the next
section.
3.3 A Closer Look at Experimental Design for
Deterministic Computer Experiments
Since engineering design commonly involves exercising
deterministic computer codes, the use of statistical
experimentation in creating metamodels of these codes warrants
a closer look. Given a response of interest, y, and a vector of
independent factors x thought to influence y, the relationship
between y and x (Equation 3) includes the random error term ε .
To apply least squares regression, the error values for each data
point are assumed to have identical and independent normal
distributions with means of zero and standard deviations of σ, or
ε i i.i.d. N(0, σ2). This scenario is shown in Figure 4(a). The
least squares estimator then minimizes the sum of the squared
differences between the actual data points and the values
predicted by the model. It is acceptable if no data point actually
lies on the predicted model, because it is assumed that the
model "smoothes out" the random error. Of course, it is likely
that the regression model itself is only an approximation of the
true behavior between x and y, so that the final relationship is
y = g(x) + ε bias + ε random (7)
where ε bias represents the error of approximation. However, for
deterministic computer experiments as illustrated in Figure
4(b), ε random has mean zero and variance zero, so after model
fitting we have the relationship
y = g(x) + ε bias
The deterministic case of Equation 7 conflicts sharply with
the methods of least squares regression. First, unless ε bias is
i.i.d. N(0,σ2) then the assumptions for statistical inference from
least squares regression are violated. Even further, because there
is no random error there is little justification for smoothing
across data points; instead the model should hit each point
exactly and interpolate between them as shown in Figure 4(b).
Finally, most standard tests for model and parameter
significance are based on computations using ε random (the mean
squared error) and are therefore impossible to compute. These
observations are supported by literature in the statistics
community; as Sacks, et al. (1989b) carefully point out,
because deterministic computer experiments lack random error:
• response surface model adequacy is determined solely by
systematic bias,
• the usual measures of uncertainty derived from least-squares
residuals have no obvious statistical meaning (deterministic
measures of uncertainty exist, e.g., max |y(x) - y(x)| over x,
but they may be very difficult to compute), and
• the classical notions of experimental blocking, replication
and randomization are irrelevant.
Similarly, according to Welch and his co-authors (1990),
current methods for the design and analysis of physical
experiments (e.g., Box and Draper, 1987; Box, et al., 1978) are
not ideal for complex, deterministic computer models. "In the
presence of systematic error rather than random error, statistical
testing is inappropriate" (Welch, et al., 1990). Finally, a
discussion of how the model should interpolate the observations
can be found in (Sacks, et al., 1989a).
y y
ε ~ N(0,σ2 )
(yi - yi )
L.S.E.=∑ (y i - y i)
2 ∑ (y i - y i)
y = g(x) = second order
least squares fit y = g(x) = spline fit
x x
(a) Non-Deterministic Case (b) Deterministic Case
Figure 4. Deterministic and Non-Deterministic
Curve Fitting
So where can these methods go wrong? Unfortunately it is
very easy to unthinkingly classify the ε bias term from a
deterministic model fit as ε random and then proceed with standard
statistical testing. Several authors have reported statistical
measures such as the F-statistics and root MSE for verification
of model adequacy (e.g., Healy, et al., 1975; Koch, et al., 1996;
Unal, et al., 1994b; Venter, et al., 1996; Welch, et al., 1990).
These measures have no statistical meaning since they assume
the observations include an error term which has mean of zero
(8)
and a non-zero standard deviation. Consequently, the use of
stepwise regression for polynomial model fitting is not
appropriate since it utilizes F-statistic values when
adding/removing model parameters.
R-Squared (when defined as the model sum of squares
divided by the total sum of squares) is really the only measure
for verifying model adequacy for deterministic computer
experiments, and often this measure not sufficient (a high R-
Squared value can be deceiving). Consequently, confirmation
testing of model validity through use of additional (different)
data points becomes essential. Residual plots may also be
extremely helpful when verifying model adequacy for
identifying trends in data, examining outliers, etc.
Some researchers (e.g., Giunta, et al., 1996; Giunta, et al.,
1994; Venter, et al., 1996) have also employed metamodeling
techniques such as RSM for modeling deterministic computer
experiments which contain numerical noise. This numerical
noise is used as a surrogate for random error, thus allowing the
standard least-squares approach to be applied. However, the
assumption of equating numerical noise to random error is
questionable, and the appropriateness of their approach warrants
further investigation.
Given the potential problems of applying least-squares
regression to deterministic applications, the trade-off then
becomes one of appropriateness vs. practicality. If a response
surface is created to model data from a deterministic computer
code using experimental design and least-squares fitting, and if
it provides very good agreement between predicted and actual
values (a situation that often occurs when the computer model
is "well-behaved"), then there is no practical reason to discard it.
It should be used, albeit with caution. However, it is important
ε=0 to be aware of the fundamental assumptions of the statistical
techniques employed, so that we avoid incorrect or misleading
statements about model significance. In the next section we
provide several guidelines pertaining to the appropriate use of
statistical methods and metamodeling of deterministic computer
experiments.
2
4 GUIDELINES AND RECOMMENDATIONS
=0
How can a design engineer efficiently apply the
metamodeling tools of Section 2 while avoiding the pitfalls
described in Section 3.3? There are two ways to answer this
question: from the bottom up (tools applications) and from
the top down (motives tools). The bottom-up approach is
presented in Section 4.1, the top-down in Section 4.2.
4 . 1 Evaluation of Metamodeling Techniques
There are two components to this section. The first is an
evaluation of the four metamodeling techniques described in
Section 2.2. The second component is choosing an
experimental design which has more direct applicability to
response surface methods but also applies to the remaining
metamodeling techniques. The explorations of which designs
are most appropriate are, as yet, open research areas.
10 Copyright © 1997 by ASME

4 . 1 . 1 Evaluation of Model Choice and Model
Fitting Alternatives. In Section 2.2 we described four
metamodeling techniques; here we present some brief guidelines
for their evaluation.
Response Surfaces: primarily intended for applications
with random error, though deterministic applications are not
necessarily incorrect. Building higher-order models with greater
than ten factors becomes difficult. (Building regression models
is tractable for fifteen to twenty factors, but actually obtaining
the data is the limiting factor.) It is the most well-established
metamodeling technique, and is probably the easiest to use.
Neural Networks: nonlinear regression approach best
suited for deterministic (non-noisy) applications. They can be a
nearly universal approximator, and so are able to handle highly
nonlinear or extremely large (~ 10,000 parameters) problems, as
long as enough data points are provided and enough computer
time is allocated to fit the model. Neural networks therefore
lend themselves to large problems, so that "Now the procedure
is to toss the data directly into the NN software, use tens of
thousands of parameters in the fit, let the workstation run 2-3
weeks grinding away doing the gradient descent, and voilá, out
comes the result" (Cheng and Titterington, 1994).
Inductive Learning: modeling technique most appropriate
where the input and output factors are primarily discrete-valued
or can be grouped into categories. The predictive model, in the
form of condition/action rules or a decision tree, may perhaps
lack the mathematical insight desired for engineering design
applications, but it is ideal for applications such as process
control and diagnosis.
Kriging: interpolation method created to handle
deterministic data. It is not based on the same assumptions as
standard regression analysis and appears to be better suited for
deterministic computer experiments, as long as the number of
factors is small (≤10, according to (Welch, et al., 1992)).
Kriging is extremely flexible due to the wide scope of
correlation functions R(w,x) which may be chosen (Laslett,
1994); it can provide an exact interpolation of the data or
"smooth the data" (Cressie, 1988). Additional advantages
include (Welch, et al., 1992):
• it provides a basis, via the likelihood, for a stepwise
algorithm to determine the important factors;
• it is flexible, allowing nonlinear and interactions effects to
emerge without explicitly modeling such effects;
• the same data can be used for screening and building the
predictor, so expensive runs are efficiently used.
The complexity of the method, however, coupled with a lack of
computer software support, may make the learning curve of this
technique prohibitive in the near term.
4.1.2 Evaluation of Experimental Designs.
There are many voices in the discussion of the relative merits of
different experimental designs, and it is therefore unlikely that
we have captured them all. However, broad trends appear, and
in general we have found that central composite designs
consistently perform and compare favorably. Other second order
designs worth investigating are the hexagonal (Montgomery and
Evans, 1975) and Box-Behnken and hybrid designs (Giovannitti-
11
Jensen and Myers, 1989). For comparisons of different design
types, refer to the following.
• Lucas (1976) compares CCD, Box-Behnken, uniform shell,
Hoke, Pesotchinsky, and Box-Draper designs, using the D-
efficiency and G-efficiency statistics.
• Myers and Montgomery (1995) provide a comprehensive
review of many experimental designs for fitting second order
response surfaces. They conclude that hybrid designs are a
good idea, if the unusual levels for the design variables can be
tolerated. For computer experiments, this is unlikely to be a
problem.
• Montgomery and Evans (1975) compare six second-order
designs: a) 3^2 factorial, b) rotatable orthogonal CCD, c)
rotatable uniform precision CCD, d) rotatable minimum bias
CCD, e) rotatable orthogonal hexagon, and f) rotatable
uniform precision hexagon. The criteria used for comparison
are average response achievement and distance from true
optimum.
• Lucas (1974) compares symmetric and asymmetric composite
and smallest composite designs for different numbers of
factors using the |X'X| criterion, and the D-efficiency and G-
efficiency statistics.
• Giovannitti-Jensen and Myers (1989) discuss several first and
second order designs, observing that the performance of
rotatable CCD and Box-Behnken designs are nearly identical.
They note that "hybrid designs appear to be very promising."
4.2 Initial Recommendations for Metamodeling
Uses
The majority of metamodeling applications are built around
the creation of low-order polynomial metamodels using central
composite designs and least squares regression. The nearly
universal popularity of this approach is due, at least in part, to
the maturity and well-established nature of response surface
methodology, the ease and simplicity of the method, and easily
accessible software support tools. However, the RSM approach
starts to break down when there are a large (> 10) number of
factors to be modeled, or when the relationship to be modeled is
highly nonlinear. And as we have shown in Section 3.3, there
are also dangers of applying the RSM approach blindly to
deterministic applications. The alternative approaches to
metamodeling described in section 4.1.1 address these
limitations, each in their own way. Our recommendations are
summarized as follows:
• If a large number of factors must be modeled in a
deterministic application, then neural networks may be the
best metamodeling choice despite their tendency to be
computationally expensive to create.
• If the underlying function to be modeled is deterministic
and highly nonlinear in a small (≤ 10) number of factors,
then kriging may be the best choice despite its statistical
complexity.
• However, in deterministic applications that have a small
number of factors and are fairly well behaved, we present
another option for exploration: applying the standard RSM
approach, augmented with a Taguchi outer array.
Copyright © 1997 by ASME
 R S M / O A approach: The fundamental problem with
applying least-squares regression to deterministic applications is
the absence of ε random in Equation 8. However, if some of the
input parameters for the computer code can be classified as
noise factors, and if these noise factors are varied across an outer
array for each setting of the control factors, then in essence a
series of replications are generated that can be used to
approximate ε random . This approximation can be justified if it is
reasonable to assume that, were the experiments performed on
an actual physical system, that the random error observed would
have been due to fluctuations in the same noise factors that are
varied in the computer code. If this assumption is reasonable,
then statistical testing of model and parameter significance can
be performed, and models of both mean response and variability
can be created from the same experimental design. We are
currently investigating this approach in our ongoing work.
5 SUMMARY AND CONCLUSIONS
In this paper we have surveyed some applications of
statistics to engineering design and have developed the idea of
metamodeling in the engineering design context, as discussed in
Section 1 and as shown in Figure 3. Metamodeling is an active
research area in more than just the statistics community, to
which our reference list is mute testimonial. However,
applying these techniques to deterministic applications in
engineering design can lead to problems, as pointed out in
Section 3.1 and discussed in Section 3.3. We present the set of
our recommendations for applying metamodeling techniques in
Section 4, but this set is by no means complete.
Comprehensive comparisons of these techniques have yet to be
performed, and the difficulties of large problem size and
nonlinearity are ever-present.
In particular, an issue of interest to us in our continuing
research is the problem of size, or scaling. As the size of a
problem (number of factors) increases, the cost associated with
creating metamodels can begin to outweigh the associated
gains. In addition, often the reduction in factors resulting from
screening is still not sufficient to bring the problem to a
manageable size. This problem is compounded by the multiple
response problem: complex engineering design problems
invariably include multiple measure of performance, or
responses, that must be modeled. The screening process also
breaks down when attempting to select the most important
factors for more than one response, as each response often
requires different important factors. The general question that
arises from these problems, then, is how can these
experimentation and metamodeling techniques be used
efficiently for larger problems (problems with greater than 10
factors after screening)? One approach that we are pursuing is
problem partitioning or decomposition. A significant literature
base exist in techniques for breaking a problem into smaller
problems; a good review of such methods can be found in
(Lewis and Mistree, 1997). Using these techniques, a complex
problem may be broken down into smaller problems that allow
efficient experimentation and metamodeling, which again will
lead to comprehensive and efficient exploration of a design
12
space. We hope to have preliminary results in this area in the
near future.
ACKNOWLEDGMENTS
Timothy Simpson and Jesse Peplinski have been supported
by NSF Graduate Research Fellowships, and Patrick Koch has
been supported by a Gwaltney manufacturing fellowship. In
addition, financial support from NSF Grants DMI-96-12365 and
DMI-96-12327, and NASA Grant NGT-51102 is gratefully
acknowledged.
REFERENCES
Balabanov, V., Kaufman, M., Knill, D. L., Haim, D., Golovidov,
O. and Giunta, A. A., 1996, "Dependence of Optimal Structural
Weight on Aerodynamic Shape for a High Speed Civil
Transport," 6th AIAA/USAF/NASA/ISSMO Symposium on
Multidisciplinary Analysis and Optimization, Bellevue, WA,
AIAA, Inc., Vol. 1, pp. 599-612.
Bauer, D. and Krebs, R., 1996, "Optimization of Deep Drawing
Processes Using Statistical Design of Experiments," 22nd
Design Automation Conference (McCarthy, J. M., ed.), Irvine,
CA, ASME, Paper No. 96-DETC/DAC-1446.
Beard, J. E. and Sutherland, J. W., 1993, "Robust Suspension
System Design," Advances in Design Automation (Gilmore, B.
J., Hoeltzel, D. A., et al., eds.), Albuquerque, NM, ASME, Vol.
65-1, pp. 387-395.
Biles, W. E., 1984, "Design of Simulation Experiments," 1984
Winter Simulation Conference (WSC) (Sheppard, S., Pooch, U.,
et al., eds.), Dallas, TX, IEEE, pp. 99-104.
Boning, D. S. and Mozumder, P. K., 1994, "DOE/Opt: A System for
Design of Experiments, Response Surface Modeling, and
Optiminzation Using Process and Device Simulation," IEEE
Trans. on Semiconductor Manufacturing, Vol. 7(2), pp. 233-244.
Booth, R., Dupas, L., Cartuyvels, R. and De Meyer, K., 1991, "New
Technique to Improve Response Surface Method Model Fitting
Accuracy for Process Technology and Device Design
Optimization," INternational Workshop on VLSI Process and
Device Modeling (VPAD), Oiso Kanagawa, Japan.
Box, G. E. P. and Draper, N. R., 1987, Empirical Model-Building
and Response Surfaces, John Wiley & Sons, New York.
Box, G. E. P., Hunter, W. G. and Hunter, J. S., 1978, Statistics for
Experimenters, John Wiley & Sons, Inc., New York.
Byrne, D. M. and Taguchi, S., 1987, "The Taguchi Approach to
Parameter Design," 40th Annual Quality Congress Transactions,
Milwaukee, Wisconsin, American Society for Quality Control.
Cartuyvels, R. and Dupas, L., 1993, "NORMAN/DEBORA: a
Powerful CAD-Integrating Automatic Sequencing System
Including Advanced DOE/RSM Techniques for Various
Engineering Optimization Problems," JSPE-IFIP WG 5.3
DIISM'93 Workshop (Design of Information Infrastructure
Systems for Manufacturing), Tokyo, Japan.
Chen, W., Allen, J. K., Mavris, D. and Mistree, F., 1996a, "A
Concept Exploration Method for Determining Robust Top-Level
Specifications," Engr. Opt., Vol. 26, pp. 137-158.
Chen, W., Allen, J. K., Mistree, F. and Tsui, K.-L., 1995,
"Integration of Response Surface Methods with the Compromise
Decision Support Problem in Developing a General Robust
Design Procedure," ASME Design Automation Conference
(Azarm, S., et al., ed.), Boston, Massachusetts, ASME, New
York, DE-Vol. 82-2, pp. 485-492.
Copyright © 1997 by ASME
Chen, W., Simpson, T. W., Allen, J. K. and Mistree, F., 1996b,
"Use of Design Capability Indices to Satisfy a Ranged Set of
Design Requirements," 1996 ASME Design Automation
Conference (Dutta, D., ed.), Irvine, CA, ASME, New York, Paper
No. 96-DETC/DAC-1090.
Cheng, B. and Titterington, D. M., 1994, "Neural Networks: A
Review from a Statistical Perspective," Statistical Science, Vol.
9(1), pp. 2-54.
Chi, H.-W. and Bloebaum, C. L., 1995, "Mixed Variable
Optimization Using Taguchi's Orthogonal Arrays," 21st Annual
Advances in Design Automation Conference (Azarm, S., Dutta,
D., et al., eds.), Boston, MA, ASME, Vol. 68-1, pp. 501-508.
Cressie, N., 1988, "Spatial Prediction and Ordinary Kriging,"
Mathematical Geology, Vol. 20(4), pp. 405-421.
Cressie, N., 1989, "Geostatistics," The American Statistician, Vol.
43(4), pp. 197-202.
Draper, N. R. and Lin, D. K. J., 1990a, "Connections Between Two-
Level Designs of Resolutions III and V," Technometrics, Vol.
32(3), pp. 283-288.
Draper, N. R. and Lin, D. K. J., 1990b, "Small Response-Surface
Designs," Technometrics, Vol. 32(2), pp. 187-194.
Engelund, W. C., Douglas, O. S., Lepsch, R. A., McMillian, M. M.
and Unal, R., 1993, "Aerodynamic Configuration Design Using
Resopnse Surface Methodology Analysis," AIAA Aircraft
Design, Systems and Operations Meeting, Monterey, CA, Paper
No. AIAA 93-3967.
Evans, B. and Fisher, D., 1994, "Overcoming Process Delays with
Decision Tree Induction," IEEE Expert, Vol. 9, pp. 60-66.
Gadallah, M. H. and ElMaraghy, H. A., 1994, "The Tolerance
Optimization Problem Using a System of Experimental Design,"
Advances in Design Automation Conference, Minneapolis, MN,
ASME, Vol. 69-1, pp. 251-265.
Giovannitti-Jensen, A. and Myers, R. H., 1989, "Graphical
Assessment of the Prediction Capability of Response Surface
Designs," Technometrics, Vol. 31(2), pp. 159-171.
Giunta, A. A., Balabanov, V., Haim, D., Grossman, B., Mason, W.
H. and Watson, L. T., 1996, "Wing Design for a High-Speed
Civil Transport Using a Design of Experiments Methodology,"
6th AIAA/USAF/NASA/ISSMO Symposium on Multidisci-
plinary Analysis and Optimization, Bellevue, WA, AIAA, Inc.,
Vol. 1, pp. 168-183.
Giunta, A. A., Dudley, J. M., Narducci, R., Grossman, B., Haftka,
R. T., Mason, W. H. and Watson, L. T., 1994, "Noisy
Aerodynamic Response and Smooth Approximations in HSCT
Design," 5th AIAA/USAF/NASA/ISSMOSymposium on Multi-
disciplinary Analysis and Optimization, Panama City, FL, Vol.
2, pp. 1117-1128.
Healy, M. J., Kowalik, J. S. and Ramsay, J. W., 1975, "Airplane
Engine Selection by Optimization of Surface Fit
Approximations," Journal of Aircraft, Vol. 12(7), pp. 593-599.
Hong, J., Dagli, C. H. and Ragsdell, K. M., 1994, "Artificial Neural
Network and the Taguchi Method Application for Robust
Wheatstone Bridge Design," Advances in Design Automation
(Gilmore, B. J., Hoeltzel, D. A., et al., eds.), Minneapolis, MN,
ASME, Vol. 69-2, 37-41.
Hunter, J. S., 1985, "Statistical Design Applied to Product
Design," Jrnl. of Quality Technology, Vol. 17(4), pp. 210-221.
Hunter, J. S. and Naylor, T. H., 1970, "Experimental Designs for
Computer Simulation Experiments," Management Science, Vol.
16(7), pp. 422-434.
Journel, A. G., 1986, "Geostatistics: Models and Tools for the
Earth Sciences," Math. Geol., Vol. 18(1), pp. 119-140.
13
Kleijnen, J. P. C., 1987, Statistical Tools for Simulation
Practitioners, Marcel Dekker, New York.
Koch, P. N., Allen, J. K. and Mistree, F., 1996, "Robust Concept
Exploration for Configuring Turbine Propulsion Systems," 1996
ASME Design and Automation Conference (Dutta, D., ed.),
Irvine, CA, Paper No. 96-DETC/DAC-1285.
Korngold, J. C. and Gabriele, G. A., 1995, "Multidisciplinary
Analysis and Optimization of Discrete Problems Using Response
Surface Methods," 21st Annual Advances in Design Automation
Conference, Boston, MA, ASME, Vol. 82-1, pp. 173-180.
Langley, P. and Simon, H. A., 1995, "Applications of Machine
Learning and Rule Induction," Communications of the ACM,
Vol. 38(11), pp. 55-64.
Laslett, G. M., 1994, "Kriging and Splines: An Empirical
Comparison of Their Predictive Performance in Some
Applications," Journal of the American Statistical Association,
Vol. 89(42), pp. 391-400.
Leech, W. J., 1986, "A Rule Based Process Control Method With
Feedback," Advances In Instrumentation, Vol. 41, pp. 169-175.
Lewis, K. and Mistree, F., 1997, "The Other Side of
Multidisciplinary Design Optimization: Accomodating a
Multiobjective, Uncertain and Non-Deterministic World,"
Engineering Optimization, in press.
Li, C.-C., Sheu, T.-S. and Lee, Y.-C., 1994, "Robustness
Consideration in Modelling a Prototype by Quality
Engineering," Design Theory and Methodology Conference
(Hight, T. K. and Mistree, F., eds.), Minneapolis, MN, ASME,
Vol. 68, pp. 55-62.
Lu, S. C.-Y. and Tcheng, D. K., 1991, "Building Layered Models to
Support Engineering Decision Making: A Machine Learning
Approach," Journal of Engr. for Industry, Vol. 113, pp. 1-9.
Lucas, J. M., 1974, "Optimum Composite Designs,"
Technometrics, Vol. 16(4), pp. 561-567.
Lucas, J. M., 1976, "Which Response Surface Design is Best,"
Technometrics, Vol. 18(4), pp. 411-417.
Lucas, J. M., 1994, "Using Response Surface Methodology to
Achieve a Robust Process," Journal of Quality Technology, Vol.
26(4), pp. 248-260.
Matheron, G., 1963, "Principles of Geostatistics," Economic
Geology, Vol. 58, pp. 1246-1266.
Mavris, D., Bandte, O. and Schrage, D., 1996, "Application of
Probabilistic Methods for the Determination of an Economically
Robust HSCT Configuration," 6th AIAA/USAF/NASA/ISSMO
Symposium on Multidisciplinary Analysis and Optimization,
Bellevue, WA, AIAA, Inc., Vol. 2, pp. 968-978.
Mavris, D. N., Bandte, O. and Schrage, D. P., 1995, "Economic
Uncertainty Assessment of an HSCT Using a Combined Design
of Experiments/Monte Carlo Simulation Approach," 17th
Annual Conference of International Society of Parametric
Analysts, San Diego, CA.
Mitchell, T., Morris, M. and Ylvisaker, D., 1988, "Existence of
Smoothed Stationary Processes on an Interval," Working Paper.
Montgomery, D. C., 1997, Design and Analysis of Experiments,
Fourth Edition, John Wiley & Sons, New York.
Montgomery, D. C. and Evans, D. M., Jr., 1975, "Second-Order
Response Surface Designs in Computer Simulation," Simulation,
Vol. 29(6), pp. 169-178.
Myers, R. H., Khuri, A. I. and Carter, W. H., 1989, "Response
Surface Methodology: 1966-1988," Technometrics, Vol. 31(2),
pp. 137-157.
Myers, R. H. and Montgomery, D. C., 1995, Response Surface
Methodology: Process and Product Optimization Using
Designed Experiments, John Wiley & Sons, New York.
Copyright © 1997 by ASME
Peplinski, J. D., Allen, J. K. and Mistree, F., 1996, "Integrating
Product Design with Manufacturing Process Design Using the
Robust Concept Exploration Method," 1996 ASME Design
Theory and Methodology Conference (Wood, K., ed.), Irvine,
CA, Paper No. 96-DETC/DTM-1502.
Phadke, M. S., 1989, Quality Engineering using Robust Design,
Prentice Hall, Englewood Cliffs, New Jersey.
Ramberg, J. S., Sanchez, S. M., Sanchez, P. J. and Hollick, L. J.,
1991, "Designing Simulation Experiments: Taguchi Methods
and Response Suface Metamodels," Proceedings of the 1991
Winter Simulation Conference (WSC) (Nelson, B. L., Kelton, W.
D., et al., eds.), Phoenix, AZ, IEEE, pp. 167-176.
Reddy, S. Y., 1996, "HIDER: A Methodology for Early-Stage
Exploration of Design Space," 1996 ASME Design Automation
Conference (Dutta, D., ed.), Irvine, CA, ASME, New York, Paper
No. 96-DETC/DAC-1089.
Renaud, J. E. and Gabriele, G. A., 1993, "Improved Coordination in
Non-Hierarchic System Optimization," AIAA Journal, Vol.
31(12), pp. 2367-2373.
Renaud, J. E. and Gabriele, G. A., 1994, "Approximation in
Nonhierarchic System Optimization," AIAA Journal, Vol. 32(1),
pp. 198-205.
Renaud, J. E. and Gabrielle, G. A., 1991, "Sequential Global
Approximation in Non-Hierarchic System Decomposition and
Optimization," Advances in Design Automation, Design
Automation and Design Optimization, Miami, FL, ASME, Vol.
32-1, pp. 191-200.
Ross, P. J., 1988, Taguchi Techniques for Qualtiy Engineering,
McGraw-Hill, New York, NY.
Roux, W. J., Stander, N. and Haftka, R. T., 1996, "Response
Surface Approximations for Structural Optimization," 6th
AIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary Analysis
and Optimization, Bellevue, WA, AIAA, Inc., Vol. 1, pp. 565-
578.
Rowell, L. F., Braun, R. D., Olds, J. R. and Unal, R., 1996, "Recent
Experiences in Multidisciplinary Conceptual Design
Optimization for Launch Vehicles," 6th AIAA/USAF/NASA/
ISSMO Symposium on Multidisciplinary Analysis and
Optimization, Bellevue, WA, AIAA, Inc., Vol. 1, pp. 642-654.
Rumelhart, D. E., Widrow, B. and Lehr, M. A., 1994, "The Basic
Ideas in Neural Networks," Communications of the ACM, Vol.
37(3), pp. 87-92.
Sacks, J., Schiller, S. B. and Welch, W. J., 1989a, "Designs for
Computer Experiments," Technometrics, Vol. 31(1), pp. 41-47.
Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P., 1989b,
"Design and Analysis of Computer Experiments," Statistical
Science, Vol. 4(4), pp. 409-435.
Sellar, R. S. and Batill, S. M., 1996, "Concurrent Subspace
Optimization Using Gradient-Enhanced Neural Network
Approximations," 6th AIAA/NASA/USAF Multidisciplinary
Analysis & Optimization Symposium, Bellevue, WA, AIAA,
Vol. 1(of 2), pp. 319-330.
Sellar, R. S., Batill, S. M. and Renaud, J. E., 1996a, "A Neual
Network-Based, Concurrent Subspace Optimization Approach to
Multidisciplinary Design Optimization," 34th AIAA Aerospace
Sciences Meeting, Reno, NV, AIAA, AIAA 96-0714.
Sellar, R. S., Stelmack, M. A., Batill, S. M. and Renaud, J. E.,
1996b, "Multidisciplinary Design and Analysis of an Integrated
Aeroelastic/Propulsive System," 37th AIAA/ASME/ASCE/
AHS/ASC Structures, Structural Dynamics, and Materials
Conference, Salt Lake City, UT, AIAA, pp. 583-593.
14
View publication stats
Shoemaker, A. C., Tsui, K. L. and Wu, J., 1991, "Economical
Experimentation Methods for Robust Design," Technometrics,
Vol. 33(4), pp. 415-427.
Stanley, D. O., Unal, R. and Joyner, C. R., 1992, "Application of
Taguchi Methods to Propulsion System Optimization for SSTO
Vehicles," Spacecraft and Rockets, Vol. 29(4), pp. 453-459.
Sundaresan, S., Ishii, K. and Houser, D. R., 1993, "A Robust
Optimization Procedure with Variations on Design Variables and
Constraints," Advances in Design Automation, Albuquerque,
NM, ASME, Vol. 65-1, pp. 379-386.
Unal, R., Lepsch, R. A., Engelund, W. and Stanley, D. O., 1996,
"Approximation Model Building and Multidisciplinary Design
Optimization Using Response Surface Methods," 6th
AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary
Analysis and Optimization, Bellevue, WA, AIAA, Inc., Vol. 1,
pp. 592-597.
Unal, R., Stanley, D. O., Engelund, W. and Lepsch, R., 1994a,
"Design for Quality using Response Surface Methods: An
Alternative to Taguchi's Parameter Design Approach,"
Engineering Management Journal, Vol. 6(3), pp. 40-48.
Unal, R., Stanley, D. O. and Joyner, C. R., 1992, "The Role of
Statistically Designed Experiments in Conceptual Aerospace
Vehicle Design," 1992 IEEE International Engineering
Management Conference, Eatontown, NJ, ASEM, pp. 211-214.
Unal, R., Stanley, D. O. and Joyner, C. R., 1994b, "Parametric
Model Building and Design Optimization Using Reponse Surface
Methods," Journal of Parametrics, Vol. 14(1), pp. 81-96.
Unal, R., Wu, K. C. and Stanley, D. O., 1994c, "A Study of Design
of Experiments Methods for Design Optimization and Quality
Improvement," 15th Annual Conference of the ASEM, ASEM.
Venter, G., Haftka, R. T. and Starnes, J. H., Jr., 1996,
"Construction of Response Sufaces for Design Optimization
Applications," 6th AIAA/USAF/NASA/ISSMO Symposium on
Multidisciplinary Analysis and Optimization, Bellevue, WA,
AIAA, Inc., Vol. 1, pp. 548-564.
Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J.
and Morris, M. D., 1992, "Screening, Predicting, and Computer
Experiments," Technometrics, Vol. 34(1), pp. 15-25.
Welch, W. J., Yu, W. K., Kang, S. M. and Sacks, J., 1990,
"Computer Experiments for Quality Control by Parameter
Design," Journal of Quality Technology, Vol. 22(1), pp. 15-22.
Wujek, B., Renaud, J. E., Batill, S. M. and Brockman, J. B., 1996,
"Concurrent Subspace Optimization Using Design Variable
Sharing in a Distributed Computing Environment," Concurrent
Engineering: Research and Appl., Vol. 4(4), pp. 361-378.
Wujek, B. A. and Renaud, J. E., 1997, "A Concurrent Engineering
Approach for Multidisciplinary Design in a Distributed
Computing Environment," Proceedings in Applied Mathematics,
Philadelphia, PA, SIAM, Vol. 80, pp. 189-208.
Yerramareddy, S., Tcheng, D. K., Lu, S. C.-Y. and Assanis, D. N.,
1992, "Creating and Using Models for Engineering Design,"
IEEE Expert, Vol. 7(3), pp. 52-59.
Yu, J.-C. and Ishii, K., 1993, "A Robust Optimization Method for
Systems with Significant Nonlinear Effects," Advances in
Design Automation, Albuquerque, NM, ASME, Vol. 65-1, pp.
371-378.
Copyright © 1997 by ASME