0 оценок0% нашли этот документ полезным (0 голосов)

0 просмотров15 страницstat

Oct 28, 2019

ASME_DTM_TJP_5_19_97

© © All Rights Reserved

PDF, TXT или читайте онлайн в Scribd

stat

© All Rights Reserved

0 оценок0% нашли этот документ полезным (0 голосов)

0 просмотров15 страницASME_DTM_TJP_5_19_97

stat

© All Rights Reserved

Вы находитесь на странице: 1из 15

net/publication/23884611

Computer Experiments

Source: NTRS

CITATIONS READS

140 1,409

4 authors, including:

Pennsylvania State University SAS Institute

363 PUBLICATIONS 13,329 CITATIONS 41 PUBLICATIONS 2,404 CITATIONS

Janet Allen

University of Oklahoma

355 PUBLICATIONS 6,092 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

The Goal-oriented Inverse Design (GoID) Method for Systems-based Design Under Uncertainty View project

Investigating the Effectiveness of Machine Learning Paradigms for Supporting Engineering Designers in Rapidly Evolving Digital Manufacturing View project

All content following this page was uploaded by Patrick N Koch on 05 June 2014.

Proceedings of DETC’97

1997 ASME Design Engineering Technical Conferences

September 14-17, 1997, Sacramento, California

DETC97DTM3881

DETERMINISTIC COMPUTER EXPERIMENTS

Systems Realization Laboratory

Woodruff School of Mechanical Engineering

Georgia Institute of Technology

Atlanta, Georgia 30332-0405, USA

ABSTRACT element codes can take from minutes to hours, if not longer.

Perhaps the most prevalent use of statistics in engineering What's more, this mode of query-and-response can often lead to

design is through Taguchi's parameter and robust design - using a trial and error approach to design, where a designer may never

orthogonal arrays to compute signal to noise ratios in a process uncover the functional relationship between x and y and

of design improvement. In our view, however, there is an therefore may never identify the best settings for input values.

equally exciting use of statistics in design that could become Statistical techniques are widely used in engineering design

just as prevalent: it is the concept of metamodeling, whereby to address these concerns. The basic approach is to construct

statistical models are built to approximate detailed computer approximations of the analysis codes that are much more effi-

analysis codes. Although computers continue to get faster, our cient to run and that yield insight into the functional

analysis codes always seem to keep pace, so that their computa- relationship between x and y. If the true nature of a computer

tional time remains non-trivial. Through metamodeling, analysis code is represented as

approximations of these codes are built that are orders of magni- y = f(x),

tude cheaper to run. These metamodels can then be linked to

optimization routines for fast analysis, or they can serve as a then a "model of the model" or metamodel (Kleijnen, 1987) of

bridge for integrating codes across different domains. the analysis code is taken to be

In this paper we first review metamodeling techniques that y = g(x), and so y = y + ε

encompass the Design of Experiments, Response Surface

Methodology, Taguchi methods, neural networks, inductive where ε represents both the error of approximation and

learning, and kriging. We discuss their existing applications in measurement or random errors. The most common meta-

engineering design and then address the dangers of applying modeling approach is to apply the Design of Experiments

traditional statistical techniques to approximate deterministic (DOE) to identify an efficient set of computer runs (x 1, x 2, ...,

computer analysis codes. We conclude with recommendations x n) and then employ regression analysis to create a polynomial

for the appropriate use of metamodeling techniques in given approximation of the computer analysis code. These approxi-

situations and how common pitfalls can be avoided. mations then can replace the existing code while providing:

KEYWORDS: Design of Experiments, Response • a better understanding of the relationship between x and y

Surface Methodology, Metamodeling, Kriging, Neural Net- (we typically lose this understanding when running computer

works, Rule Induction, D-Optimal, Taguchi Robust Design codes "blindly"),

• facilitated integration of domain dependent computer codes

(we no longer work with the codes themselves but rather

1 INTRODUCTION simple approximations of them), and

Much of today's engineering analysis work consists of run- • fast analysis tools for optimization and exploration of the

ning complex computer analysis codes -- supplying a vector of design space (we become more efficient because we are using

design variables (inputs) x and receiving a vector of responses the approximations rather than the computationally expensive

(outputs) y. Despite continual advances in computing power codes themselves).

and speed, the expense of running many analysis codes remains However, we have found that many published applications

non-trivial. Single evaluations of aerodynamic or finite- (including our own) utilizing these statistical methods for

computer-oriented design are statistically questionable, if not focus on experimental designs with particular emphasis on

incorrect. The root cause is that many of our analysis codes are (fractional) factorial designs, central composite designs, and

deterministic, which means that the error of approximation is orthogonal arrays. We also discuss measures of merit for the

not due to random effects. This calls into question most subse- evaluation of experimental designs. In Section 2.2, we discuss

quent statistical analyses of model significance. Consequently, model choice and model fitting, focusing on response surfaces,

our purpose in this paper is to highlight the potential statistical neural networks, inductive learning and kriging. We conclude

pitfalls in metamodeling and to provide some general recom- with an overview of two of the more prevalent metamodeling

mendations for the proper use of statistical techniques in design. techniques, namely, Response Surface Methodology and

In Section 2, we present a review of metamodeling techniques Taguchi's robust design.

that encompasses regression, neural networks, inductive learn-

ing, and the more advanced approach of kriging. We conclude

Section 2 with an introduction to the general statistical 2.1 Experimental Design

approaches of Response Surface Methodology (RSM) and Properly designed experiments are essential for effective

Taguchi's robust design. In Section 3, we describe the engi- computer utilization. The traditional approach in engineering is

neering design context for statistical applications, review to vary one parameter at a time within a computer analysis code

existing applications, methods and tools, and close by taking a and observe the effects, or to randomly assign different combi-

closer look at metamodeling techniques for deterministic com- nations of factor settings to be used as alternative analyses for

puter experiments. In Section 4 we present some general rec- comparison. Experimental design techniques developed for

ommendations for avoiding some important pitfalls in the effective physical experiments, however, are being applied for

application of metamodeling to computer-oriented design, and the design of engineering computer experiments to increase the

in Section 5 we conclude by discussing briefly some of the efficiency (computer time) and effectiveness of these analysis

more advanced issues that continue to make metamodeling an codes. In this section an overview of different types of experi-

active research area. ment designs is provided, along with measures of merit for

selecting or comparing different designs.

In its most basic sense, metamodeling involves (a) experimental design formally represents a sequence of experi-

choosing an experimental design for generating data, (b) ments to be performed, expressed in terms of factors (design

choosing a model to represent the data, and then (c) fitting the variables) set at specified levels (predefined values). An

model to the observed data. There are a variety of options for experimental design is represented mathematically by a matrix

each of these steps as shown in Figure 1, and we have X where the rows denote experimental runs and the columns

attempted to highlight a few of the more prevalent ones. For denote the particular factor setting for each factor for each run.

example, Response Surface Methodology usually employs Factorial Designs: The most basic experimental design is a

central composite designs, second order polynomials and least full factorial design. The number of design points dictated by a

squares regression analysis. full factorial design is the product of the number of levels for

SAMPLE each factor. The most common designs are the 2k (for evaluat-

EXPERIMENTAL MODEL MODEL METAMODELING ing main effects and interactions) and 3k designs (for evaluating

DESIGN CHOICE FITTING TECHNIQUES

main and quadratic effects and interactions) for k factors at 2 and

(Fractional) Factorial Polynomial

(linear, quadratic, etc.)

Least Squares

Regression

Response Surface

Methodology

3 levels, respectively. A 23 full factorial design is shown

Central Composite graphically in Figure 2(a).

Box-Behnken Splines Weighted Least The size of a full factorial experiment increases exponen-

(linear, cubic) Squares Regression

D-Optimal

tially with the number of factors which may lead to an unman-

G-Optimal

Best Linear ageable number of experiments, e.g., 10 factors each with 2

Frequency-Domain Unbiased Predictor Kriging

(BLUP) levels requires 210 or 1024 experiments. Fractional factorial

Orthogonal Array

Best Linear

designs can be used when experiments are costly and the

Plackett-Burman Kernel Smoothing Predictor (BLP) number of design points for a full factorial design is large. A

Hexagon

Radial Basis Functions Log-Likelihood

fractional factorial design consists of a fraction of a full factorial

Hybrid design; the most common fractional factorial designs are 2(k-p)

Latin Hypercube Network of Neurons Backpropagation Neural

Networks designs where a fraction is equal to 1/2(p). A half fraction of the

Select By Hand 23 full factorial design is shown in Figure 2(b).

Rulebase or Entropy Inductive

Random Selection Decision Tree (information-theoretic) Learning The reduction of the number of design points associated

with a fractional factorial design does not come without a price.

Design resolution and aliasing of effects are two critical (and

Figure 1. Techniques for Metamodeling related) issues which are important when selecting a fractional

factorial design. The 23 full factorial design shown in Figure

In the remainder of this section we provide a brief overview 2(a) allows estimation of all main effects (x1, x 2, x 3), all two

of several of the options listed in Figure 1. In Section 2.1, we factor interactions (x1x2, x 1x3 and x 2x3), as well as the three

factor interaction (x1x2x3). For the 23-1 fractional factorial indi- lead to an unmanageable number of design points, e.g., 7

cated by the solid dots in Figure 2(b), the main effects are alised factors at three levels requires 37 or 2187 experiments. The

(or biased) with the two factor interactions. Aliased effects can- most common second-order designs, configured to reduce the

not be estimated independently unless all but one of the aliased number of design points, are central composite and Box-

effects are known or assumed not to exist. The resolution of an Behnken designs.

experiment design is defined by the aliasing of effects: a design A central composite design (CCD) is a two level (2k or 2 (k-

p)

of resolution R is one in which no p-factor effect is aliased with ) factorial design, augmented by n0 center points and two "star"

any other effect containing less than R-p factors (Box, et al., points positioned at ±α for each factor. This design, shown for

1978). The 23-1 design in Figure 2(b) can be at most a Resolu- three factors in Figure 2(c) consists of 2(k-p) + 2 k + n 0 total de-

tion III design, denoted23-1III. sign points to estimate 2k + k(k-1)/2 + 1 coefficients. Setting

Often 2k and 2(k-p) designs are used for identifying or screen- α=1 locates the star points on the centers of the faces of the

ing important factors when the number of factors is too large to cube (for three factors), giving a face-centered central composite

evaluate higher order effects. In these cases the sparsity of (CCF) design. It should be noted that for values of α other than

effects principle (Montgomery, 1997) is invoked, in which the 1, each factor is evaluated at five levels.

system is assumed to be dominated by the main effects and their Often it is desirable to employ the smallest number of fac-

low-order interactions. Based on this principle, two level frac- tors levels (e.g., three levels to fit a quadratic model) when

tional factorial designs can be used to "screen" the factors to creating an experimental design. One common, economical

determine which have the largest effect. The sparsity of effects class of such designs is the Box-Behnken designs. These

principle may not always hold true, however. As noted by designs are formed by combining 2k factorials with incomplete

Hunter (1970), every design provides aliased estimates (quadratic block designs (see Box and Draper, 1987). These designs do

and cubic effects, if present, will bias the estimates of the mean not contain any points at the vertices of the hypercube defined

and main effects respectively when a two level factorial design by the upper and lower limits for each factor. This is desirable

is used). "Any phenomenon omitted from a fitted model will if these extreme points are expensive or impossible to test.

confound [i.e., alias] certain estimated parameters in the model More information about central composite and Box-Behnken

regardless of the design used. Good fractional factorial designs design can be found in (Montgomery, 1997).

are carefully arranged so that estimates of effects thought to be Orthogonal Arrays: The experiment designs used by

important are confounded by effects thought to be unimpor- Taguchi, called orthogonal arrays, are most often simply

tant." fractional factorial designs in two or three levels (2(k-p) and 3(k-p)

One specific family of fractional factorial designs com- designs). These arrays are constructed to reduce the number of

monly used for screening are the two level Plackett-Burman (P- design points necessary to evaluate the required effects; the two-

B) designs. These resolution III designs are constructed to study level L4, L 12, and L16 arrays, for example, allow 3, 11, and 15

k=N-1 factors in N=4m design points. The class of P-B designs factors/effects to be evaluated in 4, 12, and 16 design points

in which N is a power of two are called geometric designs are respectively. In many cases, these designs are identical to those

are identical to the usual 2k-p fractional factorials. When N is given by Plackett and Burman (Lucas, 1994). The definition of

strictly a multiple of four, the P-B designs are referred to as orthogonality, for these arrays and other experiment designs, is

non-geometric designs and have very messy alias structures. given in Section 2.1.2. An overview of Taguchi's approach to

Their use in practical problems is often problematic particularly parameter design, within which the orthogonal arrays are

when the design is saturated, i.e., the number of factors is implemented, is given in Section 2.3.

exactly N-1. If all interactions are negligible, however, these

designs allow unbiased estimation of all main effects, using 2 . 1 . 2 Measures of Merit for Evaluating Experi-

only one more design point than the number of factors, with mental Designs. Selecting the appropriate design is

smallest possible variance (Box, et al., 1978). Minimum vari-

essential for effective experimentation. Experimenters must

ance and minimum size designs will be discussed in Section

balance the desire to gain as much information as possible

2.1.2. For a more complete discussion of factorial designs,

about the response-factor relationships with the cost of

confounding, and design resolution (see Myers and

experimentation and need for efficiency (measured in number of

Montgomery, 1995).

X2

runs). Several measures of merit are available and useful for

X2 X2 evaluating and comparing experimental designs to ensure the

appropriate experiment is designed.

Orthogonality, Rotatability, Minimum Variance, and

X1 X1 X1

Minimum Bias: Four desirable characteristics of an

X3 X3

X3 experimental design, to facilitate efficient estimates of the

parameters, are orthogonality, rotatability, minimum variance,

(a) 23 Full Factorial (b) 23-1 Fractional Factorial (c) Composite Design and minimum bias. A design is orthogonal if for every pair of

Figure 2. Basic Three-Factor Designs factors xi and x j the sum of the crossproducts for the N design

points is zero. For a first order model, the estimates of all

Central Composite and Box-Behnken Designs: For estimat- coefficients will have minimum variance if the design is

ing quadratic effects the 3k or 3 k-p designs can be used but often configured so that

N D-optimal determinant value, normalized by the size of the

∑ x2iu = N matrix for comparing different size designs.

u=1

D-efficiency = (|X'X|design /|X'X|D-optimum)1/p (1)

the variance of predictions y will also have constant variance at

The D-efficiency statistic is the primary criteria with which

a fixed distance from the center of the design, and the design

designs have been compared. Other statistics for comparing

will be rotatable.

designs such as G-efficiency and A-optimality have also been

For second order modeling, Hunter (1985) suggests that

formulated (see Myers and Montgomery, 1995). Having

orthogonality loses its importance. "If the objective of the

completed our review of experimental design alternatives and

experimenter is to forecast a response at either present or future

measures of merit, we can now turn to the issues of model

settings of x, then an unbiased minimum variance estimate of

choice and model fitting.

the forecast y is required. In the late 1950's Box and his co-

workers demonstrated that rotatability....and the minimization

of bias from higher order terms....were the essential criteria for 2.2 Model Choice and Model Fitting

good forecasting." After selecting an appropriate experimental design and

Unsaturated/Saturated and Supersaturated Designs: In many performing the necessary computer runs, the next step involves

cases the primary concern in the design of an experiment is the choosing an approximating model and fitting method. Many

size (number of runs) of the resulting design. Most experiment alternative models and methods exist, but for practicality we

designs are unsaturated in that they contain at least two more review in this section the four that we have found to be most

design points than the number of factors. An experiment prevalent in the literature: response surfaces, neural networks,

design is said to be saturated if the number of design points is inductive learning and kriging.

equal to one more than the number of factor effects to be

estimated. Saturated fractional factorial designs allow unbiased

2 . 2 . 1 Response Surfaces. Given a response of

estimation of all main effects with smallest size and smallest

possible variance (Box, et al., 1978). In a supersaturated interest, y, and a vector of independent factors x thought to

design, then, the number of design points is less than or equal influence y, the relationship between y and x can be written as

to the number of factors. The most common examples of follows:

saturated designs are the Plackett-Burman two level designs y = f(x) + ε , (3)

discussed in the previous section, and the related orthogonal

where ε represents random error which is assumed to be

arrays of Taguchi. Additional aspects of supersaturated designs

normally distributed with mean zero and standard deviation σ.

can be found in (Draper and Lin, 1990a; Draper and Lin,

Since the true response surface function f(x) is usually

1990b).

unknown, a response surface g(x) is created to approximate f(x).

For estimating second order effects, small composite

designs have be developed to reduce the number of design points Predicted values are then obtained using y = g(x).

necessary for a composite design. A small composite design The most widely used response surface approximating

becomes saturated if the number of design points is 2k + k(k- functions are simple low-order polynomials. If little curvature

1)/2 + 1 (the number of coefficients to be estimated for a full appears to exist, the first-order polynomial given in Equation 4

quadratic model). Myers and Montgomery (1995) note that can be employed. If significant curvature exists, the second-

recent work has suggested that these designs may not be good order polynomial in Equation 5, including all two-factor

choices in all applications; additional discussion on small interactions, can be used.

composite designs can be found in (Box and Draper, 1987; k

Lucas, 1974). y = b0 + ∑ bixi (4)

i=1

D-Optimal, D-Efficiency, and G-Efficiency: It is most

desirable to use unsaturated designs, when building predictive k k k k

models, unless running the necessary experiments is y = b0 + ∑ bixi + ∑ biix2i + ∑ ∑ bijxixj

i<j

(5)

prohibitively expensive. For comparing experiments based on i=1 i=1 i=1 j=1

number of design points and information obtained, many The parameters of the polynomials in Equations 4 and 5 are

measures of merit exist. A common measure is D-efficiency. usually determined using a least squares regression analysis to

A design is said to be D-Optimal if for the the model the fit these response surface approximations to existing data.

volume of the joint confidence region for the vector of These approximations are normally used for prediction within

unknown parameters is minimum. As a consequence, the Response Surface Methodology. A more complete discussion

global maximum of the determinant of X'X is achieved. Here of response surfaces and least squares fitting can be found in

X is the expanded design matrix, including N rows (one for (Myers and Montgomery, 1995). An overview of RSM is

each design point or run), and p columns (one for each given in Section 2.3.

coefficient to be estimated, including the overall mean). The D-

efficiency statistic for comparing designs, given in Equation 1,

compares the determinant value achieved for the design with the 2 . 2 . 2 Neural Networks. A neural network is composed

of neurons (single-unit perceptrons) which are multiple linear

regression models with a nonlinear (typically sigmoidal) inductive learning have been in process control and diagnostic

transformation on y. If the inputs to each neuron are denoted systems, and inductive learning approaches can be used to

x1, x 2, ..., x n, and the regression coefficients are denoted by the automate the knowledge-acquisition process of building expert

weights, wi, then the output, y, might be given by systems. Excellent examples can be found in (Evans and

Fisher, 1994; Langley and Simon, 1995; Leech, 1986).

y= 1

The training data collected are in the form of {(x 1,y 1),

1 + e -η/T (x 2,y 2), ..., (x n,y n)} where each x i represents a vector of

where η = Σwi xi + β. A neural network is then created by attribute values (such as processing parameters and

assembling the neurons into an architecture; the most prevalent environmental conditions), and each yi represents a

is the multi-layer feedforward architecture. corresponding observed output value. Although attributes and

There are two main issues in building a network: 1) outputs can be real-valued, the method is better suited to

specifying the architecture, and 2) training the network to discrete-valued data. Real values must often be transformed into

perform well with reference to a training set. "To a statistician, discrete representations (Evans and Fisher, 1994). Once the data

this is equivalent to (i) specifying a regression model, and (ii) set has been collected, training algorithms build a decision tree

estimating the parameters of the model given a set of data" by selecting the "best" divisive attribute and then recursively

(Cheng and Titterington, 1994). If the architecture is made calling the resulting data subsets. Although trees can be built

large enough, a neural network can be a nearly universal by selecting attributes randomly, a more efficient approach is to

approximator (Rumelhart, et al., 1994). select the attribute that minimizes the amount of information

"Training" a neural net refers to determining the proper needed for category membership. The mathematics of such an

values of all the weights (wi ) in the architecture, and is information-theoretic approach are given in (Evans and Fisher,

accomplished most commonly through backpropagation 1994).

(Rumelhart, et al., 1994). First, a set of p training data points Although decision trees seem best-suited for applications

is assembled {(x1,y 1}, (x2,y 2), ..., (x p,y p)}. For a network with with discrete input and output values, there are also applications

a single output y, the performance of the network is then with continuous variables that have met with greater success

defined as than standard statistical analysis. For example, Leech (1986)

E = ∑ (yp - yp )2 reports a process-control application where "Standard statistical

p analysis methods were employed with limited success. Some

of the data were non-numerical, the dependencies between

where yp is the output that results from the network given variables were not well understood, and it was necessary to

input x p and E is defined as the total error of the system. The simultaneously control several characteristics of the final

weights are then adjusted in proportion to product while working within system constraints. The results

∂E ∂y . of the statistical analysis, a set of correlations for each output

of interest, were difficult for people responsible for the day-to-

∂y ∂wij day operation to interpret and use."

Neural networks are best suited to approximate

deterministic functions in regression-type applications. Cheng 2 . 2 . 4 Kriging. Since many computer analysis coedes are

and Titterington (1994) note that "In most applications of deterministic and are therefore not subject to measurement error,

neural networks that generate regression-like output, there is no the usual measures of uncertainty derived from least-squares

explicit mention of randomness. Instead, the aim is function residuals have no obvious statistical meaning (Sacks, et al.,

approximation." Typical applications are speech recognition 1989b). Consequently, some statisticians (e.g., (Sacks, et al.,

and handwritten character recognition; very often the data is 1989a; Welch, et al., 1992)) have suggested modeling the

complex and of high dimensionality . Networks with tens of response as a combination of a polynomial model (usually

thousands of parameters are not unheard of, and the amount of linear) plus departure, i.e.,

training data is similar. Gathering the training data and k

determining the model parameters is a process that can be very y(x) = ∑ βjfj(x) + Z(x)

computationally expensive. j=1

2 . 2 . 3 Rule Induction. Rule induction is one of five Z(•) represents the realization of a stochastic process and is

main paradigms of machine learning that also include neural assumed to have mean zero and covariance

networks, case-based learning, genetic algorithms, and analytic

learning (Langley and Simon, 1995). Of the five, inductive V(w,x) = σ2R(w,x)

learning is most akin to regression and metamodeling and is between Z(w) and Z(x) where σ2 is the process variance and

therefore our focus here. An inductive learning system induces R(w,x) is the correlation. The covariance structure of Z relates

rules from examples, so the fundamental modeling constructs to the smoothness of the approximating surface.

are condition-action rules. These rules in essence partition the One method of analysis for such models is kriging

data into discrete categories, and can be combined into decision (Matheron, 1963) which works as follows (Sacks, et al.,

trees for ease of interpretation. Many of the applications of 1989b). Given a design S = {s 1, s 2, ..., s n} and data ys = {y(s1),

..., y(s n)}', we are concerned with estimating the y(x) at an affect it." Finally, Biles (1984) defines RSM as the, "body of

untried x using the linear (or polynomial) predictor techniques by which one experimentally seeks an optimum set

of system conditions".

y(x) = c'(x)ys. RSM then encompasses and incorporates the design of

If we consider y(x) as random, y s becomes Y s = [Y(s 1), ..., experiments (Section 2.1), response surface model building

(Section 2.2.1), and 'model exploitation' for exploring a factor

Y(sn)]', and we can compute the mean squared error of the space and seeking optimum factor settings. The general RSM

predictor averaged over the random process. Thus, the best approach includes all or a subset of the following steps:

linear unbiased predictor (BLUP) is found by picking the n x 1 i) screening: when the number of factors is too large for a

vector c(x) to minimize comprehensive exploration and/or when experimentation is

MSE[y(x)] = E[c'(x)Ys - Y(x)]2 expensive, screening experiments are used to reduce the set

of factors to those that are most important to the

subject to the constraint for unbiasedness response(s) being investigated;

E[c'(x)Ys] = E[Y(x)] (6) ii) first-order experimentation: when the starting point is far

from an optimum (or in general when knowledge about the

R(w,x) must first be specified in order to compute Equation 6 space being investigated is sought), first order-models and

and should reflect the characteristics of the computer code. For an approach such as steepest-ascent are employed to

a smooth response, a covariance function with some derivatives "rapidly and economically move to the vicinity of the

might be preferred, whereas an irregular response might call for optimum" (Montgomery and Evans, 1975);

a function with no derivatives. Sacks, et al. (1989b) discuss iii) second-order experimentation: after the best solution using

several possible correlation functions which result in a linear first-order methods is obtained, a second-order model is fit

interpolation or cubic spline interpolation of the data (see in the region of the first-order solution to evaluate

(Mitchell, et al., 1988) for more details). Consequently, curvature effects and attempt to improve the solution.

kriging is extremely flexible due to the wide range of A more detailed description of RSM techniques and tools can be

correlation functions R(w,x) which may be chosen. Depending found in (Box and Draper, 1987), and a comprehensive review

on the choice of the correlation function, kriging can either of RSM developments and applications from 1966-1988 is

"honor the data", providing an exact interpolation of the data, or given in (Myers, et al., 1989).

"smooth the data", providing a nonexact interpolation of the

data (Cressie, 1988). More information about kriging and the

notion of modeling deterministic computer experiments as the 2 . 3 . 2 Taguchi's Robust Design. Genichi Taguchi

realization of a stochastic process can be found in (Cressie, developed an approach for industrial product design that is built

1989; Journel, 1986; Laslett, 1994; Matheron, 1963; Sacks, et on the foundation of statistically designed experiments.

al., 1989b). Cressie (1988) deals specifically with predicting Taguchi's robust design for quality engineering includes three

noiseless versions of Z, as is the case for deterministic steps: system design, parameter design, and tolerance design

computer experiments. As a final note, keep in mind that (Byrne and Taguchi, 1987). The key element of the approach is

kriging and splines (nonparametric regression models) are not the parameter design step, within which statistical

the same. In fact, in several comparative studies, kriging has experimentation is incorporated.

been seen to outperform splines and never performs worse than Rather than simply improving or optimizing a response

them (Laslett, 1994). value, the focus in parameter design is to identify the factor

settings that minimize variation in performance as well as

adjust the mean performance to a desired target. Factors

2.3 Strategies for Experimentation and included in experimentation include control factors and noise

Metamodeling factors. Control factors are those which can be set and held at

Two widely used methods incorporating experimental specific values, while noise factors are those which cannot be

design, model building, and prediction are Response Surface controlled, e.g., temperature on a shop floor. The evaluation of

Methodology and Taguchi's robust design or parameter design. mean performance and performance variation is accomplished

A brief overview of these two approaches is provided. through "crossing" two orthogonal arrays (Section 2.1.1).

Control factors are varied according to an inner array, or

"control" array, and for each run of the control array, the noise

2 . 3 . 1 Response Surface Methodology. Different

factors are varied according to an outer array, or "noise" array.

authors describe Response Surface Methodology differently. For each control factor experiment, a response value is obtained

Myers, et al. (1989) define RSM as "a collection of tools in for each noise factor design point, and the mean and variance of

design or data analysis that enhance the exploration of a region the response (measured across the noise design points) can be

of design variables in one or more responses." Box and Draper calculated. The performance characteristic used by Taguchi is a

(1987) state that, "Response surface methodology comprises a signal-to-noise (S/N) ratio defined in terms of the mean and

group of statistical techniques for empirical model building and variance of the response. Several alternate S/N ratios are

model exploitation. By careful design and analysis of available based on whether lower, higher, or nominal response

experiments, it seeks to relate a response, or output variable to

the levels of a number of predictors, or input variables, that

values are desired; specific S/N ratio formulations can be found terms, the design variables are factors, and the design objectives

in (Ross, 1988). are responses.) An intelligent search for these solutions most

The Taguchi approach does not explicitly include model commonly relies on some choice of optimization techniques,

building and optimization. Analysis of experimental results is which generate and evaluate many potential solutions in the

used to identify factor effects, plan additional experiments, and path towards improvement. Regardless of the optimization

to set factor values. Comprehensive presentation and technique employed a large number of function evaluations are

application of the Taguchi approach is given in (Phadke, 1989; required, thereby making fast analysis modules a near

Ross, 1988). Issues and techniques presented by Taguchi have imperative.

been used extensively in engineering design, and in many cases When are metamodels useful or appropriate? In the later

incorporated within the traditional RSM for efficient, effective, stages of design, when detailed information about a specific

and robust design, e.g., (Myers and Montgomery, 1995). These solution is available, highly accurate solutions are essential for

applications and associated implications for engineering design measuring system performance. In the early stages of design,

are discussed in the next section. however, the focus is on generating, evaluating, and comparing

potential conceptual configurations. In this stage, a conceptual

design problem is characterized by a large amount of

3 METAMODELING IN THE CONTEXT OF information, often uncertain, that must be managed. To ensure

ENGINEERING DESIGN the identification of a "good" system level conceptual or

How are the metamodeling techniques of the previous preliminary configuration, a comprehensive search is necessary.

section employed in engineering design? They can all be used In this case, the tradeoff between accuracy and efficiency

to produce approximations of existing computer analysis codes, becomes appropriate. The creation of metamodels allows fast

thereby producing fast analysis modules for more efficient analysis, which facilitates both comprehensive and efficient

computation. These metamodeling techniques can also yield design space search, at the expense of a (hopefully slight) loss

insight to the functional relationship between input and output of accuracy.

parameters. Having established this baseline engineering design

"context", in the next section we present a review of several

Given: Factors, responses statistical applications in engineering design, highlighting

general trends and observations. In Section 3.2 we discuss a

few of the more general statistical methods or "tools" which

Large # of YES

Run Screening have been developed for engineering applications, and we

Factors? Experiment conclude with Section 3.3 in which we take a closer look at the

some of the statistical pitfalls associated with the application of

NO Screening statistical techniques to deterministic computer experiments.

Run Modeling Reduce # This then paves the way for Section 4, where we present our

Experiment(s) Factors guidelines and recommendations for the appropriate use of

statistics in computer-oriented design.

Build Predictive

Model ( y )

Model

Building 3.1 Existing Applications in Engineering

Design

Noise YES Build Robustness In this section we provide a survey of several engineering

Factors? Model ( y , y ) applications of Design of Experiments, Response Surface

Methodology, and Taguchi's robust design method. Most of

our examples come from aerospace and mechanical engineering

NO

design applications presented at conferences in recent years; for

Search Design brevity's sake, we summarize our findings in Table 1. Our

Model Space observations follow.

Exercising

• In terms of experimental design, central composite designs

Solutions and D-Optimal designs seem to be the preferred choice among

(improved or robust) aerospace engineers while orthogonal arrays (OAs) are

Figure 3. A Principal Use of Statistics in preferred by many mechanical engineers; very few people

Computer-Oriented Design utilize grid searches and random point generation since they

Where would such models be useful? There is an implicit are less efficient.

process that is common to most of the engineering design • Optimization seems to be the principal driver for aerospace

applications, and this process is illustrated in Figure 3. A applications of DOE and RSM; these types of design

designer's ultimate goal is most often to arrive at improved or applications typically involve the use of computer intensive

robust solutions, which are in essence the values of design analysis and optimization routines, and RSM is a logical

variables that best meet the design objectives. (In statistical choice for increased efficiency.

• Mechanical engineers are more inclined to use OAs and

Taguchi for robust design instead, working with the signal-

to-noise ratio for parameter and tolerance design. 3.2 Existing Methods and Tools in

• Very few designers actually model Taguchi's loss function Engineering Design

directly (e.g., Beard and Sutherland, 1993); many prefer to In this section we present a few methods and tools

model the response instead. developed specifically for engineering design that are build

• Most applications employ second order response surface around or incorporate the statistical techniques and approaches

models; there are only a few instances were higher order reviewed in Section 2. As evident in Table 1 in the previous

models (e.g., Venter, et al., 1996) and mixed polynomial section, the Taguchi approach and RSM have been widely

models (e.g., Roux, et al., 1996) are employed. applied in engineering design; a review of literature comparing

• When orthogonal arrays are used, special care must be taken these approaches is given first, followed by an overview of

to avoid confounding main effects with interactions, Robust Design Simulation (RDS), Robust Concept

providing the interactions are not assumed to be insignificant Exploration Method (RCEM), NORMAN/DEBORA,

based on prior observations or experience. Some authors take DOE/Opt, and HIDER.

this into account while others do not.

• Most applications utilize a least squares regression analysis 3 . 2 . 1 Taguchi Approach vs. RSM. Both Taguchi

when fitting a model; only a few use stepwise regression, and techniques and Response Surface Methodology have been

this is usually because the model is not second order. applied extensively in engineering design. Many authors have

compared these techniques for different problems. It appears to

Table 1. Cited Engineering Applications of be commonly accepted that the principles associated with

DOE, RSM, and Taguchi Taguchi approach are not only useful but very appropriate for

Experimental

Design

Response

Surface Method

Taguchi

Method

industrial product design; Ramberg, et al. (1991) for example

suggest that "the loss function and the associated robust design

Consider Interactions

S/N, Loss, or Resp.

Fractional Factorial

Fit Using:

Optimize/Improve

Robust Design

Order of Eqn.

Grid/Random

Least Sqr.

D-Optimal

Stepwise

CCD

OAs

Paper

inner and outer) and the use of the loss function or signal-to-

(Balabanov, et al., 1996) x x 2nd x x noise ratio. It has been argued and demonstrated that the use of

(Bauer and Krebs, 1996) x x x 2nd x x S/N

(Beard and Sutherland, 1993) x x x L

a single experiment combining control and noise factors, a

(Chen, et al., 1996) x x 2nd x R

more traditional RSM approach rather than inner "control" and

(Chi and Bloebaum, 1995) x x R outer "noise" arrays, is more efficient in that it requires fewer

(Engelund, et al., 1993) x x 2nd x experiments (Shoemaker, et al., 1991; Unal, et al., 1994a;

2nd

(Gadallah and ElMaraghy, 1994) x x x x x S/N Welch, et al., 1990). With respect to the loss function or

(Giunta, et al., 1996) x x 2nd x signal-to-noise ratio, the drawbacks of combining response

(Giunta, et al., 1994) x x x x 2nd x

(Healy, et al., 1975) x x 2nd x x

mean and variance into a single measure, a signal-to-noise ratio,

(Hong, et al., 1994) x x 2nd x x S/N

are well-documented in the literature and many authors advocate

(Koch, et al., 1996) x x 2nd x measuring a response directly rather than loss, and separately

(Korngold and Gabriele, 1995) x 2nd x tracking mean and variance (Chen, et al., 1995; Ramberg, et

2nd

(Li, et al., 1994) x x x S/N al., 1991; Welch, et al., 1990). Shoemaker et al. (1991),

(Mavris, et al., 1996) x x 2nd x R however, warn that "A potential drawback of the response-

(Mavris, et al., 1995) x x 2nd x R

(Roux, et al., 1996) x x Mix x

model approach that is depends more critically than the loss-

(Rowell, et al., 1996) x x x 2nd x x x model approach on how well the model fits."

(Stanley, et al., 1992) x x x R Given the general acceptance of the Taguchi principles and

(Sundaresan, et al., 1993) x x x R of robust design, with these criticisms of these techniques,

2nd

(Unal, et al., 1996) x x x many advocate a combined Taguchi-RSM approach, or simply

(Unal, et al., 1992) x x x R applying more traditional RSM techniques within the

(Unal, et al., 1994a) x x 2nd x

2nd

framework defined by Taguchi (Lucas, 1994; Myers, et al.,

(Unal, et al., 1994b) x x x x x R

(Venter, et al., 1996) x x 4th x 1989; Myers and Montgomery, 1995; Ramberg, et al., 1991).

(Yu and Ishii, 1993) x x 2nd x x R It is the belief of the authors of this paper that both orthogonal,

inner and outer arrays, and single, composite experiments each

have advantages and disadvantages and appropriate uses, and that

In addition to these specific engineering design applications

separate observation of mean and variance leads to useful

of DOE, RSM, and Taguchi, many methods and "tools" have

problem insight. Regardless, the core principles of both

been developed for more general design application. We

Taguchi and RSM have provided a foundation for the specific

examine several of these methods in the next section,

design methods discussed in Section 3.2.2.

highlighting various applications of each.

algorithms. HIDER and AIMS have been applied to the design

3 . 2 . 2 An Overview of Existing Methods. Robust of a cutting process (Lu and Tcheng, 1991), a diesel engine

(Yerramareddy, et al., 1992), and a wheel loader (Reddy, 1996).

Design Simulation (RDS), combines the design of experiments

The Concurrent SubSpace Optimization (CSSO)

and response surface metamodeling aspects of RSM with Monte

methodology developed in (Renaud and Gabriele, 1993; Renaud

Carlo simulation to facilitate robust design (Mavris, et al.,

and Gabriele, 1994; Renaud and Gabrielle, 1991) utilizes design

1995). The general implementation of this approach consists

data generated during concurrent subspace optimizations to

of three sequential experimentation stages: screening, second-

develop response surface approximations of the design space.

order experimentation for model building, and a combined

Optimization of these response surface representations of the

experimentation/Monte Carlo simulation stage for

design space form the basis of the coordination procedure

incorporating noise. The objective of RDS is to achieve robust

developed in (Renaud and Gabriele, 1993; Renaud and Gabriele,

system designs: those that minimize performance variation due

1994; Renaud and Gabrielle, 1991). The data generated by the

to uncontrollable noise, while achieving as closely as possible

subspace optimizers is not uniformly centered about the current

performance target values. RDS is presented and applied to the

design as in CCD or other sampling strategies, but instead

design of a High Speed Civil Transport aircraft in (Mavris, et

follows the descent path of the subspace optimizers. In

al., 1996; Mavris, et al., 1995).

(Renaud and Gabriele, 1993; Renaud and Gabriele, 1994;

The Robust Concept Exploration Method (RCEM) has

Renaud and Gabrielle, 1991) interpolating polynomial response

been developed to facilitate quick evaluation of different design

surfaces are constructed which have either a first order or second

alternatives and generation of top-level design specifications

order basis for use in the CSSO coordination procedure. In

with quality considerations in the early stages of design (Chen,

(Wujek, et al., 1996; Wujek and Renaud, 1997) a modified

et al., 1996a). Foundational to the RCEM is the integration of

decomposition strategy is employed in developing quadratic

robust design principles, design of experiments and response

response surfaces for use in the CSSO coordination procedure.

surface methodology, and the compromise Decision Support

Finally, in (Sellar and Batill, 1996; Sellar, et al., 1996a; Sellar,

Problem (DSP) (a multiobjective decision model which is a

et al., 1996b) artificial neural network response surfaces are

hybrid formulation based on Mathematical Programming and

developed for use in the CSSO coordination procedure.

Goal Programming). This RCEM is applied to a High Speed

Additional approaches incorporating statistical techniques

Civil Transport (HSCT) in (Chen, et al., 1996a) The authors of

in design exist; only a few have been included in this section.

this paper have also applied RCEM to the design of a General

Our focus is not on the methods, but on the appropriateness of

Aviation Aircraft (Simpson, et al., 1996), to a manufacturing

the statistical techniques: many of the examples to which these

simulation problem (Peplinski, et al., 1996), and to the design

methods have been applied employ deterministic computer

of a turbine lift engine (Koch, et al., 1996).

experiments, to which the application of statistical techniques

NORMAN/DEBORA is a TCAD (Technology Computer

is questionable. The associated issues are discussed in the next

Aided Design) system created by a Belgian company, IMEC,

section.

incorporating advanced sequential design of experiments (DOE)

and Response Surface Methodology (RSM) techniques to aid in

engineering optimization and robust design problems. 3.3 A Closer Look at Experimental Design for

Advanced DOE/RSM techniques incorporated within this Deterministic Computer Experiments

system include a novel design of experiments concept, Target Since engineering design commonly involves exercising

Oriented Design (TOD), a unique parameter transformation deterministic computer codes, the use of statistical

technique, RATIOFIND, and a non-linearly constrained experimentation in creating metamodels of these codes warrants

optimizer, DEBORA (Booth, et al., 1991). The a closer look. Given a response of interest, y, and a vector of

NORMAN/DEBORA system is demonstrated for semiconductor independent factors x thought to influence y, the relationship

integrated circuit (IC) technology design and optimization in between y and x (Equation 3) includes the random error term ε .

(Cartuyvels and Dupas, 1993). To apply least squares regression, the error values for each data

DOE/Opt is a prototype computer system for performing point are assumed to have identical and independent normal

design of experiments, response surface modeling, and distributions with means of zero and standard deviations of σ, or

optimization (Boning and Mozumder, 1994). Applications ε i i.i.d. N(0, σ2). This scenario is shown in Figure 4(a). The

described for DOE/Opt are in the semiconductor process and least squares estimator then minimizes the sum of the squared

device design area and include process/device optimization, differences between the actual data points and the values

simulator tuning, process control recipe generation, and design predicted by the model. It is acceptable if no data point actually

for manufacturability. lies on the predicted model, because it is assumed that the

Hierarchical and Interactive Decision Refinement (HIDER) model "smoothes out" the random error. Of course, it is likely

is a methodology for concept exploration in the early stages of that the regression model itself is only an approximation of the

design that integrates simulation, optimization and machine true behavior between x and y, so that the final relationship is

learning to support design decision making (Reddy, 1996).

HIDER uses the Adaptive Interactive Modeling System (AIMS) y = g(x) + ε bias + ε random (7)

(Lu and Tcheng, 1991) to decompose the design space using where ε bias represents the error of approximation. However, for

distance-based, population-based, and hyperplane-based deterministic computer experiments as illustrated in Figure

4(b), ε random has mean zero and variance zero, so after model Unal, et al., 1994b; Venter, et al., 1996; Welch, et al., 1990).

fitting we have the relationship These measures have no statistical meaning since they assume

the observations include an error term which has mean of zero

y = g(x) + ε bias (8)

and a non-zero standard deviation. Consequently, the use of

The deterministic case of Equation 7 conflicts sharply with stepwise regression for polynomial model fitting is not

the methods of least squares regression. First, unless ε bias is appropriate since it utilizes F-statistic values when

i.i.d. N(0,σ2) then the assumptions for statistical inference from adding/removing model parameters.

least squares regression are violated. Even further, because there R-Squared (when defined as the model sum of squares

is no random error there is little justification for smoothing divided by the total sum of squares) is really the only measure

across data points; instead the model should hit each point for verifying model adequacy for deterministic computer

exactly and interpolate between them as shown in Figure 4(b). experiments, and often this measure not sufficient (a high R-

Finally, most standard tests for model and parameter Squared value can be deceiving). Consequently, confirmation

significance are based on computations using ε random (the mean testing of model validity through use of additional (different)

squared error) and are therefore impossible to compute. These data points becomes essential. Residual plots may also be

observations are supported by literature in the statistics extremely helpful when verifying model adequacy for

community; as Sacks, et al. (1989b) carefully point out, identifying trends in data, examining outliers, etc.

because deterministic computer experiments lack random error: Some researchers (e.g., Giunta, et al., 1996; Giunta, et al.,

• response surface model adequacy is determined solely by 1994; Venter, et al., 1996) have also employed metamodeling

systematic bias, techniques such as RSM for modeling deterministic computer

• the usual measures of uncertainty derived from least-squares experiments which contain numerical noise. This numerical

residuals have no obvious statistical meaning (deterministic noise is used as a surrogate for random error, thus allowing the

measures of uncertainty exist, e.g., max |y(x) - y(x)| over x, standard least-squares approach to be applied. However, the

but they may be very difficult to compute), and assumption of equating numerical noise to random error is

• the classical notions of experimental blocking, replication questionable, and the appropriateness of their approach warrants

and randomization are irrelevant. further investigation.

Similarly, according to Welch and his co-authors (1990), Given the potential problems of applying least-squares

current methods for the design and analysis of physical regression to deterministic applications, the trade-off then

experiments (e.g., Box and Draper, 1987; Box, et al., 1978) are becomes one of appropriateness vs. practicality. If a response

not ideal for complex, deterministic computer models. "In the surface is created to model data from a deterministic computer

presence of systematic error rather than random error, statistical code using experimental design and least-squares fitting, and if

testing is inappropriate" (Welch, et al., 1990). Finally, a it provides very good agreement between predicted and actual

discussion of how the model should interpolate the observations values (a situation that often occurs when the computer model

can be found in (Sacks, et al., 1989a). is "well-behaved"), then there is no practical reason to discard it.

It should be used, albeit with caution. However, it is important

y y ε=0 to be aware of the fundamental assumptions of the statistical

techniques employed, so that we avoid incorrect or misleading

statements about model significance. In the next section we

ε ~ N(0,σ2 ) provide several guidelines pertaining to the appropriate use of

(yi - yi ) statistical methods and metamodeling of deterministic computer

experiments.

2

4 GUIDELINES AND RECOMMENDATIONS

L.S.E.=∑ (y i - y i)

2 ∑ (y i - y i) =0

How can a design engineer efficiently apply the

y = g(x) = second order

metamodeling tools of Section 2 while avoiding the pitfalls

least squares fit y = g(x) = spline fit described in Section 3.3? There are two ways to answer this

question: from the bottom up (tools applications) and from

x x

the top down (motives tools). The bottom-up approach is

(a) Non-Deterministic Case (b) Deterministic Case

presented in Section 4.1, the top-down in Section 4.2.

Figure 4. Deterministic and Non-Deterministic

4 . 1 Evaluation of Metamodeling Techniques

Curve Fitting

There are two components to this section. The first is an

evaluation of the four metamodeling techniques described in

So where can these methods go wrong? Unfortunately it is

Section 2.2. The second component is choosing an

very easy to unthinkingly classify the ε bias term from a

experimental design which has more direct applicability to

deterministic model fit as ε random and then proceed with standard

response surface methods but also applies to the remaining

statistical testing. Several authors have reported statistical

metamodeling techniques. The explorations of which designs

measures such as the F-statistics and root MSE for verification

are most appropriate are, as yet, open research areas.

of model adequacy (e.g., Healy, et al., 1975; Koch, et al., 1996;

Jensen and Myers, 1989). For comparisons of different design

4 . 1 . 1 Evaluation of Model Choice and Model types, refer to the following.

• Lucas (1976) compares CCD, Box-Behnken, uniform shell,

Fitting Alternatives. In Section 2.2 we described four

Hoke, Pesotchinsky, and Box-Draper designs, using the D-

metamodeling techniques; here we present some brief guidelines efficiency and G-efficiency statistics.

for their evaluation. • Myers and Montgomery (1995) provide a comprehensive

Response Surfaces: primarily intended for applications review of many experimental designs for fitting second order

with random error, though deterministic applications are not response surfaces. They conclude that hybrid designs are a

necessarily incorrect. Building higher-order models with greater good idea, if the unusual levels for the design variables can be

than ten factors becomes difficult. (Building regression models tolerated. For computer experiments, this is unlikely to be a

is tractable for fifteen to twenty factors, but actually obtaining problem.

the data is the limiting factor.) It is the most well-established • Montgomery and Evans (1975) compare six second-order

metamodeling technique, and is probably the easiest to use. designs: a) 3^2 factorial, b) rotatable orthogonal CCD, c)

Neural Networks: nonlinear regression approach best rotatable uniform precision CCD, d) rotatable minimum bias

suited for deterministic (non-noisy) applications. They can be a CCD, e) rotatable orthogonal hexagon, and f) rotatable

nearly universal approximator, and so are able to handle highly uniform precision hexagon. The criteria used for comparison

nonlinear or extremely large (~ 10,000 parameters) problems, as are average response achievement and distance from true

long as enough data points are provided and enough computer optimum.

time is allocated to fit the model. Neural networks therefore • Lucas (1974) compares symmetric and asymmetric composite

lend themselves to large problems, so that "Now the procedure and smallest composite designs for different numbers of

is to toss the data directly into the NN software, use tens of factors using the |X'X| criterion, and the D-efficiency and G-

thousands of parameters in the fit, let the workstation run 2-3 efficiency statistics.

weeks grinding away doing the gradient descent, and voilá, out • Giovannitti-Jensen and Myers (1989) discuss several first and

comes the result" (Cheng and Titterington, 1994). second order designs, observing that the performance of

Inductive Learning: modeling technique most appropriate rotatable CCD and Box-Behnken designs are nearly identical.

where the input and output factors are primarily discrete-valued They note that "hybrid designs appear to be very promising."

or can be grouped into categories. The predictive model, in the

form of condition/action rules or a decision tree, may perhaps

lack the mathematical insight desired for engineering design 4.2 Initial Recommendations for Metamodeling

applications, but it is ideal for applications such as process Uses

control and diagnosis. The majority of metamodeling applications are built around

Kriging: interpolation method created to handle the creation of low-order polynomial metamodels using central

deterministic data. It is not based on the same assumptions as composite designs and least squares regression. The nearly

standard regression analysis and appears to be better suited for universal popularity of this approach is due, at least in part, to

deterministic computer experiments, as long as the number of the maturity and well-established nature of response surface

factors is small (≤10, according to (Welch, et al., 1992)). methodology, the ease and simplicity of the method, and easily

Kriging is extremely flexible due to the wide scope of accessible software support tools. However, the RSM approach

correlation functions R(w,x) which may be chosen (Laslett, starts to break down when there are a large (> 10) number of

1994); it can provide an exact interpolation of the data or factors to be modeled, or when the relationship to be modeled is

"smooth the data" (Cressie, 1988). Additional advantages highly nonlinear. And as we have shown in Section 3.3, there

include (Welch, et al., 1992): are also dangers of applying the RSM approach blindly to

• it provides a basis, via the likelihood, for a stepwise deterministic applications. The alternative approaches to

algorithm to determine the important factors; metamodeling described in section 4.1.1 address these

• it is flexible, allowing nonlinear and interactions effects to limitations, each in their own way. Our recommendations are

emerge without explicitly modeling such effects; summarized as follows:

• the same data can be used for screening and building the • If a large number of factors must be modeled in a

predictor, so expensive runs are efficiently used. deterministic application, then neural networks may be the

The complexity of the method, however, coupled with a lack of best metamodeling choice despite their tendency to be

computer software support, may make the learning curve of this computationally expensive to create.

technique prohibitive in the near term. • If the underlying function to be modeled is deterministic

4.1.2 Evaluation of Experimental Designs. and highly nonlinear in a small (≤ 10) number of factors,

There are many voices in the discussion of the relative merits of then kriging may be the best choice despite its statistical

different experimental designs, and it is therefore unlikely that complexity.

we have captured them all. However, broad trends appear, and • However, in deterministic applications that have a small

in general we have found that central composite designs number of factors and are fairly well behaved, we present

consistently perform and compare favorably. Other second order another option for exploration: applying the standard RSM

designs worth investigating are the hexagonal (Montgomery and approach, augmented with a Taguchi outer array.

Evans, 1975) and Box-Behnken and hybrid designs (Giovannitti-

R S M / O A approach: The fundamental problem with space. We hope to have preliminary results in this area in the

applying least-squares regression to deterministic applications is near future.

the absence of ε random in Equation 8. However, if some of the

input parameters for the computer code can be classified as

noise factors, and if these noise factors are varied across an outer ACKNOWLEDGMENTS

array for each setting of the control factors, then in essence a Timothy Simpson and Jesse Peplinski have been supported

series of replications are generated that can be used to by NSF Graduate Research Fellowships, and Patrick Koch has

approximate ε random . This approximation can be justified if it is been supported by a Gwaltney manufacturing fellowship. In

reasonable to assume that, were the experiments performed on addition, financial support from NSF Grants DMI-96-12365 and

an actual physical system, that the random error observed would DMI-96-12327, and NASA Grant NGT-51102 is gratefully

have been due to fluctuations in the same noise factors that are acknowledged.

varied in the computer code. If this assumption is reasonable,

then statistical testing of model and parameter significance can

be performed, and models of both mean response and variability REFERENCES

Balabanov, V., Kaufman, M., Knill, D. L., Haim, D., Golovidov,

can be created from the same experimental design. We are O. and Giunta, A. A., 1996, "Dependence of Optimal Structural

currently investigating this approach in our ongoing work. Weight on Aerodynamic Shape for a High Speed Civil

Transport," 6th AIAA/USAF/NASA/ISSMO Symposium on

Multidisciplinary Analysis and Optimization, Bellevue, WA,

5 SUMMARY AND CONCLUSIONS AIAA, Inc., Vol. 1, pp. 599-612.

In this paper we have surveyed some applications of Bauer, D. and Krebs, R., 1996, "Optimization of Deep Drawing

statistics to engineering design and have developed the idea of Processes Using Statistical Design of Experiments," 22nd

metamodeling in the engineering design context, as discussed in Design Automation Conference (McCarthy, J. M., ed.), Irvine,

Section 1 and as shown in Figure 3. Metamodeling is an active CA, ASME, Paper No. 96-DETC/DAC-1446.

research area in more than just the statistics community, to Beard, J. E. and Sutherland, J. W., 1993, "Robust Suspension

which our reference list is mute testimonial. However, System Design," Advances in Design Automation (Gilmore, B.

applying these techniques to deterministic applications in J., Hoeltzel, D. A., et al., eds.), Albuquerque, NM, ASME, Vol.

65-1, pp. 387-395.

engineering design can lead to problems, as pointed out in Biles, W. E., 1984, "Design of Simulation Experiments," 1984

Section 3.1 and discussed in Section 3.3. We present the set of Winter Simulation Conference (WSC) (Sheppard, S., Pooch, U.,

our recommendations for applying metamodeling techniques in et al., eds.), Dallas, TX, IEEE, pp. 99-104.

Section 4, but this set is by no means complete. Boning, D. S. and Mozumder, P. K., 1994, "DOE/Opt: A System for

Comprehensive comparisons of these techniques have yet to be Design of Experiments, Response Surface Modeling, and

performed, and the difficulties of large problem size and Optiminzation Using Process and Device Simulation," IEEE

nonlinearity are ever-present. Trans. on Semiconductor Manufacturing, Vol. 7(2), pp. 233-244.

In particular, an issue of interest to us in our continuing Booth, R., Dupas, L., Cartuyvels, R. and De Meyer, K., 1991, "New

research is the problem of size, or scaling. As the size of a Technique to Improve Response Surface Method Model Fitting

problem (number of factors) increases, the cost associated with Accuracy for Process Technology and Device Design

Optimization," INternational Workshop on VLSI Process and

creating metamodels can begin to outweigh the associated Device Modeling (VPAD), Oiso Kanagawa, Japan.

gains. In addition, often the reduction in factors resulting from Box, G. E. P. and Draper, N. R., 1987, Empirical Model-Building

screening is still not sufficient to bring the problem to a and Response Surfaces, John Wiley & Sons, New York.

manageable size. This problem is compounded by the multiple Box, G. E. P., Hunter, W. G. and Hunter, J. S., 1978, Statistics for

response problem: complex engineering design problems Experimenters, John Wiley & Sons, Inc., New York.

invariably include multiple measure of performance, or Byrne, D. M. and Taguchi, S., 1987, "The Taguchi Approach to

responses, that must be modeled. The screening process also Parameter Design," 40th Annual Quality Congress Transactions,

breaks down when attempting to select the most important Milwaukee, Wisconsin, American Society for Quality Control.

factors for more than one response, as each response often Cartuyvels, R. and Dupas, L., 1993, "NORMAN/DEBORA: a

requires different important factors. The general question that Powerful CAD-Integrating Automatic Sequencing System

Including Advanced DOE/RSM Techniques for Various

arises from these problems, then, is how can these Engineering Optimization Problems," JSPE-IFIP WG 5.3

experimentation and metamodeling techniques be used DIISM'93 Workshop (Design of Information Infrastructure

efficiently for larger problems (problems with greater than 10 Systems for Manufacturing), Tokyo, Japan.

factors after screening)? One approach that we are pursuing is Chen, W., Allen, J. K., Mavris, D. and Mistree, F., 1996a, "A

problem partitioning or decomposition. A significant literature Concept Exploration Method for Determining Robust Top-Level

base exist in techniques for breaking a problem into smaller Specifications," Engr. Opt., Vol. 26, pp. 137-158.

problems; a good review of such methods can be found in Chen, W., Allen, J. K., Mistree, F. and Tsui, K.-L., 1995,

(Lewis and Mistree, 1997). Using these techniques, a complex "Integration of Response Surface Methods with the Compromise

problem may be broken down into smaller problems that allow Decision Support Problem in Developing a General Robust

efficient experimentation and metamodeling, which again will Design Procedure," ASME Design Automation Conference

(Azarm, S., et al., ed.), Boston, Massachusetts, ASME, New

lead to comprehensive and efficient exploration of a design York, DE-Vol. 82-2, pp. 485-492.

Chen, W., Simpson, T. W., Allen, J. K. and Mistree, F., 1996b, Kleijnen, J. P. C., 1987, Statistical Tools for Simulation

"Use of Design Capability Indices to Satisfy a Ranged Set of Practitioners, Marcel Dekker, New York.

Design Requirements," 1996 ASME Design Automation Koch, P. N., Allen, J. K. and Mistree, F., 1996, "Robust Concept

Conference (Dutta, D., ed.), Irvine, CA, ASME, New York, Paper Exploration for Configuring Turbine Propulsion Systems," 1996

No. 96-DETC/DAC-1090. ASME Design and Automation Conference (Dutta, D., ed.),

Cheng, B. and Titterington, D. M., 1994, "Neural Networks: A Irvine, CA, Paper No. 96-DETC/DAC-1285.

Review from a Statistical Perspective," Statistical Science, Vol. Korngold, J. C. and Gabriele, G. A., 1995, "Multidisciplinary

9(1), pp. 2-54. Analysis and Optimization of Discrete Problems Using Response

Chi, H.-W. and Bloebaum, C. L., 1995, "Mixed Variable Surface Methods," 21st Annual Advances in Design Automation

Optimization Using Taguchi's Orthogonal Arrays," 21st Annual Conference, Boston, MA, ASME, Vol. 82-1, pp. 173-180.

Advances in Design Automation Conference (Azarm, S., Dutta, Langley, P. and Simon, H. A., 1995, "Applications of Machine

D., et al., eds.), Boston, MA, ASME, Vol. 68-1, pp. 501-508. Learning and Rule Induction," Communications of the ACM,

Cressie, N., 1988, "Spatial Prediction and Ordinary Kriging," Vol. 38(11), pp. 55-64.

Mathematical Geology, Vol. 20(4), pp. 405-421. Laslett, G. M., 1994, "Kriging and Splines: An Empirical

Cressie, N., 1989, "Geostatistics," The American Statistician, Vol. Comparison of Their Predictive Performance in Some

43(4), pp. 197-202. Applications," Journal of the American Statistical Association,

Draper, N. R. and Lin, D. K. J., 1990a, "Connections Between Two- Vol. 89(42), pp. 391-400.

Level Designs of Resolutions III and V," Technometrics, Vol. Leech, W. J., 1986, "A Rule Based Process Control Method With

32(3), pp. 283-288. Feedback," Advances In Instrumentation, Vol. 41, pp. 169-175.

Draper, N. R. and Lin, D. K. J., 1990b, "Small Response-Surface Lewis, K. and Mistree, F., 1997, "The Other Side of

Designs," Technometrics, Vol. 32(2), pp. 187-194. Multidisciplinary Design Optimization: Accomodating a

Engelund, W. C., Douglas, O. S., Lepsch, R. A., McMillian, M. M. Multiobjective, Uncertain and Non-Deterministic World,"

and Unal, R., 1993, "Aerodynamic Configuration Design Using Engineering Optimization, in press.

Resopnse Surface Methodology Analysis," AIAA Aircraft Li, C.-C., Sheu, T.-S. and Lee, Y.-C., 1994, "Robustness

Design, Systems and Operations Meeting, Monterey, CA, Paper Consideration in Modelling a Prototype by Quality

No. AIAA 93-3967. Engineering," Design Theory and Methodology Conference

Evans, B. and Fisher, D., 1994, "Overcoming Process Delays with (Hight, T. K. and Mistree, F., eds.), Minneapolis, MN, ASME,

Decision Tree Induction," IEEE Expert, Vol. 9, pp. 60-66. Vol. 68, pp. 55-62.

Gadallah, M. H. and ElMaraghy, H. A., 1994, "The Tolerance Lu, S. C.-Y. and Tcheng, D. K., 1991, "Building Layered Models to

Optimization Problem Using a System of Experimental Design," Support Engineering Decision Making: A Machine Learning

Advances in Design Automation Conference, Minneapolis, MN, Approach," Journal of Engr. for Industry, Vol. 113, pp. 1-9.

ASME, Vol. 69-1, pp. 251-265. Lucas, J. M., 1974, "Optimum Composite Designs,"

Giovannitti-Jensen, A. and Myers, R. H., 1989, "Graphical Technometrics, Vol. 16(4), pp. 561-567.

Assessment of the Prediction Capability of Response Surface Lucas, J. M., 1976, "Which Response Surface Design is Best,"

Designs," Technometrics, Vol. 31(2), pp. 159-171. Technometrics, Vol. 18(4), pp. 411-417.

Giunta, A. A., Balabanov, V., Haim, D., Grossman, B., Mason, W. Lucas, J. M., 1994, "Using Response Surface Methodology to

H. and Watson, L. T., 1996, "Wing Design for a High-Speed Achieve a Robust Process," Journal of Quality Technology, Vol.

Civil Transport Using a Design of Experiments Methodology," 26(4), pp. 248-260.

6th AIAA/USAF/NASA/ISSMO Symposium on Multidisci- Matheron, G., 1963, "Principles of Geostatistics," Economic

plinary Analysis and Optimization, Bellevue, WA, AIAA, Inc., Geology, Vol. 58, pp. 1246-1266.

Vol. 1, pp. 168-183. Mavris, D., Bandte, O. and Schrage, D., 1996, "Application of

Giunta, A. A., Dudley, J. M., Narducci, R., Grossman, B., Haftka, Probabilistic Methods for the Determination of an Economically

R. T., Mason, W. H. and Watson, L. T., 1994, "Noisy Robust HSCT Configuration," 6th AIAA/USAF/NASA/ISSMO

Aerodynamic Response and Smooth Approximations in HSCT Symposium on Multidisciplinary Analysis and Optimization,

Design," 5th AIAA/USAF/NASA/ISSMOSymposium on Multi- Bellevue, WA, AIAA, Inc., Vol. 2, pp. 968-978.

disciplinary Analysis and Optimization, Panama City, FL, Vol. Mavris, D. N., Bandte, O. and Schrage, D. P., 1995, "Economic

2, pp. 1117-1128. Uncertainty Assessment of an HSCT Using a Combined Design

Healy, M. J., Kowalik, J. S. and Ramsay, J. W., 1975, "Airplane of Experiments/Monte Carlo Simulation Approach," 17th

Engine Selection by Optimization of Surface Fit Annual Conference of International Society of Parametric

Approximations," Journal of Aircraft, Vol. 12(7), pp. 593-599. Analysts, San Diego, CA.

Hong, J., Dagli, C. H. and Ragsdell, K. M., 1994, "Artificial Neural Mitchell, T., Morris, M. and Ylvisaker, D., 1988, "Existence of

Network and the Taguchi Method Application for Robust Smoothed Stationary Processes on an Interval," Working Paper.

Wheatstone Bridge Design," Advances in Design Automation Montgomery, D. C., 1997, Design and Analysis of Experiments,

(Gilmore, B. J., Hoeltzel, D. A., et al., eds.), Minneapolis, MN, Fourth Edition, John Wiley & Sons, New York.

ASME, Vol. 69-2, 37-41. Montgomery, D. C. and Evans, D. M., Jr., 1975, "Second-Order

Hunter, J. S., 1985, "Statistical Design Applied to Product Response Surface Designs in Computer Simulation," Simulation,

Design," Jrnl. of Quality Technology, Vol. 17(4), pp. 210-221. Vol. 29(6), pp. 169-178.

Hunter, J. S. and Naylor, T. H., 1970, "Experimental Designs for Myers, R. H., Khuri, A. I. and Carter, W. H., 1989, "Response

Computer Simulation Experiments," Management Science, Vol. Surface Methodology: 1966-1988," Technometrics, Vol. 31(2),

16(7), pp. 422-434. pp. 137-157.

Journel, A. G., 1986, "Geostatistics: Models and Tools for the Myers, R. H. and Montgomery, D. C., 1995, Response Surface

Earth Sciences," Math. Geol., Vol. 18(1), pp. 119-140. Methodology: Process and Product Optimization Using

Designed Experiments, John Wiley & Sons, New York.

Peplinski, J. D., Allen, J. K. and Mistree, F., 1996, "Integrating Shoemaker, A. C., Tsui, K. L. and Wu, J., 1991, "Economical

Product Design with Manufacturing Process Design Using the Experimentation Methods for Robust Design," Technometrics,

Robust Concept Exploration Method," 1996 ASME Design Vol. 33(4), pp. 415-427.

Theory and Methodology Conference (Wood, K., ed.), Irvine, Stanley, D. O., Unal, R. and Joyner, C. R., 1992, "Application of

CA, Paper No. 96-DETC/DTM-1502. Taguchi Methods to Propulsion System Optimization for SSTO

Phadke, M. S., 1989, Quality Engineering using Robust Design, Vehicles," Spacecraft and Rockets, Vol. 29(4), pp. 453-459.

Prentice Hall, Englewood Cliffs, New Jersey. Sundaresan, S., Ishii, K. and Houser, D. R., 1993, "A Robust

Ramberg, J. S., Sanchez, S. M., Sanchez, P. J. and Hollick, L. J., Optimization Procedure with Variations on Design Variables and

1991, "Designing Simulation Experiments: Taguchi Methods Constraints," Advances in Design Automation, Albuquerque,

and Response Suface Metamodels," Proceedings of the 1991 NM, ASME, Vol. 65-1, pp. 379-386.

Winter Simulation Conference (WSC) (Nelson, B. L., Kelton, W. Unal, R., Lepsch, R. A., Engelund, W. and Stanley, D. O., 1996,

D., et al., eds.), Phoenix, AZ, IEEE, pp. 167-176. "Approximation Model Building and Multidisciplinary Design

Reddy, S. Y., 1996, "HIDER: A Methodology for Early-Stage Optimization Using Response Surface Methods," 6th

Exploration of Design Space," 1996 ASME Design Automation AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary

Conference (Dutta, D., ed.), Irvine, CA, ASME, New York, Paper Analysis and Optimization, Bellevue, WA, AIAA, Inc., Vol. 1,

No. 96-DETC/DAC-1089. pp. 592-597.

Renaud, J. E. and Gabriele, G. A., 1993, "Improved Coordination in Unal, R., Stanley, D. O., Engelund, W. and Lepsch, R., 1994a,

Non-Hierarchic System Optimization," AIAA Journal, Vol. "Design for Quality using Response Surface Methods: An

31(12), pp. 2367-2373. Alternative to Taguchi's Parameter Design Approach,"

Renaud, J. E. and Gabriele, G. A., 1994, "Approximation in Engineering Management Journal, Vol. 6(3), pp. 40-48.

Nonhierarchic System Optimization," AIAA Journal, Vol. 32(1), Unal, R., Stanley, D. O. and Joyner, C. R., 1992, "The Role of

pp. 198-205. Statistically Designed Experiments in Conceptual Aerospace

Renaud, J. E. and Gabrielle, G. A., 1991, "Sequential Global Vehicle Design," 1992 IEEE International Engineering

Approximation in Non-Hierarchic System Decomposition and Management Conference, Eatontown, NJ, ASEM, pp. 211-214.

Optimization," Advances in Design Automation, Design Unal, R., Stanley, D. O. and Joyner, C. R., 1994b, "Parametric

Automation and Design Optimization, Miami, FL, ASME, Vol. Model Building and Design Optimization Using Reponse Surface

32-1, pp. 191-200. Methods," Journal of Parametrics, Vol. 14(1), pp. 81-96.

Ross, P. J., 1988, Taguchi Techniques for Qualtiy Engineering, Unal, R., Wu, K. C. and Stanley, D. O., 1994c, "A Study of Design

McGraw-Hill, New York, NY. of Experiments Methods for Design Optimization and Quality

Roux, W. J., Stander, N. and Haftka, R. T., 1996, "Response Improvement," 15th Annual Conference of the ASEM, ASEM.

Surface Approximations for Structural Optimization," 6th Venter, G., Haftka, R. T. and Starnes, J. H., Jr., 1996,

AIAA/USAF/NASA/ISSMO Symp. on Multidisciplinary Analysis "Construction of Response Sufaces for Design Optimization

and Optimization, Bellevue, WA, AIAA, Inc., Vol. 1, pp. 565- Applications," 6th AIAA/USAF/NASA/ISSMO Symposium on

578. Multidisciplinary Analysis and Optimization, Bellevue, WA,

Rowell, L. F., Braun, R. D., Olds, J. R. and Unal, R., 1996, "Recent AIAA, Inc., Vol. 1, pp. 548-564.

Experiences in Multidisciplinary Conceptual Design Welch, W. J., Buck, R. J., Sacks, J., Wynn, H. P., Mitchell, T. J.

Optimization for Launch Vehicles," 6th AIAA/USAF/NASA/ and Morris, M. D., 1992, "Screening, Predicting, and Computer

ISSMO Symposium on Multidisciplinary Analysis and Experiments," Technometrics, Vol. 34(1), pp. 15-25.

Optimization, Bellevue, WA, AIAA, Inc., Vol. 1, pp. 642-654. Welch, W. J., Yu, W. K., Kang, S. M. and Sacks, J., 1990,

Rumelhart, D. E., Widrow, B. and Lehr, M. A., 1994, "The Basic "Computer Experiments for Quality Control by Parameter

Ideas in Neural Networks," Communications of the ACM, Vol. Design," Journal of Quality Technology, Vol. 22(1), pp. 15-22.

37(3), pp. 87-92. Wujek, B., Renaud, J. E., Batill, S. M. and Brockman, J. B., 1996,

Sacks, J., Schiller, S. B. and Welch, W. J., 1989a, "Designs for "Concurrent Subspace Optimization Using Design Variable

Computer Experiments," Technometrics, Vol. 31(1), pp. 41-47. Sharing in a Distributed Computing Environment," Concurrent

Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P., 1989b, Engineering: Research and Appl., Vol. 4(4), pp. 361-378.

"Design and Analysis of Computer Experiments," Statistical Wujek, B. A. and Renaud, J. E., 1997, "A Concurrent Engineering

Science, Vol. 4(4), pp. 409-435. Approach for Multidisciplinary Design in a Distributed

Sellar, R. S. and Batill, S. M., 1996, "Concurrent Subspace Computing Environment," Proceedings in Applied Mathematics,

Optimization Using Gradient-Enhanced Neural Network Philadelphia, PA, SIAM, Vol. 80, pp. 189-208.

Approximations," 6th AIAA/NASA/USAF Multidisciplinary Yerramareddy, S., Tcheng, D. K., Lu, S. C.-Y. and Assanis, D. N.,

Analysis & Optimization Symposium, Bellevue, WA, AIAA, 1992, "Creating and Using Models for Engineering Design,"

Vol. 1(of 2), pp. 319-330. IEEE Expert, Vol. 7(3), pp. 52-59.

Sellar, R. S., Batill, S. M. and Renaud, J. E., 1996a, "A Neual Yu, J.-C. and Ishii, K., 1993, "A Robust Optimization Method for

Network-Based, Concurrent Subspace Optimization Approach to Systems with Significant Nonlinear Effects," Advances in

Multidisciplinary Design Optimization," 34th AIAA Aerospace Design Automation, Albuquerque, NM, ASME, Vol. 65-1, pp.

Sciences Meeting, Reno, NV, AIAA, AIAA 96-0714. 371-378.

Sellar, R. S., Stelmack, M. A., Batill, S. M. and Renaud, J. E.,

1996b, "Multidisciplinary Design and Analysis of an Integrated

Aeroelastic/Propulsive System," 37th AIAA/ASME/ASCE/

AHS/ASC Structures, Structural Dynamics, and Materials

Conference, Salt Lake City, UT, AIAA, pp. 583-593.