Journal of Simulation (2009) 0, 112

r 2009 Operational Research Society Ltd. All rights reserved. 1747-7778/09

www.palgrave-journals.com/jos/

Restricted subset selection and its application

E Jack Chen
BASF Corporation, Rockaway, New Jersey, USA
This paper develops procedures for selecting a set of normal populations with unknown means and unknown variances
such that the nal subset of selected populations satises the following requirement: with probability at least P*, the
selected subset will contain a population or only and all of those populations whose mean or means are within a value
of d* from the smallest mean. The size of the selected subset is random; however, at most, m populations will be chosen.
A restricted subset attempts to exclude populations that deviate more than d* from the smallest mean. Here P*, d*, and
m are user-specied parameters. These procedures can be used when the unknown variances across populations are
unequal. We then extend the sequentialized procedure to perform a selection with constraints. An experimental
performance evaluation demonstrates the validity and efciency of these restricted-subset-selection procedures.
Journal of Simulation advance online publication, 13 November 2009; doi:10.1057/jos.2009.18
Keywords: simulation; statistical analysis; ranking and selection; sample size allocation

1. Introduction
Discrete-event simulation has been widely used to compare
alternative system designs or operating policies. When
evaluating k alternative system designs, we select one or
more systems as the best and control the probability that the
selected systems really are the best. Let mi denote the
expected response of system i. Our goal is to nd the system
with the smallest expected response m* min1pipkmi. If the
system with the biggest expected response is desired, just
replace min with max in the formula. We achieve this goal by
using a class of ranking and selection (R&S) procedures.
Most R&S procedures, for example Dudewicz and Dalal
(1975) or Rinotts (1978) indifference-zone selection procedures, have focused on identifying the best system. Nevertheless, Koenig and Law (1985) have developed a two-stage
indifference-zone procedure to select a subset of size m
containing the v best of k systems; where (1pvpmok).
A restricted subset attempts to exclude populations that
deviate more than d* from the smallest mean. Extending the
work of Gupta and Santner (1973), Sullivan and Wilson
(1989) have developed a two-stage restricted-subset-selection
procedure, denoted VE, that determines a subset of maximum size m that contains at least one system that is within
a pre-specied amount of the best. In this paper, we derive
the probability of correct selection of determining a restricted subset based on the distribution of order statistics in
a clear and concise manner. Furthermore, the proposed
restricted-subset-selection procedures have the capability to
determine subsets of maximum size m that contain only and
Correspondence: E Jack Chen, BASF Corporation, 333 Mount Hope
Avenue, Rockaway, NJ 07866, USA.

all of those systems that are within a pre-specied amount

of the best.
In many practical situations, we need to select systems
based on multiple criteria (attributes, objectives, or performance measures). Researchers have developed procedures to
address this problem. Butler et al (2001) combine the
multiple attribute utility theorem with the Rinott (1978)
procedure to handle multiple performance measures. Before
performing the selection, users transform multiple performance measures into one utility score. The procedure then
nds the system that gives the highest utility. Chen and Lee
(2009) incorporate the indifference-zone approach into
multi-objective selection to nd non-dominated solutions,
that is the non-dominated Pareto set. A solution is called
Pareto-optimal if there exists no other solution that is better
in all criteria. Andradottir and Kim (2007) develop procedures to nd the system with the best primary performance
measure in the presence of a stochastic constraint on a
secondary performance measure, that is selection with
constraints. Morrice and Butler (2006) extend the multiple
attribute utility theorem to perform selection with constraints. There are no hard constraints in selecting the Pareto
set; however, in selection with constraints, systems that do
not satisfy the constraints will be removed from further
consideration.
Selection procedures that are developed based on the
Least Favorable Conguration (LFC, see the Indifferencezone selection section) are conservative and become
inefcient when the number of systems is large. Newer
approaches take into account the difference of sample
means, which can signicantly increase the efciency of
selection procedures. Hence, we incorporate this approach
into subset selection procedures as well.

2 Journal of Simulation Vol. ] ], No. ] ]

The paper is organized as follows. In the Background

section, we provide the context necessary to understand the
proposed procedures. In the Methodology section, we
present our methodologies and the proposed procedures for
selecting a subset that contains up to m d*-near-best systems.
Furthermore, we show how these procedures can be extended when there are multiple objectives. In the Empirical
Experiment section, we show our empirical-experiment
results. In the Conclusion section, we give concluding
remarks. An earlier version of this paper appears in Chen
(2008).

2. Background
First, some notation:
Xij:
Ni:
mi:
 i:
X
s2i :

S2i (Ni):

the observations from the jth replication or batch

of the ith system
the number of replications or batches for system i
the expected performance measure for system i,
that is mi E(Xij)
the sample mean performance measure for system
P i
i, that is N
j1 Xij =Ni
the variance of the observed performance measure
of system i from one replication or batch, that is
s2i Var(Xij)
the sample variance of system i with
Ni replications or batches, that is S2i Ni
PNi
 2
j1 Xij  Xi =Ni  1.

2.1. Indifference-zone selection

Let mil be the lth smallest of the mis, so that
mi1 pmi2 p . . . pmik . In unrestricted subset selection, the goal
is to select the v systems with the smallest expected
responses, that is systems i1 through iv. Let CS denote the
event of correct selection. In a stochastic simulation, a CS
can never be guaranteed with certainty. The probability of
CS, denoted by P(CS), is a random variable depending on
sample sizes and other uncontrollable factors. Moreover, in
practice, if the difference between miv and miv 1 is very small,
we might not care if we mistakenly choose system iv 1,
whose expected response is miv 1 . The practically signicant
difference d* (a positive real number) between a desired
and a satisfactory system is called the indifference zone in
statistical literature, and it represents the smallest difference
that we care about. Therefore, we want a procedure that
avoids making a large number of replications or batches to
resolve differences less than d*. That means we want
P(CS)XP* provided that miv 1  miv Xd  , where the minimal
CS probability P* and the indifference amount d* are both
specied by the users.
The two-stage procedure of Dudewicz and Dalal (1975)
to select the best of k systems has been widely studied

and applied. They perform the selection on the weighted

 (1)
 (2) for i 1, 2, y , k.
sample means Xi Wi1X
i Wi2Xi
Here, the weights
v
!3
u
2

u
n0
Ni
Ni  n0 d 5
1
Wi1 41 t1 
Ni
n0
h21 S2i n0
2

and Wi2 1Wi1, for i 1, 2, y, k. The expression for Wi1

was chosen to guarantee (Ximi)/(d*/h1) have a t distribution
with n01 degrees of freedom (df, see Dudewicz and Dalal,
1975). Note that h1 (which depends on k, n0, and P*) is a
constant that can be found from the tables in Koenig and
Law (1985) or Law and Kelton (2000). The procedure
proceeds as follows.
1. Simulate the initial n0 samples for all systems. Compute
the rst-stage sample means
n0
1 X
1
Xi
Xij
n0 j1

and sample variances

n0
P

S2i n0

j1

1
Xij  Xi 2

n0  1

for i 1, 2, y, k.
2. Compute the required sample sizes
l
m
i maxn0 1; h1 Si n0 =d 2 ;

for i 1; 2; . . . ; k
where Jzn is the smallest integer that is greater than or
equal to the real number z.
3. Simulate additional Nin0 samples.
4. Compute the second-stage sample means
2
Xi

Ni
X
1
Xij
Ni  n0 jn 1
0

5. Compute the weighted sample means

1
2
X~i Wi1 Xi Wi2 Xi

and select the system with the smallest Xi.

2.2. Selecting a subset

In this section, we derive the required sample sizes for
selecting a subset of size m that contains at least c of the v
best of k systems. Note that c, v, and m are user-specied
parameters. Under the LFC, mi1 mi2 . . . miv and

ChenRestricted subset selection and its application 3

miv d miv 1 . . . mk , the procedure guarantees with

probability P* that the selected subset of size m contains at
least c of systems il for l 1, 2, y, n.
Let Y be a random variable with probability density
function (pdf) f and cumulative distribution function (cdf) F.
Hogg and Craig (1995, p 198) show that the distribution of
the uth order statistics of m observations of Y is
gm;u yu bFyu ; u; m  u 1fyu

The value of h is determined such that, under the LFC,

P[TcpTu h] P*. Let t TcTu, then P[tph] P*. That
is, under the LFC, the value of h is the P* quantile of the
distribution of t. The required sample sizes Ni for subset
selection is then computed by (1) with h1 replaced with h.
After simulating Ni samples for systems i 1, 2, y, k, the
procedure returns the m systems that have the smallest
weighted sample means.

where
Ga b a1
bx; a; b
x 1  xb1
GaGb
is the beta distribution with shape parameters a and b and
G(a) is the gamma function. Note that G(a) (a1)! for any
positive integer a. Furthermore, the cdf of the uth order
statistics
Gm;u yu

Zyu

gm;u tdt

1

Let X[c] be the cth smallest weighted sample mean from X~il
for l 1, 2,y, n and let m[c] be its unknown true mean. Let
X[u] be the uth (u mc 1) smallest weighted sample mean
from X~il for l n 1,n 2,y, k and let m[u] be its unknown
true mean. To simply the notation, we use Xc, Xu, mc, and mu
instead of X[c], X[u], m[c], and m[u] in remainder of this section.
We can write
~ u
~ c oX
PCS PX


m m
P Tc oTu u  c
d =h
Z1
X
Gv;c tu hdGkv;mc 1 tu
1

In this case, f and F are functions of the t distribution with

n01df. The inequality follows as mumcXd* under the
indifference-zone approach. We equate the right-hand side to
P* and solve for h (which depends on c, v, m, k, n0, and P*).
In the case that c v, that is we are interested in selecting a
subset of size m containing the v best, the equation is
simplied to:
Z1
PCSX

1
Z1

Gv;v tu hgkv;mv 1 tu dtu

Ftu hv gkv;mv 1 tu dtu

3. Methodologies
In this section, we extend the subset selection procedures to
select a restricted subset. As with most selection procedures,
the proposed selection procedures require the input data to
be independent and identically distributed (iid) normal.
However, the variance can be different across systems. Many
performance measures of interest are taken over some
average of a sample path or a batch of samples. Thus, many
applications tend to have a normally distributed simulation
output. If the non-normality of the samples is a concern,
users can use batch means to manufacture samples that
appear to be iid normal, as determined by the tests
of independence and normality (see, for example, Chen
and Kelton, 2007). In the selection procedures described
below, the sampling operations can be carried out independently across systems. Hence, one can deploy the selection
procedures in a parallel and distributed environment.

3.1. Comparison with a control

We use the approach of Chen (2006) to derive the required
sample sizes for restricted subset selection. In comparison
with a control (alternative 0 with unknown mean m0), the
procedure guarantees:


Alternative i will be included with probability P

Alternative i will be excluded with probability P

Suppose the sample sizes are Ni and N0, respectively, for

systems i and 0 (which is designated as a control). The test at
condence level 1a of H0: miom0 d* against the
alternative H1: miXm0 d* is based on the test statistic
Xi  X0  d
Z q
s2i =Ni s20 =N0
The acceptance region for this test is Zpz1a, where
z1a is the 1a quantile of the standard normal distribution.
That is,

1

If m v 1, then the goal is to choose the best system.

If m4v 1, we are interested in choosing a subset of size m
containing the best. If m v41, we are interested in
choosing the m best systems.

when mi pm0
when m0 d omi

Xi  X0  d pz1a

q
s2i =Ni s20 =N0

Let F denote the standard normal cdf. If mim04d*, the

probability of committing a Type II error, which concludes

4 Journal of Simulation Vol. ] ], No. ] ]

that the null hypothesis is true when in fact it is false, is

b PXi  X0  d pz1a

q
s2i =Ni s20 =N0 

pFz1a 1  a
The inequality holds because Z has a standard normal
distribution when mim0 d*.
To control Type I errors, which occur when we do not
accept the null hypothesis when it is true, and Type II errors,
we use the test statistic
Xi  X0
Z q
s2i =Ni s20 =N0

Note that the constant h is the same here as that for

(unrestricted) subset selection. We then generate additional
Nin0 samples for system i in the second stage and compute
the weighted sample mean Xi. However, H d*/2 instead of
d* is used when computing the weight Wi1. Sort the weighted
sample means such that X~b1 pX~b2 p . . . pX~bk . Select system
bl if and only if X~bl pminX~bm ; X~b1 d  =2. We call this
two-stage restricted-subset-selection procedure RSS.

3.2. The rationale

The acceptance region for this test is still Zpz1a or

Xi  X0 pz1a

a two-stage procedure are

l
m
Ni maxn0 1; hSi n0 =H2 ; for i 1; 2; . . . ; k 2

q
s2i =Ni s20 =N0

Note that, with this test statistic, the probability of

accepting the null hypothesis that miom0 d* will be less
than 1a when m0omiom0 d* and decreases as mi deviates
more from m0. If mim04d*, the probability that the test
statistic falls in the acceptance region is

In this section, we provide the rationale of using H instead of

d* in (2) to determine the required sample sizes of restricted
subset selection.
Let wcu denote the one-tailed P* CI half width of mumc.
We conclude that the sample sizes allocated by (1) with
h1 h achieves wcupd*. Essentially, the indifference amount
d* in (1) corresponds to the upper bound of the desired CI
half width wcu. Hence, under the LFC (ie mc d* mu), the
sample sizes allocated by (1) ensures that

PCS PX~c oX~u 

PX~c  X~u  d omc  mu 

d
B
C
bpF@z1a  qA
2
2
si =Ni s0 =N0

XPX~c  X~u  wcu omc  mu 

P

0


For xed d* and a, b can be evaluated as a function of

sample sizes Ni and N0. For more detail, see Rice (1995).
Suppose we want to limit the probability of b. The sample
sizes Ni and N0 should be large enough such that

The last equality follows because of the property of the CI

half width. Note that if wcu4d*, then P(CS)oP*.
If we allocate the sample sizes by (2), then wcupH d*/2.
Let X~b1 pX~c denote the smallest weighted sample mean.
Then

d
z1a  q pzb
s2i =Ni s20 =N0

PX~b1 d =2oX~u 
XPX~c d =2oX~u 
XPX~c  wcu d oX~u 
XPX~c  X~u  wcu omc  mu 

Let a b 1P*. Then

d
2z1a p q
s2i =Ni s20 =N0
Hence, the condence interval (CI) half width
q
w0i z1a s2i =Ni s20 =N0 pd =2
Consequently, to control both Type I and Type II errors,
the allocated sample sizes should be large enough such that
the 1a one-tailed CI half width is less than H d*/2
when the difference between the two systems is at least d*.
Note that the indifference amount in (1) corresponds to
the targeted CI half width (Chen, 2009). The required
sample sizes for selecting a restricted subset (up to size m) in

P
Hence,
PX~u pX~b1 d =2p1  P
There is no more than 1P* probability that system u (ie
the system whose weighted sample means is the uth smallest
among system il for l n 1, n 2, y, k) will be included in
the restricted subset.
Under the conguration that mil mi1 d  for
l 2, 3, y, k (ie the LFC and v 1) and vom, the
probability of X~bm 1 pX~b1 d =2 will be less than
1P*. However, the probability of X~bl pX~b1 d =2 for
l 2, 3, y, m will be greater than 1P*. Note that under

ChenRestricted subset selection and its application 5

this conguration, the RSS procedure will allocate more

samples than those allocated by the VE procedure of Sullivan
and Wilson (1989). Under this conguration, the subset
determined by VE will, on average, contain (m1)/2 non-d*near-best systems (Sullivan and Wilson, 1989). Since the
precision becomes higher as sample sizes become larger and
RSS allocates more samples than VE for each system, the
subset determined by RSS will, on average, contain no
more than (m1)/2 non-d*-near-best systems under the same
conguration. Under the LFC and v m, the probability of
X~bm 1 pX~b1 d  =2 will be less than 1P*. In other words,
there is at least P* probability that all the selected systems in
the restricted subset are d*-near-best. However, the size of
the restricted subset may be less than m, that is some of the
d *-near-best systems may not be selected.
Note that the value of v is a user-specied parameter via
the constant of h. As the value of v increases so will the value
of h and the sample sizes; consequently, the average number
of the non-d*-near-best systems included in the subset will
decrease. In the special case that v m, the P(CS) is the
probability that only and all of the v d*-near-best system(s)
are included in the subset. For example, if mi1 d omi2 and
the constant h is obtained with v m(41) and P*, then the
procedure guarantees that the nal subset contains only
system i1 with probability P*. On the other hand, under the
LFC (ie mil d  min 1 , for l 1, 2, y, n) the procedure
guarantees that the nal subset contains system il for l 1,
2, y, n with probability P*.

3.3. Sequential procedure of selecting a restricted subset

We now present a cost-effective sequential approach, called
the Sequential Restricted Subset Selection (SRSS) procedure, to select a restricted subset of a size up to m. To
improve the efciency of the procedure, we take into account
the difference of sample means when computing the required
sample sizes. Let UXbv and LXbm 1 , respectively, be the
upper and lower P* condence limits of mbn and mbm 1 . Then

1plpv
maxd ; LXbm 1  Xbl
3
dbl
maxd ; Xbl  UXbv
v 1plpk
Let (x) max(0, x). The SRSS procedure is as follows.

3.4. SRSS procedure

 i,t
1. Let Ni,t be the sample size allocated for system i and X
be the sample mean of system i at the tth iteration.
Simulate n0 samples for all systems. Set the iteration
number t 0, and N1,t N2,t y Nk,t n0. Note that
for tX1, Ni,t can have different values for different i.
Specify the value of the indifference amount d* and the
required precision P*.
2. Calculate the sample means and sample variances. Rank
the sample means such that Xb1 pXb2 p . . . pXbk .

3. Calculate
the required
l
m sample size Nbl ;t 1 max
n0 ; 2hSbl Nbl ;t =dbl 2 , for l 1, 2, y, k. Here dbl is
computed according to (3).
4. If Ni,t 1pNi,t, for i 1, 2, y, k go to step 6.
5. Simulate additional J(Ni,t 1Ni,t) /2n samples for
system i. Set t t 1. go to step 2.
6. Select system bl iff Xbl pminXbm ; Xb1 d =2.
 i, instead of the weighted
Note that sample means X
sample means Xi, are used to determine the subset. This is
because there are more than two stages of sample means and
we can no longer use the approach of Dudewicz and Dalal
 i for
(1975) to compute the weighted sample means. While X
i 1, 2, y, k are still t-distributed, they have different df.
Hence, this sequential procedure is a heuristic. Nevertheless,
our empirical studies indicate that this procedure performs
well in terms of P(CS) and sample sizes. The critical value h
depends on c, v, m, k, n0, and P*. Even though the sample
sizes for each system change at each iteration, we use the
initial value of h through all iterations, thereby simplifying
the programming effort and providing conservative estimates of the sample sizes.
Let us consider the steps for taking additional samples,
that is steps 25, as one iteration. We can reduce the number
of iterations with a larger incremental sample size for system
i at the tth iteration, but we run the risk of allocating more
samples than necessary to non-promising systems. For
example, two-stage procedures allocate all the required
samples at the second stage based on the information
obtained at the rst stage. At the other extreme, some
selection procedures, for example Andradottir and Kim
(2007) take only one additional sample from each system
that is still under consideration at each iteration. While these
procedures generally require smaller sample sizes, they
require many iterations and incur the associated overhead.
We propose to use the sample size allocation strategy of
Chen and Kelton (2005) to compute the incremental sample
size dynamically with all the information obtained up to the
current iteration. The additional sample size for alternative i
at iteration t 1 is


di;t 1 Ni;t 1  Ni;t =2

 2i )r/(r1) to
Since we use the equation S2i (r) (Srj X2ij/rX
compute the variance estimator, we are only required to
PNi;t
PNi;t 2
Xit ; t1
Xit instead of the entire
store the triple Ni;t ; t1
sequence Xi1 ; Xi2 ; . . . ; XiNi;t .

3.5. Multi-objective selection

In this section, we show how the restricted-subset-selection
procedures can easily be extended to select a good system
when there are multiple selection criteria. We use the

6 Journal of Simulation Vol. ] ], No. ] ]

Bonferroni inequality (Law and Kelton, 2000, p 542) to

compute the required precision in each objective (performance measure). For example, if there are o objectives and
the desired overall P(CS) is P*, then the required P(CS) in
* 1(1P*)/o.
each objective is Po
Let mi[g] be the expected performance measure of the gth
objective of system i. When there are hard constraints, say
C[g] for some gA{1, 2, y, o}, systems i having mi[g]4C[g]
will be removed from consideration. To incorporate the
indifference-zone approach, C[g] can be written as
m0[g] d*[g], where m0[g] is the soft constraint and d*[g] is
the indifference amount of performance measure g. In
Andradottir and Kim (2007), this approach is represented by
the target value q[g] and the tolerance e[g], where e[g] d*[g]/
2 and q[g] m0[g] e[g]. It is ideal to select desirable systems i
having mi[g]om0[g]; however, it is acceptable if m0[g]pmi[g]om0[g] d*[g]. Hence, systems i having mi[g]om0[g] d*[g]
are feasible systems. With the indifference-zone approach,
desirable systems will be included in the subset with
high probability; the acceptable systems may be included
 i[g]
in the subset, but there is no probability guarantee. Let X
denote the sample mean of the gth performance measure
of system i. Let M[g] denote the selected subset based on
the gth performance measure. The proposed procedure
will include systems bl having Xbl gpminXbm g; m0 g
d g=2 in subset M[g]. Here m0[g] d*[g]/2 is used instead
of Xb1 g d g=2. This is analogous to Comparison with
the Standard vs. Comparison with a Control. The true
means of the standard m0[g] are known and do not need
to be estimated. Here M[g] | when m0 g d g=2o
Xb1 g.
The multi-objective procedure is also a sequential
procedure. For each performance measure, the critical
*. It is possible to
constant h depends on k, m, v, n0, and Po
have different m and v values for different performance
measures; hence, there are different h values for different
performance measures. If h is obtained with v 1, then
under the LFC the size of the restricted subset is no more
than (m 1)/2, that is on average (m1)/2 non-d*-near-best
systems will be included in the subset. On the other hand, if h
is obtained with v m, there is no more than 1P*
o
probability that any of the non-d*-near-best systems will
be included in the subset. However, this precision comes at a
cost: with a xed m, the value of h increases as the value of v
increases.

3.6. Comparison with the standard

In this section, we derive the required sample size for
selection with constraints based on comparisons with the
standard, where m0 is known and s20 0. Note that no
simulation is required for the standard. Suppose the sample
*)1/k. Then the sample
sizes are Ni for system i. Let P (Po

sizes should be large enough such that

(
PX~i pm0 d =2XP when mi pm0
PX~i 4m0 d =2XP when m0 d pmi
First, when mipm0,
PCS PX~i pm0 d =2
~

Xi  mi m0  mi
d =2
P 
p 

d =hs
d =hs
d =hs
XPTi phs =2
Fhs =2
Similarly, when m0 d *pmi,
PCS PX~i 4m0 d =2


m  X~i mi  m0 d =2
P i
o 
 
d =hs
d =hs
d =hs
XPTi phs =2
Fhs =2
If we set F(hs/2) P, then hs 2tP,n01, where t1a,f is the
1a quantile of the t-distribution with f df. Hence, the
required sample sizes are
l
m
Ni maxn0 1; hs Si n0 =d 2 ; for i 1; 2; . . . ; k
We apply this sample-size-allocation rule to each secondary performance measure. To take into account the
 i[g]) and U(X
 i[g]),
difference of sample means, let L(X
respectively, be the lower and upper P* condence limits
of mi[g] and

maxd g; LXi g  m0 g when Xi g4m0 g
di g
maxd g; m0 g  UXi g when Xi gpm0 g
5
Then let the sample size of system i with respect to
performance measure g be
l
m
Ni g maxn0 1; hs Si n0 =di g2 ; for i
1; 2; . . . ; k

With this sample size, the achieved CI half width

 i[g]p
w0i[g]Edi[g]/2Xd*[g]/2. If mi[g]m0[g]Xdi[g], then P[X
m0[g] di[g]/2]p1P. Consequently, for each performance
* condence that all
measure we have (approximately) Po
desirable systems survived the screening and for all performance measures we have (approximately) 1(o1)(1P*)/
o condence that all desirable systems survived the screening. As di[g] is computed based on the sample means (instead
of the true means), the theoretical P(CS) is not guaranteed
statistically. Furthermore, because the test to remove inferior
systems is performed at each iteration, the probability of
incorrectly removing a desirable system may be greater than

ChenRestricted subset selection and its application 7

1P. Note that the test statistics are correlated as earlier

samples are included in computing the subsequent test
statistics. Our results indicate that in this setting the
probability of incorrectly removing a desirable system is
generally less than 1P when the indifference amount is
greater than 10% of the standard error of the performance
measure.

3.7. Selection with constraints

In this section, we dene additional notations and formulate
a selection-with-constraints problem. We call this the
Selection-with-Constraints (SWC) procedure. Let Xij[g] be
an observation associated with the performance measure g
from replication (or batch) j of system i. Let g 1 be the
primary performance measure. The performance measures
are dened as mi[g] E [Xij[g]] for i 1, 2, y, k and
g 1, 2, y, o. The goal is to nd the best feasible system
arg min mi 1
i1; ...; k

s:t: mi gpCg for g 2; 3; . . . ; o

Recall that C[g] m0[g] d*[g].
Let M be the set of systems that are still under
consideration and is initialized to include all k systems.
Note that M will be changed from iteration to iteration. Let
 i[g] denote the sample mean of the gth performance
X
measure of system i. Let Xb1 1 mini2M Xi 1 and let
UXb1 g be the upper P* condence limits of mb1 1. Then
di 1 maxd 1; Xi 1  UXb1 1

For the primary performance measure, we are performing

an unrestricted selection with m v 1; hence, the sample
size will be computed according to (1) with d* replaced by
di[1]. For the secondary performance measures, we are
performing a Comparison with the Standard and the sample
size will be computed according to (6). If there is no more
than b (1P*)/o probability that a system that is infeasible with respect to performance measure g be declared
feasible, then there is no more than bj probability that
a system is infeasible with j different performance measures
be declared feasible. Consequently, with these sample sizes,
we have (approximately) P* condence that the selected
system is a feasible d*-near-best system.
To improve the efciency of the procedure, we also
compare systems that are still under consideration with
systems that are found to be feasible. Note that M contains
systems that are declared to be feasible and systems that
need more sampling to verify whether they are feasible.
Systems that are found to be infeasible are excluded from M.
Let S2i (g,Ni) denote the sample variance of the gth
performance measure of system i with sample size Ni. When
Xij[g] is iid normal, it is known that the random variable
 i[1]X
 j[1] (iaj) has approximately a t distribution
Yij X

with fij df, where

fij

S2i 1; Ni =Ni S2j 1; Nj =Nj 2

S2i 1; Ni =Ni 2 =1; Ni  1 S2j 1; Nj =Nj 2 =Nj  1
8

see Law and Kelton (2000, p 559) for details. Since fij
will not, in general, be an integer, interpolation will probably be necessary. The procedure will eliminate system j
 I(1) wij for some feasible system i.
 j(1)4X
such that X
q
Here wij tP;fij S2i 1; Ni =Ni S2j 1; Nj =Nj is the one1/k
) CI half width. The SWC procedure is
tailed P( (P*)
o
as follows.

4. Selection-with-constraints procedure
1. Let Ni,t be the sample size allocated for system i and
 i,t[g] be the sample mean of the gth performance
X
measure of system i at the tth iteration. Simulate n0
samples for all systems. Set the iteration number t 0,
and N1,t N2,t y Nk,t n0. Specify the value of
the indifference amount d*[g] for g 1, 2, y, o, the
soft constraints m0[g] for g 2, 3, y, o, and the
required precision P*. Let M be the set of systems that
are still under consideration and is initialized to include
all k systems. Compute P*
o 1(1P*)/o and
1/k
.
The
critical
constant
h1 is obtained with
P (P*)
o
k, n0, and P*.
o
2. Calculate sample means and sample variances for each
performance measure of each system. Obtain the index
 i[1]), that is the system having the
b1( arg miniAMX
smallest sample means of the primary performance
measure.
3. Calculate the new sample size Ni,t 1[1] max
(n0, J(h1Si(1,Ni,t)/di[1])2n), for iAM and Ni,t 1[g]
max(n0,J(2tP,Ni,t1Si(g,Ni,t)/di[g])2n), for iAM and
g 2, 3, y, o. Here di[1] is computed according
to (7), di[g], for g 2, 3, y, o, is computed according
to (5), and S2i (g,Ni,t) is the sample variance of the
gth performance measure of system i with sample
size Ni,t.
p
4. Let wi0 g tP;Ni;t 1 Si g; Ni;t = Ni;t . For each system
 i[g]4m0[g] wi0[g] for some g 2, 3, y, o,
iAM, if X
then remove system i from M. For each system iAM, if
 i[g]om0[g]wi0[g] for some g 1, 2, y, o, then set
X
Ni,t 1[g] Ni,t[g].
5. For each iAM having Ni,t 1[g] Ni,t[g], for g
2, 3, y, o, then for each jAM compute wij
q
 i[1]
 j[1]4X
tP;fi;j S2 1; Ni;t =Ni;t S2j 1; Nj;t =Nj;t . If X
i

wij, then remove system j from M.

6. If M |, then terminate the program. There is no
feasible solution.

8 Journal of Simulation Vol. ] ], No. ] ]

0

7. Let Ni,t 1 maxg 1, y, o Ni,t 1[g]. Let N i,t 1

maxg 1, y, o {Ni,t 1[g]Ni,t[g]|Ni,t 1[g]Ni,t[g]40}.
8. If Ni,t 1pNi,t, for i 1, 2, y, k, go to step 10.

0
9. Simulate additional min(Ni,t
1,J(Ni,t 1Ni,t) /2n)
samples for systems iAM. Set t t 1. Go to step 2.
 i[1].
10. Select system b1 miniAMX
In the case that there is only one secondary performance
measure, a tighter lower bound can be achieved. Let b1 be
the probability that an unacceptable system is included in the
subset and let b2 be the probability that the best feasible
system is not selected when compared to other feasible
systems in isolation. When b1 b2 b, Andradottir and
Kim (2007) show that
PCSX21  bk1=2  1  b P

In this experiment, there is only one performance measure,

and we use these procedures to select a restricted subset. We
choose the rst-stage sample size to be n0 20. The number
of systems under consideration is k 10. The indifference
amount d* is set to 1. The targeted size of the subset m is set
to 5. The minimal P(CS) of P* is set to 0.90 and 0.95.
Furthermore, we select the m best systems based on the
 is) instead of the weighted sample means
sample means (ie X

(ie Xis).
We tested the following two congurations:
 Setting 1: m1 0, and mj d*, for j 2, 3, y, k.
 Setting 2: mi 0, for i 1, 2, y, 5, and mj d*, for
j 6, 7, y, k.

If the goal is to select a subset or a restricted subset, then

different critical constant h can be used in step 3. We
then select a subset or a restricted subset instead of a single
best system in step 10. Instead of performing the feasibility
check and optimizing the primary performance measure
simultaneously, Andradottir and Kim (2007) have investigated a two-phase procedure that searches for feasible
systems before optimizing. Another alternative is to obtain a
restricted subset rst and then check for feasibility.

4.1. Multiple comparisons with the best (MCB)

Multiple comparisons provide simultaneous CIs on selected
differences among the systems. It is known that indifferencezone selection procedures also guarantee that the CI
coverage of MCB have the same condence level of the
selection procedures. These CIs bound the differences
between the performance of each system and the best
of the others with a pre-specied condence level. Let
q
wij tP;fij S2i Ni =Ni S2j Nj =Nj be the one-tailed P
(P*)1/(k1) CI half width of mimj; see Law and Kelton (2000,
p 557) on how to estimate fij. Then the MCB CIs are
Pmi  min mj 2 maxXi  Xj  wij  ;
jai

5.1. Restricted subset selection

jai

max Xi  Xj wij ; 8iXP

jai

Here (x) denotes min(0, x). See Chen (2006) on how

these MCB CIs are derived.

5. Empirical experiments
In this section, we present some empirical results of
performing restricted subset selection and SWC.

In setting 1, a correct selection means the selected subset

contains system 1. In setting 2, a correct selection means
none of the non-best systems (ie, systems 6 through 10) are
selected. Note that the size of the subset may be less than m.
We did not perform any experiments of the non-LFC, as it
has been shown that procedures that take into account the
difference of sample means can signicantly reduce the
required sample sizes.
The setting of variances is as follows.
 Equal Variances: s2i 62, for i 1, 2, y, k.
 Increasing Variances: s2i i2, for i 1, 2, y, k.
 Decreasing Variances: s2i (11i)2, for i 1, 2, y, k.
We perform 10 000 independent experiments to obtain the
actual P(CS). The number of times the selected subset
contains the desired systems is counted among the 10 000
independent experiments. The observed correct selection
proportion, PCS1, is then obtained by dividing this number
by 10 000.
We also compute PCS2, the proportion that the selected
subset contains only and all of the v d*-near-best system(s),
that is, in setting 1 it contains only system 1, and in setting 2
it contains systems 15. See Appendix on how to compute
the theoretical values of PCS1 and PCS2. We list the results
for RSS and SRSS. The results include PCS1, PCS2, the
average nal size of the subset m, the average sample size of
10 000 k
each simulation run T, that is T SR
1 Si 1NR,i/10 000,
NR,i is the total number of replications or batches for design i
in the Rth independent run, and the standard error of T.
Furthermore, the Iter row lists the average number of
iterations of SRSS.
Table 1 lists the results of setting 1, where system 1 is the
only best system. The v 1 and v 5 columns list the results
when the critical constant h is obtained with v 1 and v 5,
respectively, and m 5, k 10, n0 20. When h is obtained

ChenRestricted subset selection and its application 9

Table 1 Setting 1 and equal variances

P* 0.90

P* 0.95

v1

v5

v1

v5

RSS

PCS1
PCS2
m
T
std(T)

0.9813
0.4633
2.12
2967
303

1.0
0.9580
1.06
20947
2163

0.9973
0.5979
1.80
4986
514

1.0
0.9795
1.03
26214
2675

SRSS

PCS1
PCS2
m
T
std(T)
Iter

0.9866
0.4886
2.10
2664
310
10

1.0
0.9753
1.03
20864
248
13

0.9992
0.6314
1.74
4749
351
11

1.0
0.9912
1.01
26215
261
14

with v 1, that is the selected subset contains the best

system, the average size of the subset is 2 and the observed
PCS1s are close to 1. The selected subset contains only
system 1 (ie the best system) about 0.55 fraction of the time.
When h is obtained with v 5, that is the selected subset
contains ve d*-near-best systems, the average size of the
subset is close to 1. These procedures correctly determine
that alternatives other than system 1 are not d*-near-best.
Furthermore, the observed PCS1s are all 1, that is the
selected subset always contains system 1. The subset contains
only system 1 about 0.97 fraction of the time.
Table 2 lists the results of setting 2, where systems 15 are
the best systems. The observed P(CS)s are greater than the
nominal values. When h is obtained with v 1, the average
size of the subset is 3.45 (ie (3.11 3.70 3.18 3.78)/4).
When h is obtained with v 5, the average size of the subset
is 4.94 (ie (4.89 4.95 4.92 4.97)/4). These procedures
incorrectly determine some of the best systems as non-d*near-best. When h is obtained with v 1, the selected subset
contains systems 15 about 0.21 (ie (0.1254 0.2888
0.1355 0.3128)/4) fraction of the time. On the other hand,
with v 5, it contains systems 15 about 0.94 (ie (0.9132
0.9570 0.9323 0.9749)/4) fraction of the time. The
sequentialized procedure SRSS has better performance than
the two-stage procedure RSS in terms of P(CS) and sample
sizes. Moreover, the number of iterations initiated by the
SRSS procedure is small.
We did not list the results of the increasing or decreasing
variance congurations in the paper. Those results are
generally similar to these experiments when the variances are
equal.

5.2. Selection with constraints

In this experiment, we test the Selection-with-Constraints
procedure. In order to compare with other known procedures (eg AK and AK of Andradottir and Kim, 2007), we

use a similar setting in Andradottir and Kim (2007). The

number of systems under consideration k 25. There are
two performance measures, that is o 2. The primary
performance measure
8
0;
i 1; 2; . . . ; b  1
>
>
< 
ib
d ;
mi 1
0
i b 1; . . . ; b a
>
>
:
1  id ; i b a 1; . . . ; k
and the secondary performance measure
8
< e; i 1; 2; . . . ; b
mi 2 0
i b 1; . . . ; b a
:
e;
i b a 1; . . . ; k
The primary and secondary performance measures of each
system are independent. The variances of both performance
measures of each system are 1. The initial sample size n0 is
set to n0 20. Moreover, the indifference amount of the
primary performance measure d*[1] and the tolerance e are
p
set to 1= n0 . Furthermore, the soft constraint m0[2] e
and the indifference amount of the secondary performance
measure d*[2] 2e, thus, the hard constraint C[2] e. The
desired P(CS) is set to P* 0.95. We make 10 000
independent replications to obtain the observed P(CS). We
solve (9) with P* 0.95 to obtain bE0.002. Hence, instead
of setting P (1(1P*)/o)1/kE0.999 for (6), we set P
(10.002) 0.998. Furthermore, the critical constant h1
of (1) is obtained with 12 (ie (k1)/2) and Po
* 0.976 (ie
0
0.9981/k ).
Table 3 shows the observed P(CS) and the average of the
allocated sample sizes T of SWC, AK, and AK for
various numbers of acceptable systems a when k 25
and a b 13. For SWC, we also list the standard error
of T and the average number of iterations. The results of AK
and AK are extracted from Andradottir and Kim (2007)
directly; hence, the results of SWC are obtained with

10 Journal of Simulation Vol. ] ], No. ] ]

Table 2 Setting 2 and equal variances

P* 0.90

P* 0.95

v1

v5

v1

v5

RSS

PCS1
PCS2
m
T
std(T)

0.9331
0.1254
3.11
2967
303

1.0
0.9132
4.89
20947
2163

0.9761
0.2888
3.70
4986
514

1.0
0.9570
4.95
26214
2675

SRSS

PCS1
PCS2
m
T
std(T)
Iter

0.9456
0.1335
3.18
2570
265
10

1.0
0.9323
4.92
20850
264
13

0.9845
0.3128
3.78
4633
321
11

1.0
0.9749
4.97
26206
266
14

Table 3 The performance of selection with constraints

b 13

b 12

b 10

b7

b3

b1

a0

a1

a3

a6

a 10

a 12

SWC

PCS
T
std(T)
Iter

0.954
4971
445
13

0.955
4881
434
12

0.953
4702
416
12

0.959
4429
380
12

0.965
4058
351
11

0.972
3869
347
9

AK

PCS
T

0.973
4063

0.973
4109

0.973
4184

0.974
4319

0.975
4502

0.975
4604

AK

PCS
T
d
Iter

0.960
3976
140

0.962
3749
130

0.963
3726
130

0.963
3686
128

0.966
3615
125

0.968
3581
124

d row under AK
different random number streams. The Iter
lists the lower bound of the number of iterations. The
observed P(CS)s are all greater than the specied nominal
value. Procedure AK is a two-phase procedure; it performs the feasibility check on the secondary performance
measure before it performs the optimization on the primary
performance measure. On the other hand, AK and SWC
perform the feasibility check and optimization simultaneously. Even though the allocated sample sizes of SWC are
greater than those of AK , SWC requires far fewer
iterations and may take less runtime. For example, when
b 1 SWC terminates, on average, with nine iterations. On
the other hand, AK terminates, on average, with no less
than 124 (ie 3581/2520) iterations.

can be incorporated with other procedures to select

promising systems from large alternatives for follow-up
processing. Furthermore, these procedures can be extended
to perform multi-objective selection, such as selecting the
best feasible system.
Selection procedures that are developed based on the LFC
are conservative. Newer approaches utilize both the means
and variances from earlier stages; the marginal computational effort is minimal, yet the achieved improvement in
efciency is signicant. The sequential procedure preserves
the simple structure of indifference-zone selection while
being more efcient in situations where many alternative
systems are non-competitive. We strongly recommend the
use of the sequentialized version of the selection procedures
as they perform better than two-stage procedures in terms of
sample size and probability of correct selection.

6. Conclusions
We have presented two restricted-subset-selection procedures that provide effective means for screening a large set of
systems. These procedures are versatile, easy to apply, and

AcknowledgementsThe author thanks the referees and Alice Huang

for their comments, which improved both the content and exposition
of the paper.

ChenRestricted subset selection and its application 11

Appendix
Given v, m, k, n0, and P*, let h be the value such that
Z1

Gv;v t hgkv; mv 1 tdt P

1

The probability that the best system is selected in the

restricted subset is
Pc PX~i1 oX~b2  PX~i1 oX~bm 1   PX~i1 oX~b2 
Z1
Ft 2hgk1;1 tdt
PX~i1 oX~b1 d =2X
1

Let Pj be the probability that at least j(pv) of the v best

systems are selected in the unrestricted subset of size m. Then
Pj is the probability that the jth smallest sample mean of
systems i1, i2, yin is smaller than the (mj 1)th smallest
sample mean of systems in 1, in 2, y,ik.
From the discussion in the Selecting a subset section,
when sample sizes are computed according to (2)
"

#
X~j  mj X~u  mu
mu  mj
o

Pj P 
d =2h d =2h
d =2h
Z1

Gv;j t 2hgkv;mj 1 tdt

1

Note that Gv,j(t 2h) is used in the above equation

because H d*/2 is used to compute the required sample
sizes. For example, the probability that the unrestricted
subset contains at least one of the v best system is
Z1
Pj1 X

Gv;1 t 2hgkv;m tdt

1

In the setting that mi1 d mi2    mik , that is the

LFC and v 1, the probability that only the best system is
selected in the restricted subset is

01
1
Z1
Z
Ft 2hgk1;1 tdtA
@ Ft 2hgk1;m tdt
1

Z1

In the setting that mi1    mim and mim d mim 1

   mik , that is, the LFC and v m, the probability that all
of the v best systems are selected in the restricted subset is
Pd Pjm PX~bl oX~b1 d =2; for l 2; 3; . . . ; m
Z1
Z1
Gm;m t 2hgkm;1 tdt
Gm;m t hgm;1 tdt
X

1
Z1

1

Ft 2hm gkm;1 tdt

1

Z1

Ft hm gm;1 tdt

1

The probability that at least one of the best v( m)

systems are selected in the restricted subset is the probability
that the smallest sample mean of systems i1, i2, y, im is
smaller than the mth smallest sample mean of systems
im 1, im 2, y, ik and is smaller than d*/2 plus the smallest
sample mean of systems im 1, im 2, y, ik. Hence, when
mpkm
Z1

1
Z1

Ft 3hgk1;1 tdt

1

Pe X

Pb PX~i1 d =2oX~b2 
"
#
X~i1  mi1 X~b2  mb2
mb2  mi1
d =2
P 
o 


d =2h
d =2h
d =2h d =2h


mb2  mi1
P Ti1 oTb2 
h
d =2h

Z1 
mb2  mi1
 h gk1;1 tdt
F t 

d =2h

1

1
Z1
1
Z1

Gm;1 t 2hgkm;m tdt

Z1

Gm;1 t 3hgkm;1 tdt

1

1  Ft  2hm gkm;m tdt

1  Ft  3hm gkm;1 tdt

1

References
Ft hgk1;1 tdt

1

Note that if the event X~i1 d =2oX~b2 is true, then the

event X~i1 oX~b2 is true. Furthermore, X~b1 X~i1 , mi2 pmb2 , and
mi2  mi1 Xd .

Andradottir S and Kim SH (2007). Fully sequential procedures for

comparing constrained systems via simulation. Submitted.
Butler J, Morrice DJ and Mullarkey PW (2001). A multiple
attribute utility theory approach to ranking and selection. Mngt
Sci 47: 800816.
Chen EJ (2006). Comparison with a standard via all-pairwise
comparisons. J Discrete Event Dyn S 16(3): 385403.

12 Journal of Simulation Vol. ] ], No. ] ]

Chen EJ (2008). Restricted subset selection. In: Mason SJ, Hill RR,
Moench L, Rose O, Jefferson T and Fowler JW (eds).
Proceedings of the 2008 Winter Simulation Conference, IEEE;
Piscataway, NJ, pp 281289.
Chen EJ (2009). Subset selection procedures. J Simulation. To
Appear.
Chen EJ and Kelton WD (2005). Sequential selection procedures:
Using sample means to improve efciency. Eur J Opl Res 166(1):
133153.
Chen EJ and Kelton WD (2007). A procedure for generating batchmeans condence interval for simulation: Checking independence and normality. Simulation: Trans of the Soc for the
Modeling and Simul International 83: 683694.
Chen EJ and Lee LH (2009). A multi-objective selection procedure
of determining a Pareto set. Vienna, Vol. 1., pp. 478486.
Comput Opns Res 36: 18721879.
Dudewicz EJ and Dalal SR (1975). Allocation of observations
in ranking and selection with unequal variances. Sankhya B37:
2878.
Gupta SS and Santner TJ (1973). On selection and ranking
proceduresA restricted subset selection rule. Proceedings of the
39th Session of the International Statistical Institute, Vol. 1.,
Vienna, pp 478486.

Hogg RV and Craig AT (1995). Introduction to Mathematical

Statics, 5th edn. Prentice Hall: New Jersey.
Koenig LW and Law AM (1985). A procedure for selecting a
subset of size m containing the l best of k independent
normal populations. Commun StatSimul and Comput B 14:
719734.
Law AM and Kelton WD (2000). Simulation Modeling and
Analysis, 3rd edn. McGraw-Hill: New York.
Morrice DJ and Butler JC (2006). Ranking and selection with
multiple targets. In: Perrone LF, Wieland FP, Liu J, Lawson
BG, Nicol DM and Fujimoto RM (eds). Proceedings of the
2006 Winter Simulation Conference, Institute of Electrical and
Electronics Engineers; Piscataway, NJ, pp 222230.
Rice JA (1995). Mathematical Statistics and Data Analysis, 2nd edn.
Duxbury Press: Belmont, CA.
Rinott Y (1978). On two-stage selection procedures and related
probability inequalities. Commun Stat A 7: 799811.
Sullivan DW and Wilson JR (1989). Restricted subset selection
procedures for simulation. Opns Res 37: 5271.

Received 6 January 2009

accepted 18 August 2009 after one revision