IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO.

2, JUNE 2007 223

A Comprehensive Empirical Study of Count Models

for Software Fault Prediction
Kehan Gao, Member, IEEE, and Taghi M. Khoshgoftaar, Member, IEEE

Abstract—Count models, such as the Poisson regression model, predicted probability

and the negative binomial regression model, can be used to obtain expected value
software fault predictions. With the aid of such predictions, the de-
velopment team can improve the quality of operational software. variance
The zero-inflated, and hurdle count models may be more appro- mean value of the dependent variable for
priate when, for a given software system, the number of modules
with faults are very few. Related literature lacks quantitative guid- module
ance regarding the application of count models for software quality , and parameter and/or parameter vectors to be
prediction. This study presents a comprehensive empirical investi- estimated
gation of eight count models in the context of software fault predic- log-likelihood function
tion. It includes comparative hypothesis testing, model selection,
and performance evaluation for the count models with respect to probability of a module being perfect
different criteria. the selected threshold value
The case study presented is that of a full-scale industrial soft-
ware system. It is observed that the information obtained from hy- likelihood ratio statistic
pothesis testing, and model selection techniques was not consistent chi-square distribution
with the predictive performances of the count models. Moreover,
the comparative analysis based on one criterion did not match that ratio of predicted probabilities of the two models
of another criterion. However, with respect to a given criterion, the mean of
performance of a count model is consistent for both the fit, and test
data sets. This ensures that, if a fitted model is considered good standard deviation of
based on a given criterion, then the model will yield a good predic- Vuong’s statistic
tion based on the same criterion. The relative performances of the
eight models are evaluated based on a one-way ANOVA model, and the maximum value that the dependent variable
Tukey’s multiple comparison technique. The comparative study is takes
useful in selecting the best count model for estimating the quality observed frequency (i.e., the fraction of the
of a given software system. sample with )
Index Terms—ANOVA, count models, hypothesis testing, infor- predicted frequency
mation criteria, Pearson’s chi-square, software metrics, software
-significance level
quality, Tukey’s multiple comparison.
-score in the table
number of mutually exclusive cells into which
NOTATION the range of is divided
module (observation) identifier SS sum of squares
number of modules (observations) MS mean squares
vector of independent variables for module df degree of freedom
vector of [1, ] F statistic
dependent variable for module
predicted value of the dependent variable for ACRONYM1
module AAE average absolute error
probability density function AIC Akaike information criterion
ANOVA analysis of variance
ARE average relative error
Manuscript received January 2003; revised September 2004, November 2004,
and January 2005; accepted June 2005. This work was supported in part by NSF
BIC Bayesian information criterion
Grant CCR-9970893. Associate Editor: M. Vouk. CAIC consistent Akaike information criterion
K. Gao is with the Department of Mathematics and Computer Science,
Eastern Connecticut State University, Willimantic, CT 06226 USA. EM expectation-maximization
T. M. Khoshgoftaar is with theEmpirical Software Engineering Laboratory,
Department of Computer Science and Engineering, Florida Atlantic University,
IC information criterion
Boca Raton, FL 33431 USA (e-mail: taghi@cse.fau.edu).
Digital Object Identifier 10.1109/TR.2007.896761 1The singular and plural of an acronym are always spelled the same.

0018-9529/$25.00 © 2007 IEEE

224
LR likelihood ratio
HNB1 hurdle model with threshold 1, and negative
binomial distribution
HNB2 hurdle model with threshold 2, and negative
binomial distribution
HP1 hurdle model with threshold 1, and Poisson
distribution
HP2 hurdle model with threshold 2, and Poisson
distribution
MLE maximum likelihood estimation
MSD minimum -significant difference
MSE mean squared error
NBRM negative binomial regression model
PDF probability density function
PRM Poisson regression model
ZIP zero-inflated Poisson
ZINB zero-inflated negative binomial
I. INTRODUCTION
IVEN the goal of delivering a software product that has
G minimal corrective maintenance, software quality mod-
eling techniques are useful tools for applying timely software
quality improvement efforts. For example, by predicting the
number of faults in program modules, a software fault predic-
tion model can direct the software quality assurance team in
targeting the most faulty modules first.
The relationship between software complexity metrics, and
the occurrence of faults in program modules has been used
by software quality prediction models, such as case-based
reasoning [1], regression trees [2], [3], fuzzy logic [4], and
multiple linear regression [5]. Typically, a software quality
model of a given software system is calibrated using software
metrics, and fault data collected from a previous system release
or similar projects. The model can then be applied to predict
the software quality of a release currently under development,
or similar projects.
Software fault prediction based on count model techniques
[6] is attractive because a specific count model can be chosen
such that it best represents the fault occurrence process of the
given software system. In addition, they have a unique feature
in that count models can provide the probability that a given
number of faults will occur in any given program module. Count
models can also be used for software quality classification of
program modules, i.e., fault-prone, and not fault-prone [7], [8].
We feel that the application of count models in software engi-
neering has been very limited [6], [9].
Poisson regression is the basis of the various count models
which are derived from it. It has a statistical characteristic called
equidispersion, which implies the equality of the mean to the
variance of the dependent variable (a non-negative discrete in-
teger). In the software fault prediction problem, it is often ob-
served that the distribution of the number of faults (dependent
variable) is such that its variance exceeds its mean value. Such a
scenario is referred to as overdispersion. One way of accounting
for overdispersion is introducing an unobserved heterogeneity
IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO. 2, JUNE 2007
term (gamma distributed), yielding the negative binomial re-
gression model (NBRM).
Another variation of count models are zero-inflated, and
hurdle models, which are used when there is an excess of zeros
for the dependent variable. The proportion of faulty modules
of a high assurance software system (such as telecommu-
nications) is usually very small. Under such scenarios, the
zero-inflated, and hurdle count models may be more appro-
priate. The zero-inflated model assumes that the population
of the software modules consists of two groups, i.e., perfect
modules, and non-perfect modules. For the perfect modules,
no faults occur; while for the non-perfect modules, the number
of faults follows some standard distribution, such as Poisson,
or negative binomial. If the non-perfect group is assumed to
follow a Poisson distribution, then a zero-inflated Poisson (ZIP)
model is obtained. If the non-perfect group is assumed to follow
a negative binomial distribution, then a zero-inflated negative
binomial (ZINB) model is obtained.
The hurdle model also consists of two parts. However, in-
stead of having perfect, and non-perfect groups, it divides the
modules into lower, and higher count groups, based on a binary
distribution. The dependent variable of these groups is assumed
to follow a separate distribution process. In the case of hurdle
count models, two factors may affect its predictive quality: the
dependent variable’s threshold2 value that is used to form the
two groups, and the specific distribution each group is assumed
to follow. We investigated these two factors to study their impact
on the prediction performances of hurdle models. For the case
study presented, we chose two threshold values for the number
of faults: 1, and 2.
The distributions chosen for the count groups of the hurdle
models are the Poisson, and negative binomial distributions.
Hence, we have four kinds of hurdle models: 1) hurdle model
with threshold 1, and Poisson distribution (HP1); 2) hurdle
model with threshold 2, and Poisson distribution (HP2); 3)
hurdle model with threshold 1, and negative binomial dis-
tribution (HNB1); and 4) hurdle model with threshold 2, and
negative binomial distribution (HNB2). Previous works related
to hurdle models were focused on applying HP1, and HNB1 in
econometrics [10]–[12]. The application of hurdle regression
techniques with different threshold values to software quality
estimation modeling is a contribution of this study. To our
knowledge, this is the first study that has investigated hurdle
models for software quality estimation.
In related literature, we did not find any comprehensive em-
pirical study that provides a clear guidance for applying existing
count models for software quality estimation. In one of the very
few studies, Evanco [9] applied a Poisson Regression Model
(PRM) to determine the fault locality, and fault correction ef-
fort during unit, system, and acceptance testing of a software
project. However, no reasoning was provided regarding the ap-
propriateness of the PRM for the software data set. Moreover, the
study did not use any evaluation criteria to assess the quality of
the fitted count model.
2In our study the dependent variable is the number of faults in a software
module.
GAO AND KHOSHGOFTAAR: EMPIRICAL STUDY OF COUNT MODELS FOR SOFTWARE FAULT PREDICTION
Our preliminary efforts investigated the benefits of applying
ZIP models for software fault prediction [6]. This study presents
a comprehensive empirical investigation of eight different count
models for software fault prediction. Six of the count models
are commonly used in non-software engineering fields, such as
econometrics [11], [13]. The other two, i.e., HP2, and HNB2, are
investigated for the first time in this study. A statistically-based
comparative analysis, including several model selection tech-
niques, is presented to provide a definitive guidance for applying
count models for software fault prediction.
The case study presented is of software metrics, and fault data
collected from two Windows-based embedded software systems
configured for wireless telecommunications. The eight count
models built for the system were compared with each other. A
one-way ANOVA model, and Tukey’s multiple comparison tech-
nique were used in our comparative analysis.
The remainder of this paper presents details of the count
models in Section II, model selection tests & techniques in
Section III, a case study in Section IV, and conclusions in
Section V.
II. COUNT MODELING TECHNIQUES
Count models are generally a variation of the commonly
used Poisson regression model. Another commonly used count
model is the negative binomial regression model, which is a
derivative of the PRM. Other count models have been derived by
combining different distributions, for example, finite-mixture
models [13]. In a finite-mixture model, a random variable is
postulated as a draw from a super-population that is an additive
mixture of a group of distinct populations, each of which has its
own unique distribution, such as Poisson, or negative binomial.
Zero-inflated count models, such as zero-inflated Poisson, and
zero-inflated negative binomial models are specific cases of fi-
nite-mixture models. Hurdle models, including hurdle Poisson,
and hurdle negative binomial models are also special cases of
finite-mixture models.
In addition to the finite-mixture models discussed in this
paper, we investigated several other mixture models. How-
ever, it was observed that they were either unsuitable for the
embedded software system being studied, or yielded similar
results as compared to some of the count models presented.
For example, upon investigating the Poisson-inverse Gaussian
model [14], it was observed that its prediction results were very
similar to those of the NBRM. Therefore, the modeling results
of other mixture models are not presented in the paper.
A. Poisson Regression Model
The Poisson regression model is derived from the Poisson dis-
tribution by allowing the expected value of the dependent vari-
able to be a function associated with the independent variables.
Let ( , ) be an observation in a data set, such that , and
are the dependent variable, and vector of independent vari-
ables for the observation, respectively. Given , assume
225
is Poisson distributed with the probability density function (PDF)
of
(1)
where is the mean value of the dependent variable . The
expected value, and the variance of are given by
(2)
Equation (2) demonstrates an important property of the Poisson
regression model, i.e., equidispersion, which implies that the
expected value of is equal to its variance.
To ensure that the expected value of is nonnegative, the link
function which displays a relationship between the expected
value, and the independent variables should have the form [13]
(3)
where denotes an unknown parameter
vector, and represents the transpose of the vector , which
is equal to . Note that both , and are
vectors, where is the number of independent variables used
in the data set. (1), and (3) jointly define the Poisson regression
model.
The maximum likelihood estimation (MLE) technique is often
used in the parameter estimation for regression models [15].
Given a set of observations, the log-likelihood function of the
PRM is given by
(4)
The commonly used method in MLE is the Newton-Raphson
iteration technique. Other methods, such as the EM algorithm,
also perform successfully for parameter estimation [16].
B. Zero-Inflated Poisson Model
A data set with an excess of zeros for the dependent vari-
able is a commonly observed phenomenon in software quality
modeling. This observation is especially prevalent for high as-
surance, and mission-critical software systems.
The ZIP model first introduced by Lambert [16] assumes that
all zeros come from two sources: the source representing the
perfect modules in which no faults occur, and the source repre-
senting the non-perfect modules in which the number of faults in
the modules follows the Poisson distribution. In the ZIP model, a
parameter is introduced to denote the probability of a module
being perfect. Hence, the probability of the module being non-
perfect is . The PDF of the dependent variable under the
ZIP model is therefore,
,
.
(5)
226 IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO. 2, JUNE 2007

A ZIP model may be obtained by adding the following two link where the gamma function denoted by is defined in [13],
functions: and is a constant parameter. The expected value of for the
negative binomial distribution is the same as that for the Poisson
(6) distribution, i.e., . However, the condi-
(7) tional variance of the negative binomial is given by

where is an unknown parameter vector (12)

of dimension .
The expected value, and the variance of in the ZIP model The conditional variance of in the NBRM exceeds the condi-
are, respectively, tional mean, because both , and are positive. Equation (12)
becomes if . In fact, the negative binomial distribution
(8) reduces to a Poisson distribution when . The standard pa-
rameter estimation technique for the NBRM is also MLE, and can
and
be derived in a similar fashion as presented for the PRM. How-
(9) ever, for further details, one can refer to [13].

Equations (8), and (9) display that the variance of exceeds its
expected value, which is known as overdispersion. An excess
of zeros for the dependent variable implies the overdispersion
phenomenon.
We use the MLE technique as the standard parameter estima-
tion technique for the ZIP model. Let denote an in-
dicator variable that takes the value of 1 when , and the
value of 0 otherwise. The log-likelihood function for the ZIP is
therefore given by
D. Zero-Inflated Negative Binomial Model
The zero-inflated negative binomial model is similar to the ZIP
model. The primary difference is that, in the case of the ZINB
model, the negative binomial distribution is used for the non-
perfect modules group, as compared to the Poisson distribution
used in the ZIP model.
The probability density function for the ZINB model is given
by (13) at the bottom of the page. The ZINB model is obtained
by adding the following two link functions:

(14)
(15)

where , and are the parameter vectors that are to be estimated.

(10)
C. Negative Binomial Regression Model
Another approach that addresses the overdispersion issue is
the negative binomial regression model (NBRM) [13], [15]. Var-
ious methods are available to obtain the NBRM. For example,
introducing unobserved heterogeneity is an intuitive motivation
to develop such a model [13].
The negative binomial probability distribution finally can be
obtained by the function
(11)
Similar to the other count models, the MLE technique can be used
to estimate the model parameters , , and . Further details
regarding the maximum likelihood estimation technique, and
the likelihood function for the ZINB model are presented in [13].
E. Hurdle Regression Models
The hurdle models partition the observations (modules in our
study) into two parts, i.e., a lower count group, and a higher
count group, depending on the selected threshold value, .
The chosen threshold can be any positive integer, according to
the problem domain. The model assumes that each group fol-
lows some standard distribution.
In our empirical study, we wanted to investigate whether the
threshold selection affected the performance of the respective
hurdle model. We selected the threshold values of 1, and 2 faults
to obtain the lower, and upper count groups. Therefore, software

,
(13)
GAO AND KHOSHGOFTAAR: EMPIRICAL STUDY OF COUNT MODELS FOR SOFTWARE FAULT PREDICTION 227

modules that had a number of faults greater than or equal to the tributions. The probability density function of HNB2 is therefore
selected threshold are categorized into the upper count group, given by
and into the lower count group otherwise. Moreover, for each
threshold value, we chose two distributions (Poisson, and nega- ,
tive binomial) to represent the groups of the hurdle model. When , (21)
the Poisson distribution is used, the resulting model is a hurdle
Poisson (HP) model. Whereas, when the negative binomial dis-
tribution is used, the resulting model is a hurdle negative bino- where is the same as in (20), and
mial (HNB) model. The hurdle model has a general form of
(22)
,
, and have the same forms as , and except
(16)
, for the replacement of parameters , and by , and ,
respectively.
The log-likelihood function for each hurdle model can be
where , and are some density functions that may have written as a sum of two individuals [13]: , and .
the same format, but with different parameters; and repre-
sents the selected threshold value. In this paper, we discuss four
different hurdle models. (23)
1) Case 1 (HP1): For a threshold value of 1 fault, both ,
and are defined to have Poisson distributions. Therefore,
the HP1 density function becomes (24)

, where is the function of the parameters for the lower counts

(17)
, ( for HP, ( ) for HNB); while is the function of the
parameters for the higher counts ( for HP, ( ) for HNB).
where is the Poisson mean for the group of Because, , and are functionally independent of each other,
lower counts, and is the Poisson mean for the the joint log-likelihood can be maximized by separately maxi-
group of higher counts. For this case, , and respectively mizing each individual function.
represent the zeros, and the positives groups.
2) Case 2 (HP2): For a threshold value of 2 faults, both , III. MODEL SELECTION TECHNIQUES
and are defined to have Poisson distributions. Therefore,
the HP2 density function becomes A. Likelihood Ratio Hypothesis Testing
The NBRM reduces to the PRM when the dispersion parameter
, . Hence, the null hypothesis, i.e., , against the
, alternative hypothesis, i.e., , can be tested by the LR
. test
(18) (25)
3) Case 3 (HNB1): For a threshold value of 1 fault, both ,
and are defined to have negative binomial distributions. where , and represent log-likelihood functions of
The probability density function of HNB1 is therefore given by the PRM, and the NBRM, respectively. Generally, if is true, the
likelihood ratio statistic, , follows a distribution with de-
, grees of freedom equal to the number of independent constraints
(19)
, [13]. However, the distribution of this statistic is nonstandard,
because of the restriction that .
where is The asymptotic distribution of the likelihood ratio statistic is
a weighted chi-square [13] distribution. For a given -signif-
(20) icance level, , is rejected when the test statistic exceeds
rather than . The statistic shows which regres-
Note that has the same form as , except for the re- sion model, PRM or NBRM, is preferred or is more appropriate.
placement of parameters , and by , and , respec- The same tests are used for the pairs ZIP versus ZINB, HP1 versus
tively. More specifically, has a negative binomial distri- HNB1, and HP2 versus HNB2. In the case of model simplification
bution with parameters , and . from HP to PRM, or from HNB to NBRM, a standard nested hy-
4) Case 4 (HNB2): Similarly, for a threshold value of 2 faults, pothesis is involved; therefore, the standard likelihood ratio test
both , and are defined to have negative binomial dis- is suitable.
228
B. Vuong’s Hypothesis Testing
Most hypothesis tests on the regression coefficient, and dis-
persion parameters in count models can be obtained by applying
the three classic hypothesis testing techniques: Likelihood Ratio
test, Wald test, and Lagrange Multiplier (LM) test [15]. However,
there are still some hypothesis tests which cannot be obtained by
directly using any of these methods. The primary reason is that
the relationship between the two models are different than the
assumptions of the hypothesis tests.
Given two conditional models, we can group the relationship
between the two models into two categories: nested, and non-
nested models. Generally speaking, two models are considered
nested if one model is a special case of the other. In contrast,
non-nested models imply that neither model can be represented
as a special case of the other.
Greene [15] pointed out that the PRM, and ZIP models are not
nested. Consequently, he used a test proposed by Vuong [17]
for the non-nested models, which is now briefly described. Let
be the predicted probability of the first model (Model
1), be the predicted probability of the second model
(Model 2), and be defined by
(26)
Then, the Vuong’s statistic for testing the nested hypothesis of
Model 1 versus Model 2 is given by
(27)
where denotes the mean of , for , and
denotes the standard deviation of , for . 2) Absolute, and Relative Errors: The accuracy of
Vuong’s statistic, , is used to test the hypothesis that
, i.e., for each observation, the two models have
the same predicted probabilities for a given dependent variable
. Vuong showed that the statistic is bidirectional, and
asymptotically -normal. For a given -significance level, , if
, then Model 1 is chosen; if , then
Model 2 is selected; otherwise, i.e., , either of
the two models can be selected as the appropriate model. In
this case, a simpler model is always preferred. For our study,
Vuong’s test is also suitable for testing the relative appropriate-
ness of the NBRM, and the ZINB model.
C. Information Criteria
Information criteria-based model selection techniques, which
are based on the fitted log-likelihood function, can be used for
both nested, and non-nested count models. It is assumed that
the log-likelihood will increase as more parameters are added
to a model. The penalty of increasing log-likelihood takes into
account the number of parameters as well as the number of ob-
servations. The three information criteria measures considered
in this study include
IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO. 2, JUNE 2007
1. Akaike information criterion: AIC = , where
is the number of parameters in the model, and is the
log-likelihood function.
2. Bayesian information criterion: BIC = ,
where is the number of observations in the fit data set.
3. Consistent Akaike information criterion: CAIC =
,
A count model with the lowest respective criterion-value (AIC,
CAIC, or BIC) will be selected. As compared to hypothesis
testing, model-selection based on information criteria (IC) can
facilitate an ordering of the various models.
D. Performance Measures
1) Pearson’s Chi-Square Measure: In the case of parametric
models, a crude diagnostic of studying the performance of a
prediction model is to observe how close the fitted probabili-
ties compare to the actual frequencies of the dependent variable.
Suppose the dependent variable takes values , where
. Let the observed frequency, i.e., the fraction of
the sample with , be denoted by ; and the corresponding
fitted frequencies be denoted by , , where the
fitted frequency, , is computed as the average over the obser-
vations of the predicted probabilities fitted for count .
Model performance can then be measured by comparing
with . The closer the two values are, the better is the model
performance. Pearson’s chi-squared is a closure measurement in
statistics [13]. Suppose the range of is divided into mutually
exclusive cells, where each cell may include one or more values
of . For example, , will
result in . The Pearson’s chi-square statistic is
(28)
the fault prediction models are measured by the av-
erage absolute error (AAE), and the average relative error
(ARE), i.e., AAE = , and ARE =
, where is the number of
modules, are the actual values of the dependent variable, and
are the predicted values of the dependent variable. In the case
of ARE, because the actual value of the dependent variable may
be zero, we add a “1” to the denominator to make the definition
always well-defined [5].
IV. EMPIRICAL CASE STUDY
A. System Description
The software metrics, and fault data for this case study was
collected from initial releases of two large Windows-based em-
bedded software applications used primarily for customizing
the configuration of wireless telecommunications products. The
two applications, written in C++, provided similar functionali-
ties, and contained common source code; and hence, are anal-
ysed as one software system. Each application contained more
than 27.5 million lines of code. A program module is comprised
of a single source file. Upon preprocessing & cleaning of the
GAO AND KHOSHGOFTAAR: EMPIRICAL STUDY OF COUNT MODELS FOR SOFTWARE FAULT PREDICTION
TABLE I
SOFTWARE METRICS
collected software metrics, and fault data, i.e., removing incom-
plete or illogical data points, 1211 modules remained in the data
set.
The software metrics for each module were collected using a
combination of tools, and databases. Among the 1211 modules,
over two-thirds had no faults, and the maximum number of the
faults in one module was 97. The dependent variable, Fault, in-
dicated the number of the software faults discovered in a source
file during system test. In the context of this case study, Table I
lists the five software metrics that are used as independent vari-
ables for calibrating software quality models.
The fit, and test data sets were obtained by applying an im-
partial data splitting technique to the original data set of 1211
program modules. Consequently, two-thirds of the modules, i.e.,
807, were assigned to the fit data set, whereas, the remaining
one-third of the modules, i.e., 404, were assigned to the test data
set. To avoid biased results due to a lucky (or unlucky) data split-
ting, the original data set was randomly split 50 times, yielding
50 pairs of the fit, and test data sets. Empirical studies were per-
formed for all 50 data splits.
B. Results & Analysis
According to the modeling methodology of each count model
(see Section II), eight count models were calibrated for each
of the fifty data splits. To determine the best fitted models for
the case study, we employed pairwise hypothesis testing tech-
niques. Because we considered eight different count models, the
number of possible pairwise comparisons is . How-
ever, we present only ten of the pairwise comparisons, and sub-
sequently selected the three best count models. Other compar-
isons were either not performed because hypothesis testing was
not feasible, or are not presented because the respective pair-
wise tests were irrelevant. The eight pairwise likelihood ratio-
based hypothesis testing included PRM versus NBRM, ZIP versus
229
TABLE II
HYPOTHESIS TESTING RESULTS
ZINB, PRM versus HP1, PRM versus HP2, NBRM versus HNB1,
NBRM versus HNB2, HP1 versus HNB1, and HP2 versus HNB2.
In addition, the non-nested models were examined by applying
Vuong’s hypothesis testing to two pairwise comparisons: PRM
versus ZIP, and NBRM versus ZINB.
The testing results, which were consistent for all fifty data
splits, are summarized in Table II. The first column of the table
lists the ten pairwise comparisons, the second column shows
the hypothesis testing techniques utilized, the third column
indicates the recommended model according to the results of
the respective pairwise testing, and the fourth column presents
the -values of the comparisons. The -values indicate the
-significance by which the recommended model is -better
than its counterpart. The overall conclusion from the hypothesis
testing indicated that the ZINB, and HNB models are -signif-
icantly better than the other count models, for both threshold
values. Determining the best model among these three is not
feasible with hypothesis testing, i.e., to our knowledge, no
hypothesis testing-based technique exists that can be used
to compare ZINB versus HNB models, and the respective HNB
models with different thresholds. Therefore, with respect to hy-
pothesis testing-based model selection, these three models are
considered to have similar performance. However, if one were
to select the best count model among them, then information
criteria (IC)-based model selection techniques could be used.
The next step involved IC-based model selection, i.e., AIC, BIC,
and CAIC, and evaluating the results with those obtained by hy-
pothesis testing. It was observed that for all three information
criteria, similar model selection results were obtained. More-
over, the top three models selected by IC were the same as those
determined by hypothesis testing. More specifically, ZINB, and
HNB have the lower IC values, indicating that they are the pre-
ferred models for this case study. In contrast, the PRM demon-
strated the largest values with respect to information criteria,
whereas the NBRM, ZIP, and HP models depicted intermediate
information criteria values.
230 IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO. 2, JUNE 2007

TABLE III TABLE IV

MODEL SELECTION BASED ON INFORMATION CRITERIA FOR COUNT MODELS ANOVA TABLE:  (FIT DATA SETS)

An important advantage of the information criteria-based

comparative techniques over the hypothesis testing-based tech-
niques is that the modeling techniques can be sorted according
to their IC statistics. Table III shows the sorted models based
on the IC, discussed in Section 3.3, when applied to the fit data
set. For all three IC techniques, i.e., AIC, BIC, and CAIC, the
same performance order was obtained. The symbol “ ” in the
table indicates that the left hand side model has lower values
of IC than the model on the right hand side, i.e., the left hand
side model is preferred over the right hand side model. The
modeling techniques sorted from left to right are the same for
all fifty data splits except for one split (split number 22). As
compared to the other 49 data splits, the only difference in the
performance-order for split 22 is that the NBRM is on the right
hand side of the ZIP, and HP1 models as compared to being on
their left hand side for the other data splits. The different order TABLE V
of models obtained for split 22 may be due to a few influential ANOVA TABLE:  (TEST DATA SETS)

data points grouped into the fit data set.

The next step in our empirical study involved evaluating the
count models with respect to the Pearson’s chi-square measure.
A comparison of the actual, and fitted frequency distributions
was made for the eight count models. The range of was divided
into seven cells, i.e., , and .
In the context of the Pearson’s chi-square measure, we further
grouped the cells into two sub-parts: zeros, which included cell
; and positives, which included all the other cells. The lower
the Pearson’s chi-square statistic, the better is the performance
of the model.
The Pearson’s chi-square statistics for the eight count models
were computed for each of the fifty data splits. The one-way
ANOVA model indicated whether the eight count models per-
formed -significantly different than each other across all the
data splits. The ANOVA results for the fit, and test data sets
are presented in Tables IV and V, respectively. Each table is
comprised of three components: Overall, Zeros, and Positives.
Overall includes all seven cells, Zeros only includes cell ,
and Positives includes the other cells. We observe that for both
fit, and test data sets, and for each component, the average
chi-square statistics over the fifty splits of the eight models are any two group means exceeds this value of MSD, the differences
-significantly different from each other, i.e., all -values are 0. are -significant at the given level. Multiple comparisons re-
Because the ANOVA test indicated that the eight count sults for fit, and test data sets are shown in Tables VI and VII,
models performed -significantly different than each other, we respectively. Each table consists of three parts: Overall, Zeros,
proceeded with Tukey’s multiple comparison tests with the and Positives. For each part, the mean values of methods over
Pearson’s chi-square measure as the dependent variable. For the fifty splits are sorted in descending order; N represents the
a given -significance level , we can calculate the minimum group size (50 in our study), and the -significance level is
-significant difference (MSD) [18], [19]. If the difference of 0.15.
GAO AND KHOSHGOFTAAR: EMPIRICAL STUDY OF COUNT MODELS FOR SOFTWARE FAULT PREDICTION 231

TABLE VI TABLE VII

MULTIPLE PAIRWISE COMPARISONS:  (FIT DATA SETS) MULTIPLE PAIRWISE COMPARISONS:  (TEST DATA SETS)

Note: Means with the same letter are not s-significantly different. Note: Means with the same letter are not s-significantly different.

models represented with the same letters belong (according to

The first column of the table with label “Tukey’s Grouping” relative -significance) to the same group, and members in the
categorizes the models into performance-based groups. The same group are not -significantly apart from each other. In
232
TABLE VIII
ANOVA TABLE: AAE, AND ARE (FIT DATA SETS)
contrast, models represented with different letters are in dif-
ferent groups, and members in the different groups are -sig-
nificantly apart from each other. For example, for the Overall
part in Table VI, there are five groups: Group A—PRM, Group
B—HP2, Group C—HP1 & ZIP, Group D—NBRM, and Group
E—ZINB & HNB1 & HNB2. Hence, the chi-square mean of PRM
is -significantly larger than the other seven models, and the
chi-square mean of HP1 is larger than that of ZIP, but the dif-
ference is not -significant. The multiple comparison results of
the Pearson’s chi-square measure demonstrated several empir-
ical observations for the software system studied in this paper:
• The PRM, and HP2 had -significantly poorer performances
than the other six methods in the Overall, Zeros, and Pos-
itives parts.
• All methods, except the PRM, and HP2 model, had similar
performances in the Zeros part.
• The ZINB, and HNB models had -significantly better per-
formances than the other five methods for the Positives
part, and the NBRM model had -significantly better per-
formance than the ZIP, and HP models for the same part.
• The performances of the models in the Positives part deter-
mined the Overall performances, because they had similar
performances in the Zeros part, except for PRM, and HP2.
IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO. 2, JUNE 2007
TABLE IX
ANOVA TABLE: AAE, AND ARE (TEST DATA SETS)
• The performance order (rank among the eight models) of
a given count model for the test data set was the same as
its performance order for the fit data set, except for three
models in the Zeros part. The HNB2 model has better per-
formance than the HP1, and ZINB models for the test data
set. But for the fit data set, it performs worse than the HP1,
and ZINB models. However, this inconsistency occurred
only for models that have an -insignificant difference in
their relative performances. The top three models (for the
Overall part) were the same as those determined by hypoth-
esis testing-based, and information criteria-based model
selection. These three models, in no particular order, are
the ZINB, HNB1, and HNB2 models.
The final model selection & evaluation techniques investi-
gated included the performance metrics AAE, and ARE. The use
of these measures is similar to our previous studies related to
count models [6]. The AAE, and ARE values were computed for
both the fit, and test data sets. Once again, we grouped each
data set (Overall) into two parts, i.e., Zeros, and Positives. The
modules with no faults were assigned to the Zeros part, whereas
those with non-zero faults were assigned to the Positives part.
The respective AAE, and ARE values for the three parts were
computed for all the fifty data splits. The one-way ANOVA, and
Tukey’s multiple comparison tests once again were employed
GAO AND KHOSHGOFTAAR: EMPIRICAL STUDY OF COUNT MODELS FOR SOFTWARE FAULT PREDICTION 233

TABLE X
MULTIPLE PAIRWISE COMPARISONS: AAE, AND ARE (FIT DATA SETS)

Note: Means with the same letter are not s-significantly different.
234 IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO. 2, JUNE 2007

TABLE XI
MULTIPLE PAIRWISE COMPARISONS: AAE, AND ARE (TEST DATA SETS)

Note: Means with the same letter are not s-significantly different.
GAO AND KHOSHGOFTAAR: EMPIRICAL STUDY OF COUNT MODELS FOR SOFTWARE FAULT PREDICTION
to analyse the prediction performances of the different methods
with respect to the AAE, and ARE values. Tables VIII and IX
summarize the ANOVA test results for the fit, and test data sets,
respectively. In each table, the upper half reports the ANOVA test
results for AAE, whereas the lower half reports the ANOVA test
results for ARE. The ANOVA test results indicate that the mean
values of AAE, and ARE of the eight models over the fifty splits
-significantly differ from each other for the Overall, Zeros,
and Positives parts. Hence, we once again proceeded with per-
forming multiple comparisons of the different models.
Tukey’s multiple comparison results for the fit, and test data
sets are shown in Tables X and XI, respectively. Each table con-
sists of the comparisons of AAE (left), and ARE (right) of the
different models. These comparisons were conducted for the
Overall, Zeros, and Positives parts. The methods were sorted
by their mean values of AAE, and ARE in descending order, and
are grouped with critical values at a -significance level of 0.15.
The empirical observations based on these tables are summa-
rized below.
• The NBRM showed -significantly poor performance (for
both AAE, and ARE) than the other seven count models for
the Positives, and Overall parts. For these two parts, the
other seven models were similar at the -significance level
of 0.15.
• In the Zeros part, the prediction performance of models
were segregated into four groups: Group A—PRM, Group
B—ZIP & HP1, Group C—ZINB & NBRM & HNB, and Group
D—HP2. Models from the different groups are -signifi-
cantly apart from each other. It is seen that Group D had
the best prediction accuracy; Group A had the worst pre-
diction accuracy; and Groups B, and C had intermediate
performances.
• The performance order for the test data sets was similar to
that for the fit data sets. Exceptions occurred only for the
models that have -insignificant differences in their per-
formances. For example, PRM was an exception. Based on
AAE, PRM had better fitted accuracy than ZINB, and HNB
in the Positives, and Overall parts; but worse (in the same
parts) for the prediction accuracy based on the test data
sets.
With respect to AAE, and ARE, the HP2 model is the preferred
model because it demonstrated the best predictive quality, and
a good quality of fit. This contradicts the hypothesis testing, in-
formation criteria, and Pearson’s chi-square measure model se-
lection results, which gave evidence that ZINB, HNB1, and HNB2
were the preferred models. This contradiction indicates that the
best fitted model chosen by hypothesis testing-based or infor-
mation criteria-based model selection may not ensure the best
performance with respect to some other evaluation perspective.
However, the consistency, and similarity of the model’s quality
of fit, and the model’s prediction quality existed in the context of
the Pearson’s chi-square, AAE, and ARE measures. This implies
that a good performance for the fit data set with respect to the
Pearson’s chi-square measure will yield a good performance for
the test data set with respect to the Pearson’s chi-square mea-
sure. This is also true in the case of AAE, and ARE measures.
235
V. CONCLUSION
Count models, such as Poisson regression, are appropriate
for software quality modeling because software faults are
usually recorded as numerical counts. A comprehensive em-
pirical investigation on the comparative performances of eight
count models for predicting faults in program modules was
performed. Case studies of multiple software systems were
performed; however, only a case study of a telecommunications
system has been presented. Hypothesis testing-based, and in-
formation criteria-based techniques were applied to determine
the appropriateness of different count models. The relative
predictive accuracies of the models were evaluated based on
a one-way ANOVA model, and Tukey’s multiple comparison
technique.
Among the different count models, the zero-inflated nega-
tive binomial, and the hurdle negative binomial models demon-
strated better fitting according to the hypothesis testing-based,
information criteria-based, and the Pearson’s chi-square mea-
sure-based model selection. The predictive accuracy (for test
data) according to the AAE, and ARE suggested that the HP2 per-
formed relatively better than the other count models. The contra-
diction of results based on hypothesis testing, AAE-based predic-
tion, and ARE-based prediction quality suggested that different
selection criteria may yield different models. However, it was
found that the best fitted model using a given performance mea-
sure (Pearson’s chi-square measure, AAE, or ARE) yielded the
best predictive accuracy with respect to the same measure.
Future work related to software quality modeling with count
models will involve additional empirical case studies; and will
investigate the effect of outliers, and noise in the fit data set
on the model performance. Specifically, effective techniques to
identify & eliminate such data points will be investigated with
the aim of improving model performance.
ACKNOWLEDGMENT
The authors would like to thank Naeem Seliya for his sugges-
tions and assistance with patient editorial reviews, and modifi-
cations of the paper.
REFERENCES
[1] K. Ganesan, T. M. Khoshgoftaar, and E. B. Allen, “Case-based
software quality prediction,” International Journal of Software En-
gineering and Knowledge Engineering, vol. 10, no. 2, pp. 139–152,
2000.
[2] S. S. Gokhale and M. R. Lyu, “Regression tree modeling for the predic-
tion of software quality,” in Proceedings: 3rd International Conference
on Reliability and Quality in Design, H. Pham, Ed., Anaheim, Cali-
fornia, USA, March 1997, pp. 31–36, International Society of Science
and Applied Technologies.
[3] T. M. Khoshgoftaar and N. Seliya, “Tree-based software quality
models for fault prediction,” in Proceedings: 8th International Soft-
ware Metrics Symposium., Ottawa, Ontario, Canada, Jun. 2002, pp.
203–214, IEEE Computer Society.
[4] Z. Xu, T. M. Khoshgoftaar, and E. B. Allen, “Application of fuzzy
linear regression model for predicting program faults,” in Proceedings:
Sixth ISSAT International Conference on Reliability and Quality in De-
sign, H. Pham and M.-W. Lu, Eds., Orlando, Florida, USA, Aug. 2000,
pp. 96–101, International Society of Science and Applied Technolo-
gies.
236
[5] T. M. Khoshgoftaar, J. C. Munson, B. B. Bhattacharya, and G. D.
Richardson, “Predictive modeling techniques of software quality from
software measures,” IEEE Trans. Software Engineering, vol. 18, no.
11, pp. 979–987, Nov. 1992.
[6] T. M. Khoshgoftaar, K. Gao, and R. M. Szabo, “An application of
zero-inflated Poisson regression for software fault prediction,” in Pro-
ceedings of the Twelfth International Symposium on Software Relia-
bility Engineering, Hong Kong, China, November 2001, pp. 66–73,
IEEE Computer Society.
[7] L. C. Briand, W. L. Melo, and J. Wust, “Assessing the applicability of
fault-proneness models across object-oriented software projects,” IEEE
Trans. Software Engineering, vol. 28, no. 7, pp. 706–720, July 2002.
[8] T. M. Khoshgoftaar, B. Cukic, and N. Seliya, “Predicting fault-prone
modules in embedded systems using analogy-based classification
models,” International Journal of Software Engineering and Knowl-
edge Engineering, vol. 12, no. 2, pp. 201–221, Apr. 2002, World
Scientific Publishing.
[9] W. M. Evanco, “Modeling the effort to correct faults,” Journal of Sys-
tems and Software, vol. 29, pp. 75–84, 1995.
[10] P. Deb and P. K. Trivedi, “Demand for medical care by the elderly: A
finite mixture approach,” Journal of Applied Econometrics, vol. 12, pp.
313–336, 1997.
[11] J. Mullahy, “Specification and testing of some modified count data
models,” Journal of Econometrics, vol. 33, pp. 341–365, 1986.
[12] W. Pohlmeier and V. Ulrich, “An ecometric model of the two-part de-
cision making process in the demand for health care,” The Journal of
Human Resources, vol. 30, pp. 339–361, 1995.
[13] A. C. Cameron and P. K. Trivedi, Regression Analysis of Count Data.
: Cambridge University Press, 1998.
[14] C. Dean, J. F. Lawless, and G. E. Willmot, “A mixed Poisson-inverse-
Gaussian regression model,” The Canadian Journal of Statistics, vol.
17, no. 2, pp. 171–181, 1989.
[15] W. H. Greene, Econometric Analysis, 4th ed. Upper Saddle River,
New Jersey: New York University: Prentice-Hall Inc., 2000.
[16] D. Lambert, “Zero-inflated Poisson regression, with an application to
defects in manufacturing,” Technometrics, vol. 34, no. 1, pp. 1–14, Feb.
1992.
IEEE TRANSACTIONS ON RELIABILITY, VOL. 56, NO. 2, JUNE 2007
[17] Q. H. Vuong, “Likelihood ratio tests for model selection and non-nested
hypotheses,” Econometrica, vol. 57, no. 2, pp. 307–333, Mar. 1989.
[18] SAS/STAT User’s Guide. Cary, NC, USA: SAS Institute Inc, 1990,
vol. 2.
[19] M. L. Berenson, D. M. Levine, and M. Goldstein, Intermediate Statis-
tical Methods and Applications: A Computer Package Approach. En-
glewood Cliffs, New Jersey: Prentice Hall, 1983.
Kehan Gao received the Ph.D. degree in Computer Engineering from Florida
Atlantic University, Boca Raton, FL, USA, in 2003. She is currently an
Assistant Professor in the Department of Mathematics and Computer Science
at Eastern Connecticut State University. Her research interests include software
engineering, software metrics, software reliability and quality engineering,
computer performance modeling, computational intelligence, and data mining.
She is a member of the IEEE Computer Society, and the Association for
Computing Machinery.
Taghi M. Khoshgoftaar is a professor of the Department of Computer Science
and Engineering, Florida Atlantic University, and the Director of the Empirical
Software Engineering Laboratory, and the Data Mining and Machine Learning
Laboratory. His research interests are in software engineering, software met-
rics, software reliability and quality engineering, computational intelligence,
computer performance evaluation, data mining, machine learning, and statis-
tical modeling. He has published more than 300 refereed papers in these areas.
He is a member of the IEEE, IEEE Computer Society, and IEEE Reliability So-
ciety. He was the program chair, and General Chair of the IEEE International
Conference on Tools with Artificial Intelligence in 2004, and 2005 respectively.
He has served on technical program committees of various international confer-
ences, symposia, and workshops. Also, he has served as North American Editor
of the Software Quality Journal, and is on the editorial boards of the journals
Software Quality and Fuzzy systems.