JOURNAL OF INFORMATION SYSTEMS

Vol. 18, No. 2

Fall 2004
pp. 107-110

Discussion of:
Firm Performance Effects in Relation to the
Implementation and Use of Enterprise
Resource Planning Systems
Jacqueline L. Reck
University of South Florida
I. INTRODUCTION

E xisting research examines whether the implementation of an ERP system improves firm
fmancial performance. The mixed findings support the need for additional research into
factors that contribute to improved financial performance from an ERP system implementa-
tion. The Nicolaou (2004) study has the potential to address this need by introducing four possible
factors that could impact the effectiveness of BRP implementation. My comments on the Nicolaou
paper relate to its contribution, method, results, and conclusion.

II. CONTRIBUTION
Published studies by Poston and Grabski (2001) and Hunton et al. (2003) provide evidence on
the ERP implementation's impact on long-term performance. These studies rely upon a variety of
perfonnance indicators to assess firm performance up to three years after ERP implementation.
Results of the studies provide mixed evidence concerning ERP implementation's impact on im-
proved performance. Since the paper under discussion conducts essentially the same analysis in the
first hypothesis, the paper's potential contribution lies in the second hypotheses, which attempt to
identify factors that may interact with ERP implementation to enhance financial performance. Unlike
research by Hunton et al. (2003), who investigated the impact of vendor, firm health, and firm size,
the current paper investigates three factors specific to the implementation process.
The three implementation factors hypothesized in H2b-2d are scope of implementation, ERP
implementation objectives, and time to implementation, respectively. A fourth factor, ERP vendor
(H2a), tests a factor previously examined in Hunton et al. (2003). These hypotheses have the
potential to provide insights into the mixed results in the existing literature. However, the relation-
ships among the variables are underdeveloped. As an illustration, consider the ERP vendor variable.
The justification for including vendor relates to the use of "best practices" promoted by all vendors;
however, the use of "best practices" often affects the extent to which a firm must adapt its own
processes before an ERP system can be implemented. It is unclear from the explanation, how
separating vendors dichotomously (0/1 dummy) captures the problem identified in the variable
explanation and how the use of a "best practice" relates to each, or any, of the eight financial
perfonnance variables used to measure successful ERP implementation. That is, why would SAP or
Oracle (the most common vendors), as opposed to all other vendors, interact with ERP implementa-
tion to positively or negatively impact retum on sales or cost of goods sold?

107
108 Reck

Additionally, the relationship among the firm factors is unexamined. For example, it would seem
that the scope of an implementation should impact the time to implementation. If some ERP vendors'
systems are more complex than others, complexity could impact the time to implementation, and
perhaps even the scope of implementation. An interesting question might be how the three factors—
vendor, scope of implementation, and implementation objectives—interact to impact time to imple-
mentation. While the paper defines time to implementation in terms of change management, it can be
argued that other identified firm factors also influence the time to implementation.
Basically, the study tests whether, in ERP firms, the four firm factors are significantly related to
several ad hoc financial variables. The absence of theory to support directional hypotheses makes
the study exploratory. For an exploratory paper to contribute to the literature, it should explain why
variables are chosen for hypotheses and the hypothesized relationships among the variables. Since
the direction cannot be specified ex ante, interpreting and discussing possible explanations for
significant results ex post would enhance the paper's contribution.

III. RESEARCH METHOD

The paper uses a matched pair design to test the event ERP implementation. A commendable
effort is made to ensure that the firm selected for the match is not an ERP implementation firm.
However, it appears that the method for defining the test periods does not follow the conventional
archival research method. The paper indicates that firms are matched at time / - I , i.e., the year prior
to ERP adoption, and that time tO is the year of ERP completion. Archival research traditionally uses
-1 to indicate one year prior to year 0. However, Panel C of Table 1 indicates that all 105 firms
providing actual time to completion information required two or more years to complete the ERP
implementation. For the remaining 142 firms, an inferred time to completion of 9.92 months is
assumed. This implies that t-l, rather than being one year less than tO, is actually anywherefi-omtwo
to six years less than tO for 105 ofthe firms included in the study. If true, what is the impact of
omitting a varying number of years from the longitudinal analysis (due to the years being a part ofthe
implementation period)? Are there uncontrolled events during the "black box" (omitted) period? Are
there differences between firms providing actual time to completion information and those firms for
which the time to completion is assumed?
In the models tested, the paper carefully controls for pre-implementation performance, which
has been found (Barber and Lyon 1996) to be important in event studies testing for changes in
performance. A second variable included in the models is the independent variable, ERP Event,
which is a 0/1 dummy variable. According to the paper, the value ofthe dummy variable is based on
the direction ofthe dependent variable, change in performance, and is equal to 1 if the implementing
firm's performance is greater than the matched firm's and is equal to 0 if the implementing firm's
performance is less than the matched firm's. As seen from the definition, the value ofthe ERP Event
variable depends on the relationship ofthe implementing firm's perfonnance to the matched firm's
perfonnance. However, the results ofthe regression models are interpreted using a different defini-
tion of the ERP Event variable, as evidenced by the statement, "the ERP Event dummy solely
depends on the independent observation as to whether a firm is an ERP adopter. Model (1) avoids
concems for model endogeneity." That is, the paper interprets the ERP Event variable as represent-
ing the effect of implementing firm versus matched firm rather than relative positive performance
versus relative negative performance. However, the reader should interpret a significant ERP Event
as indicating that for an ERP implementer with relatively positive perfonnance there is a signifi-
cantly greater association with the fmancial performance measure than there is for an ERP implementer
with relatively negative performance.

Journal of Information Systems, Fall 2004

Discussion of: Firm Performance Effects in Relation to the Enterprise Resource Planning Systems 109

IV. RESULTS
As with prior studies, the paper provides mixed support for the first hypothesis. The strongest
support for ERP implementation's positive impact on performance is found for return on assets,
operating income, and cost of goods sold. Firm factors studied in hypotheses two tend to be most
highly associated with the same three performance measures. However, an examination of Table 4
indicates that the data for these performance measures is quite skewed. The skew in the data is fairly
typical in performance data (Barber and Lyon 1996), indicating the need to conduct tests for viola-
tions of regression assumptions, including heteroscedasticity and outliers. The dichotomous vari-
ables have been centered to minimize potential collinearity and coefficients appear to be standardized.
However, there is no indication that tests have been conducted to address heteroscedasticity or
outlier problems. Including nonparametric tests for robustness of results is also a method for ad-
dressing possible parametric problems.
Table 6 indicates that for retum on assets, operating income, and cost of goods sold, perfor-
mance was significant in the pre-performance period, raising a question about whether it is ERP
implementation or some other factor(s) driving the firm's performance. The paper would benefit
from additional analysis and discussion ofthe pre-performance period, and the "black box" period
(implementation period).
As with HI, H2a-2d yield mixed results. Understanding the hypotheses' results is problematic
since the paper analyzes the results by indicating that ERP Event is based on implementing firm
versus matched firm rather than relative positive performance of the implementing firm versus
relative negative performance ofthe implementing firm.
Since the paper is exploratory, and since the abstract indicates that future papers should consider
controlling for the four firm factors identified, it would be helpful to see all four firm factors included
in the same regression. The variables for scope of implementation and time to implement could be
collapsed into dichotomous variables and the models could include the four main effect dummies
and the four interaction variables, along with the ERP Event and pre-performance variable. Theory
should guide the use of additional variables, such as three-way or four-way interactions. However,
since the paper is exploratory rather than theory driven, an initial analysis to see if higher order
interactions are relevant might be conducted. Here interpretation ofthe interactions is not as impor-
tant as validating that the firm factors are necessary controls in future research on ERP implementations.

V. CONCLUSION
The paper's potential to make a significant contribution to the ERP literature is through discuss-
ing and examining firm factors that impact the association between ERP implementation and firm
performance. However, the manner in which the paper is developed limits its potential contribution,
in that the discussion relating to how and why the firm factors might moderate the relationship of
ERP implementation and, thus, impact various financial performance measures is underdeveloped.
The paper's mixed results concerning the impact of ERP implementation factors on firm perfor-
mance, and the lack of explanation for the results, indicates that the need continues for research into
what factors are contributing to the successful (i.e., performance enhancing) implementation of ERP
systems in some firms, but not others.

Journal of Information Systems, Fall 2004

110 Reck

REFERENCES
Barber, B. M., and J. D. Lyon. 1996. Detecting abnormal operating performance: The empirical power and
specification of test statistics. Journal of Financial Economics 41 (3): 359-399.
Hunton, J. E., B. Lippincott, and J. L. Reck. 2003. Enterprise resource planning systems: Comparing firm
perfonnance of adopters and non-adopters. International Journal ofAccounting Information Systems 4
(September): 165-184.
Nicolaou, A. 1. 2004. Firm perfonnance effects in relation to the implementation and use of enterprise resource
planning systems. Journal of Information Systems 18 (Fall): 79-105.
Poston, R., and S. Grabski. 2001. Financial impacts of enterprise resource planning implementations. Interna-
tional Journal ofAccounting Information Systems 2: 271—294.

Journal of Information Systems, Fall 2004