Вы находитесь на странице: 1из 9

Information & Management 50 (2013) 553–561

Contents lists available at ScienceDirect

Information & Management

journal homepage: www.elsevier.com/locate/im

Case studies in research

Using balanced scorecards for the evaluation of

Sangjae Lee a, Sung Bum Park b,*, Gyoo Gun Lim c
School of Business Administration, Sejong University, Republic of Korea
National Information Society Agency, Republic of Korea
School of Business, Hanyang University, Republic of Korea


Article history: To overcome the problem of limited resources, increasing numbers of small- and medium-sized
Received 17 August 2012 companies (SMEs) are adopting ‘‘Software-as-a-service’’ (Saas) as an efficient tool for IS implementation.
Received in revised form 24 January 2013 The balanced scorecard (BSC) has been adopted by SMEs to evaluate Saas via four measures: learning and
Accepted 11 July 2013
growth, internal business processes, customer performance, and financial performance. The survey
Available online 1 August 2013
results for 101 Software-as-a-service adopters indicate that learning and growth, internal business
processes, and customer performance are causally related to financial performance. The results show
that these four key elements for Saas success are interrelated, supporting the core premise of the BSC.
Software-as-a-service (Saas)
Balanced scorecard (BSC)
ß 2013 Elsevier B.V. All rights reserved.
Causal relationships

1. Introduction These attempts at ‘‘utility computing’’ are taking off due to the
availability of sufficient bandwidth for such services. Broadband
IS maintenance costs comprise a major portion (70%) of the communication has become cheap and plentiful enough for
total IS implementation costs. Companies choose to invest their utilities to deliver computing services with the speed and
resources and manpower in their core capability to provide reliability that businesses previously enjoyed from their local
products or services. The demand for IT outsourcing and the machines. Typical Iaas offerings include Amazon’s Elastic Compute
Software-as-a-service (Saas) model, which integrates network, Cloud (EC2) and Simple Storage Service (S3), Joyent’s Accelerator,
hardware, and software, is increasing as IT sophistication itself and Rackspace’s Mosso. The possible reasons for adopting cloud
increases [31]. computing include (1) avoiding capital expenditure in hardware,
Saas can be defined as applications and computer-based software, and IT support and (2) using the flexibility and scalability
services delivered and managed from a remote center to multiple of IT resources. The major issues of cloud computing include
customers via the Internet or a VPN. Saas shares common themes integrity of services and data, confidentiality of corporate data, and
with On-Demand Service [27]. There is a growing use of other reliability perceptions, to name a few [5,8].
related and advanced platform services, such as cloud computing, The Saas provider acts as a mediator, mediating services
infrastructure-as-a-service (Iaas) and platform-as-a-service (Paas), between independent software vendors (ISVs). Saas customers do
representing a large pool of usable resources, such as hardware and not possess, manage or maintain the applications, but only use
software, that are easily accessible via the Internet [11]. It is them as final products by accessing services with IT support. While
estimated that by 2013, the cloud market will have reached $8.1 Saas is advantageous in that it reduces the repair costs of
billion [3]. Industry leaders predict that revenues from cloud application-based construction and maintenance, the risk of data
computing enterprises will grow to $160 billion. Experts describe leakage becomes a major disadvantage because application servers
cloud computing as ‘‘an emerging IT development, deployment are constructed by outside companies.
and delivery model, enabling real-time delivery of products, Saas is one type of ASP (application service provider service).
services and solutions over the Internet’’ [16]. After being introduced between 1998 and 1999, the ASP service
market increased rapidly from 2000 to 2001 due to excessive
expectations and ASP service provider mergers and acquisitions
(M&A). While the number of customers and the market size are
* Corresponding author.
continuously growing, the growth rate is plateauing. As the ASP
E-mail addresses: sangjae@sejong.ac.kr (S. Lee), parksb@nia.or.kr (S.B. Park), market worsens, it is essential to improve ASP planning and
gglim@hanyang.ac.kr (G.G. Lim). management, as it is now harder for many ASP providers to survive

0378-7206/$ – see front matter ß 2013 Elsevier B.V. All rights reserved.
554 S. Lee et al. / Information & Management 50 (2013) 553–561

[10]. For ASP services to be successful, system and service qualities by SMEs [18]. Despite the wide scope of BSC application as a
must be well prepared [32]. In small- and medium-sized decision-supporting instrument, cause-and-effect relationships
companies (SMEs), both ASP service and information quality are and time-delayed elements between measures are still elusive
significant factors in enhancing user satisfaction, trust, and the [47]. While causal relationships among the four BSC measure-
intention to use [38]. ASPs are said to have achieved considerable ments are the core focus of BSC, empirical studies (via the testing of
success with big businesses, but their success has been less notable causal relationships among categories) examining how well BSCs
with SME markets [1]. For example, lack of ASP customization can be applied are almost nonexistent. This study intends to fill this
options and concerns about financial stability, service reliability, gap. This study suggests some measures to evaluate Saas and tests
and functional capability flexibility are problems for SMEs. For these measures using data collected from companies that have
these reasons, SMEs are less likely to adopt ASP services than large adopted Saas. The discussion and implications are included in the
companies, making it necessary to develop empirical studies to study.
examine how specialized measures may be applied to SMEs.
Outcome assessment in the traditional field of business 2. Theoretical background
administration places excessive emphasis on financial perfor-
mance measures [49]. Because it is impossible to assess an 2.1. ASP service evaluation models
organization in a competitive environment based merely on
financial performance, it is necessary to measure Saas performance The criteria considered for ASP service selection are credibility,
by using the balanced scorecard (BSC) approach, which balances appropriateness, and efficiency, which further includes ‘‘prior
leading and lagging indicators, as well as by using financial and experience,’’ ‘‘ASP service expectation,’’ ‘‘perceived provider
non-financial measures [29]. To provide a balanced approach to the performance,’’ and ‘‘expectation-disconfirmation.’’
measurement of organizational performance, including sub-areas, Kern et al. [31] suggested six propositions based on resource
such as knowledge management (KM), business processes, and dependency theory, resource-based theory, transaction cost
financial performance, BSC measures four categories: learning theory, and agency theory. These propositions are as follows: (1)
and growth, internal business processes, customer performance, the use of an ASP service is a strategic decision to supplement
and financial performance. Small companies develop multiple necessary parts of IS; (2) ASP customer service depends heavily on
scoreboards, each tailored to the strategy and goals of a specific the ASP service; (3) ASP service generally has a lower cost; (4) ASP
subunit. To survive in today’s global and volatile business service prices increase over time; (5) SMEs are more interested in
environment, SMEs are using newer management systems, such ASP service than are large businesses; and (6) the suitability of an
as BSCs, to clarify their vision and strategy and to translate them ASP service is determined by the capability of the customer and the
into action [35]. variety of the ASP service. Kern et al. articulated specific
BSCs also include a financial perspective because such a characteristics derived from the relationship between ASP service
perspective can easily summarize previous financial activities providers and their customers.
and yield predictable economic outcomes. The financial perspec- This framework for measuring the SME benefits and risks of
tive indicates whether a strategic operation contributed to net Internet-based applications is designed for companies that do not
profit improvement. The customer category assesses the extent to have any specific team or methodology for measuring proposed IT
which the target market was captured. The internal business outsourcing. Currie [10] suggested five categories for measuring
processes category focuses on core processes aimed at customer the risk and benefit of applications provided by an ASP service,
satisfaction and at financial objective achievement. The learning including delivery and enablement, integration, management and
and growth category assesses the construction of necessary long- operation, business transformation, and customer/vendor rela-
term growth and improvement infrastructure [30]. tionship. This framework provides a difference in viewpoint
BSCs have been used to measure the performance of a wide between ASPs and their customers regarding key performance
variety of businesses. For instance [71], demonstrated the indicators (KPIs) of ASP services. Susarla et al. [58] suggested that
application of BSCs at an institutional level in a collaborative perceived provider performance has a positive impact on user
effort to develop a performance measurement framework for the satisfaction with an ASP. Further expectations about ASP services
Food Research Institute (FRI). Withanachchi et al. [65] applied BSCs have a significant influence on any performance evaluation of ASPs.
to evaluate an organizational development program (TQM) that Leam and Lee [37] proposed items for auditing and verifying the
was implemented at a tertiary-care public hospital. Moreo et al. reliability of ASP service through a survey of 35 Korean companies.
[46] suggested that BSCs could be used by managers to quantify the By using categories, including network, data center, application,
environmental and financial impact of a company and to help and security and customer support, Leam and Lee evaluated three
environmental quality to stakeholders, including hospitality ASP vendors in Korea. Leam and Lee derived the ASP life cycle and
owners and stockholders. Smandek et al. [55] developed and auditing items based on the evaluation results. The items for
implemented a BSC system for IP management to optimize auditing ASPs were classified by function and performance into 12
licensing income generation, cut costs, and keep inventors’ items, and the relative importance of those 12 items was
motivations high. Homburg et al. [24] tested and applied BSCs investigated. The items for auditing an ASP focus on assessing
to marketing performance management to show the comprehen- usefulness, extensibility, and application usability. Zviran et al.
sive relationship between a marketing performance measurement [72] suggested that perceived usefulness is one of the factors
system and firm performance as conditional on marketing affecting user satisfaction with enterprise resource planning (ERP)
alignment and market-based knowledge. Taylor and Baines [60] systems, which are typical examples of ASP services.
applied BSCs to evaluate the performance management of higher Kim et al. [33] identified three characteristics of ASPs that affect
education in terms of the formation, monitoring, and evaluation of the satisfaction of ASP customers: stability, IT infrastructure, and
strategy and policy, as well as issues of motivation. service flexibility. Kim and Kim [32] suggested five characteristics
While BSCs have been applied in various contexts, empirical of ASPs that affect ASP customer satisfaction: credibility, system
studies on the application of BSCs to specific IT services, such as currency, security, acceptability, and system support. Susarla et al.
Saas, are lacking. Further, while SMEs have difficulty rationalizing [59] used prior literature in transaction cost economics (TCE) to
their operational practices and strategic processes, there is a dearth posit that the contract design for an ASP service should consider
of comprehensive performance management system applications such factors as uncertainty in specifying service requirements,
S. Lee et al. / Information & Management 50 (2013) 553–561 555

interdependence between the ASP application and the IT systems achieve a goal, such a strategic causal relationship can form a set of
in the client organization, and the need for specific investments to causes and effects. Thus, the result (lagging indicator), focused on
address transaction costs that result due to contractual incom- each business’ strategy, may be appropriately connected to the
pleteness and opportunism. Schwarz et al. [53] suggested 10 performance cause (leading indicator), allowing all indicators in
attributes that firms consider when deciding on the outsourcing of each BSC category to be related to the financial goal.
applications and indicated that the three most significant drivers of
an IT application service choice were cost, risk, and vendor 3. Research model
capability. Yao et al. [70] suggested that firms who specialize in
providing software applications to other firms over the Internet Despite the wide scope of BSC application as a decision-
should be evaluated in terms of measures, such as the possibility of supporting instrument, few empirical studies on the application of
using an ASP service, software customization requirements, BSC to specific IT services such as Saas have been conducted
financial stability, and exit strategies. (Fig. 1). Further, it is necessary to support SMEs with specialized
However, these measures are limited, as they are based only on performance measurement systems such as BSCs. As a way to
the BSC customer’s perspective. Other BSC perspectives should also accomplish these research objectives, this paper examines causal
be added to Saas evaluation measures. effects in each BSC category to determine how well BSCs can be
applied to Saas. BSC proponents expect that companies who
2.2. The balanced scorecard continuously improve their capabilities (e.g., by implementing
advanced workplace practices, monitored via innovation and
The need for a comprehensive performance measurement, learning) should achieve better performance for their customers.
including knowledge components, has motivated the development All such efforts should lead to improved financial performance. Sim
of the balanced scorecard (BSC). The BSC attempts to integrate all and Koh [54] positively verified a causal relationship between
interests of key stakeholders, such as shareholders, customers, and several individual performance indicators among the four BSC
employees, on a scoreboard [30]. categories through relationship and regression analysis. They used
Recent studies have been conducted on BSC application in data obtained from 83 electronics companies in the U.S. BSC can be
various fields, including the fields of organizational and IS used as a tool to align business processes with new strategies,
performance evaluation. Lee et al. [39] proposed an approach moving away from cost reduction and toward growth opportu-
based on the fuzzy analytic hierarchy process (FAHP) and balanced nities based on more customized, value-adding products and
scorecard (BSC) processes for evaluating an IT department in the services. Wu [68] used a multiple criteria analysis tool to
Taiwanese manufacturing industry. The results offered guidance determine causal relationships between key performance indica-
for Taiwanese IT departments in the manufacturing industry tors for each BSC perspective to create a visualized strategy map
regarding strategies for improving department performance. with logical links to improve banking performance.
Huang [25] suggested the use of an analytic hierarchy process If cause-and-effect relationships are not adequately reflected on
(AHP) to prioritize the measures and strategies of a BSC framework. a balanced scorecard, the scorecard will not translate to a
This was done with the intention of developing an intelligent BSC company’s vision and strategy. Sohn et al. [56] posited that a
planning and management decision support system for strategic BSC’s merit is that it seeks a balance between financial and non-
planning. Wu et al. [69] suggested a Fuzzy Multiple Criteria financial measures, categorized into financial, customer, and
Decision Making (FMCDM) approach for banking performance internal business processes, and innovation and learning factors.
evaluation based the four perspectives of BSCs. Wu et al. Firms using IS must determine which specific BSC measures to
summarized performance indexes to fit the banks to construct a focus on and which to ignore. However, relatively few studies have
hierarchical framework of performance evaluation; Asosheh et al. been carried out on the specific weights of each BSC perspective.
[2] adopted an integrated use of the balanced scorecard (BSC) and Northcott and Taulapapa [48] indicate the need for improved
data envelopment analysis (DEA), and proposed a new approach to theorization on several issues that present particular challenges for
IT project selection. This approach used BSCs as a comprehensive BSC practice in the public sector: BSC measure modification,
framework for defining IT project evaluation criteria and adopted measure design that captures important qualitative outcomes,
DEA as a nonparametric technique for ranking IT projects. Velcu ‘‘customer’’ suggestion and achievement of a genuine multi-
[63] tested the interrelations between strategic alignment, stakeholder approach, and mapping BSC causality relationships.
management of ERP implementations, process changes, and the In an era of global economic competition, organizations should
business performance of companies implementing ERP, and their be creative in learning. Organizational capability for learning and
study was based on a balanced scorecard approach to analyzing creating knowledge is the cornerstone for operating businesses.
business performance. Kunz and Schaaf [36] suggested an expert Many researchers in the field of knowledge management (KM)
system for health care management for the formalization of BSC have emphasized the role of learning in knowledge management
evaluation. They provided a general definition of an indicator systems (KMS) based on the interrelatedness of learning and
system for each BSC category for clinical application. knowledge. Organizational knowledge is defined as the improve-
BSCs will be able to represent and deliver strategies according ment of actions through knowledge acquisition. Organizational
to each business, rather than offering the simple combination of
financial and non-financial indicators sorted by individual
categories. The strategy of BSC organization is represented by Learning and
Growth H2
performance connected through causal relationships and ongoing
performance indicators. In general, final performance indicators, or Customer’s Financial
‘‘lagging indicators,’’ are represented by improved financial Performance H4
H1 Performance
performance, which is the ultimate goal of the strategy.
Meanwhile, ongoing performance indicators, or ‘‘leading indica- H3
tors,’’ inform all organization members of what will be performed Internal Business
in learning and business processes [30]. Because a strategy is the
hypothetical representation of cause and effect regarding the
capability of an organization to correspond to market changes to Fig. 1. Research model – causal relationship among Saas measures.
556 S. Lee et al. / Information & Management 50 (2013) 553–561

learning occurs when individuals and subunits acquire knowledge In addition, how the effect of internal business processes on
after understanding the possibility of organizational change. both customer and financial performance, as well as customer
Organizational learning improves the potential capability for performance, affect financial performance can be investigated. The
effective actions of organizations and individuals through im- goals of internal business processes in the BSC model are to
proved business processes and customer service. innovate and improve the process of identifying and satisfying
In KM, organizational learning is established across entire customer demand, as well as to provide excellent customer
organizational hierarchies when the skills and knowledge acquired management service afterward. Customers can then recognize that
by individuals are internalized into organizational entities. the service company provides the best customer value, and thus
Organizations become efficient and skillful initiators of change customer satisfaction increases. The customer’s performance
by creating, acquiring, and transferring knowledge. Organizational facilitates market share and customer profitability in the target
improvement is determined by learning processes that require the market, allowing financial goals to be achieved.
addition or change of mental models. Organizations can create Previous studies have suggested that customer and financial
knowledge continuously from inquiry into understanding of new performance BSC measures are causally interrelated. For instance,
environments [13]. Ittner and Larcker [26] studied the relationship between customer
Improvement in customer service depends on the effectiveness satisfaction and financial performance by using various data sorted
of situational behaviors in their contingencies. Under the rapid, by company, business, and customer. As a result, the measured
innovative, and discontinuous change of environments, the value of customer satisfaction had a significant impact on future
assumptions lying beneath work practices stored in a knowledge financial performance. Behn and Riley [4] demonstrated, in their
repository should be continuously improved for better customer research, the announcement that customer satisfaction had an
service. Thus, organizations that do not engage in creative learning effect on American airline companies’ future financial perfor-
through KM increase entropy, which ultimately leads to failure in mance, i.e., that non-financial indicators function as leading
customer service. Top management’s view of the business indicators of quarterly financial performance.
environment should be adjusted in a timely manner according The ultimate aim of many balanced IS scorecards is to support IS
to changes in environments. Successful customer service compa- management, such that the overall financial outcomes of the
nies have better and more rapid creative learning. Mason [43] enterprise are improved [42]. This study suggests, based on this
indicated that one crucial measure, besides financial performance, premise, that balanced IS scorecards should improve financial
is learning, which is necessary in assessing the extent to which performance. To realize financial performance, customer perfor-
organizations acquire new knowledge to maintain competitive- mance should be improved. The determinants for customer
ness. Pfeffer [51] showed that organizational perception can performance are, in turn, internal business processes, and learning
determine organizational performance and that creative learning and growth [14]. Gonzalez-Padron et al. [21] confirmed that
can result in innovation and dramatic performance improvement. customer performance affects financial performance. Learning and
This section examines the effects of learning and growth on growth facilitate internal business processes. These facts lead to
customer service improvement for Saas performance. The learning the following hypotheses:
and growth of companies is a fundamental force driving customer
service performance and customer relationship management. For Hypothesis H3. Internal business processes positively affect cus-
example, better staff skills will reduce the frequency of bugs in an tomer performance after Saas is offered.
application. An application with fewer bugs will be more likely to
Hypothesis H4. Customer performance positively affects financial
meet end-user expectations. This, in turn, will enhance the support
performance after Saas is offered.
of customer service processes. Nonfinancial indicators, such as the
customer satisfaction index and customer loyalty, are prerequi-
sites and pre-conditional indicators for financial indicators [4,26]. 4. Research method
Customer performance has been proven to be a decisive factor in
financial performance, as learning and growth affect financial 4.1. Measurement of variables
performance through customer performance. For instance, using
data from multinational corporations, Gonzalez-Padron et al. [21] Most of the variables in the model are measured by items
used BSCs to assess how organizational learning affects actions written in the form of a statement with which the respondent is to
relating to global marketing strategy and subsequent financial agree or disagree on a 7-point Likert-type scale. The latent
performance. variables and measurement items used in this study are shown in
Thus, organizational change based on learning is a crucial factor Table 1. The measurement items were measured by partially
for improved organizational process and customer service. Top revising the leading and lagging indicators suggested in the BSC
management’s ability to understand and learn in uncertain and model. The research variables consisted of four measures,
competitive environments is important for business processes and including learning and growth, internal business processes,
customer service. Top management interprets information on customer performance, and financial performance.
behalf of organizations, and the creative learning of top manage- The measures are based on and adapted from previous studies
ment likely affects the creative learning of organizations. Thus, on intellectual capital or BSCs, such as [7,14,22,42,44,40,50,57],
organizations should acquire knowledge of business environments and so on.
and perform learning processes faster than competitors to The measure of learning and growth represents the extent to
maintain good business and customer relations. This increases which using Saas continued employee learning and development,
the demand for efficient processes based on KM tools and new as well as knowledge sharing, to make efficient use of business
infrastructure capability [6]. resources. Saas and knowledge sharing can promote learning and
development in an organization. Internal business processes
Hypothesis H1. Learning and growth positively affect internal indicate the extent to which value is brought to the customer
business processes in the client’s organization after Saas is offered. via efficient use of business resources. Customer performance
indicates the extent to which the customer market and service are
Hypothesis H2. Learning and growth positively affect the custo- improved. Financial performance indicates return on investment
mer’s performance after Saas is offered. and created shareholder value.
S. Lee et al. / Information & Management 50 (2013) 553–561 557

Table 1
Research variables.

Latent variables Measurement items Sources

Learning and Growth (LG) The work process is processed extensively through Saas (LG1). Roos and Roos [52]
Extensive knowledge sharing within the organization exists after Saas is adopted (LG2). Epstein and Manzoni [14]
Extensive knowledge sharing with other organizations (trading partners) Kaplan and Norton [29,30]
exists after Saas is adopted (LG3). Harvey and Lusch [22]
Lipe and Salterio [40]
Kim et al. [33]

Internal Business The customer order processing time is greatly reduced after Saas is adopted (IP1). Martinsons et al. [42]

Process Performance (IP) The customer request time limit is strictly followed after Saas is adopted (IP2). Olve et al. [49]
Kim et al. [33]

Customer Performance (CP) Customer service is greatly improved after Saas is adopted (CP1). Hoffecker and Goldenberg [23]
The company experiences a great annual average increase in its customer market (CP2). Butler et al. [7]
Frigo et al. [17]
Epstein and Manzoni [14]
Harvey and Lusch [22]
Martinsons et al. [42]
Petty and Guthrie [50]
Lipe and Salterio [40]
Stewart [57]

Financial Performance (FP) The ordering cost is greatly reduced after Saas is adopted (FP1). Hoffecker and Goldenberg [23]
The company’s revenue is greatly increased every year after Saas is adopted (FP2). Butler et al. [7]
Johnson [28]
Epstein and Manzoni [14]
Harvey and Lusch [22]
Kaplan and Norton [29,30]
Lipe and Salterio [40]
Mendoza and Zrihen [45]

Learning and growth include the percentage of work processed

Table 2
through Saas, and knowledge sharing within an organization and
Profile of sample.
with other organizations (trading partners) after Saas is offered.
The category of internal business processes is composed of Number of Percentage
companies (%)
customer order processing time reduction, observance of customer
request time limits, and the improvement of customer service after Industry of the company
Saas is offered. Customer performance consists of the annual Manufacturing industry 22 22
Retail industry 19 19
average increase in the customer market, cost reduction in
Service industry 51 51
ordering, and revenue increase per year after Saas is offered. Others 8 8
Total 100 100
4.2. Data collection
Size of company (Number of employees) (mean = 79)
1–10 39 39
Five-hundred interviewees were selected randomly among 11–50 42 42
SMEs registered by the Small & Medium Business Administration More than 50 19 19
Total 100 100
in Korea. Data were collected using a structured questionnaire, in
Korean, through telephone and personal interviews with IT staff Years of Saas adoption (years) (mean = 3)
members who used Saas. The sample was composed of companies 1–2 43 43
3–4 38 38
that are diverse in terms of business area, corporate size, and so on.
More than 4 19 19
Among the 500 corporations, 101 corporations were included in Total 100 100
the final sample, after excluding missing values and incomplete
Percentage of Saas use in total processes (%) (mean = 45.2)
responses. IT corporations accounted for 51% (51 corporations) of
0–10 22 22
the 101 corporations that responded, while non-IT corporations 11–50 41 41
made up 49% (49 corporations) of the sample. The descriptive More than 50 37 37
statistics of the sample data are shown in Table 2. Organizations Total 100 100
within the sample had an average of 79 employees and had Monthly Saas usage fee (dollar) (mean = 85)
adopted Saas for an average of three years. The average percentage 0–10 41 41
of Saas use was 45.2%. The majority of applications using Saas were 11–30 20 20
31–100 24 24
core businesses (sales, production, purchasing, and logistics
More than 100 15 15
applications) (71%). Total 100 100

Applications using Saas

5. Results
Supporting applications (accounting, 36 36
personnel, and administration)
5.1. Assessment of measurement model Core business (sales, production, 71 71
purchasing, and logistics applications)
To ensure convergent and discriminant validity, exploratory Business-to-business applications 32 32
Total 100 100
factor analysis with Varimax rotation was performed to produce a
558 S. Lee et al. / Information & Management 50 (2013) 553–561

Table 3
Exploratory factor analysis.

Learning and Growth (LG) Internal Business Process Performance (IP) Customer’s Performance (CP) Financial Performance (FP)

LG1 0.814 0.383 0.372 0.346

LG2 0.929 0.651 0.683 0.462
LG3 0.851 0.014 0.013 0.037
IP1 0.674 0.954 0.711 0.557
IP2 0.752 0.966 0.85 0.593
C1 0.716 0.825 0.966 0.624
C2 0.701 0.754 0.965 0.674
F1 0.586 0.612 0.682 0.949
F2 0.384 0.505 0.571 0.927

The bold values are the largest factor loading in the low.

cross-loading matrix. The matrix shows a correlation between Table 4 shows individual item factor loading and each latent
latent variables and measurement items, suggesting that the variable’s CR and AVE. All loadings are significant at a = 0.001.
measurement items load highly on their theoretically assigned Because the CRs of all eight-measurement items shown in Table 4
latent variables, while not loading highly on other factors [19]. are higher than 0.70, all latent variables can be said to have
Table 3 shows the cross-loadings of individual items compared convergent validity. The factor loadings of all items exceed 0.70.
across all latent variables. Each item is assigned highly to a Therefore, the research variables have strong reliability and
respective latent variable, thus ensuring convergent and discrimi- validity.
nant validity. Additionally, the AVE square root can be used to examine
Data analysis was performed using the partial least squares discriminate validity. As suggested by Fornell and Larcker [15],
(PLS) method. The use of PLS has increased in IS fields [9]. PLS, a diagonal elements should be larger than the entries in correspond-
component-based structural equation modeling technique, is ing rows and columns. Table 5 indicates that the AVE square root of
similar to regression but simultaneously models paths among a latent variable (the entry in the diagonal of Table 5) is larger than
variables. A large sample size is required, and standard distribution the correlation of other latent variables. The correlations among all
is assumed for SEM. In contrast, PLS, introduced by Wold et al. [66], constructs are well below 0.70, suggesting that all constructs are
focuses on maximizing the variance of dependent variables distinct from one another.
explained by an independent variable. The PLS approach places
minimal restrictions on sample size and residual distribution. In 5.2. Testing of research hypotheses
addition, a PLS algorithm allows each indicator to vary in how
much it contributes to the composite score of the latent variable The structural model estimation and research hypotheses tests
instead of assuming equal weights for all indicators of a scale. were performed using the PLS method, as shown in Fig. 2 and
Tobia [61] posited that PLS is useful in screening out negligible Table 6.
factors affecting the dependent variable. In the entire sample, three of four path coefficients are higher
A reliability test was performed using Cronbach’s Alpha and than 0.5 (LG-IP: 0.746; IP-CP: 0.610; CP-FP: 0.672). All of the paths
composite reliability (CR). Table 4 shows that there is no significant in the research model are significant and positive. All four research
defect in internal consistency. CR greater than 0.70 implies that a hypotheses are acceptable for the entire sample, as well as for the
construct retains both internal consistency and convergent validity non-IT corporation sample. This acceptability indicates that causal
[64]. Table 4 additionally illustrates how each item contributes to
the total reliability of each construct. Tests were only conducted for
the construct variable ‘‘LG’’, which has more than three indicators. Table 5
Correlations among latent variables. (The numbers in diagonals are the square root
To test validity, factor loading and average variance extracted of the average variance extracted.).
(AVE) were examined. AVE measures the variance percentage
captured via a construct by showing the variance sum ratio Latent variables LG IP CP FP

captured by the construct and the measurement variance. It is Learning and Growth (LG) 0.834
acceptable when an individual item factor loading is greater than Internal Business Process 0.734 0.960
Performance (IP)
0.70 and an AVE exceeds 0.50 [20].
Customer’s Performance (CP) 0.746 0.818 0.966
Financial Performance (FP) 0.526 0.6 0.672 0.960
Table 4
Reliability and validity of research variables.
Measurement items Loading Composite
Reliability (CR) 0.814** 0.852** 0.826**

Learning and Growth (LG)

LG1 0.8290 0.872 Learning and H2
LG2 0.8537 Growth 0.610**
LG3 0.8175 0.672**
H1 Customer’s Financial
0.746** Performance
Internal Business Process Performance (IP) H3 Performance
IP1 0.9603 0.960 0.279*
IP2 0.9603 0.965** 0.950** 0.927**
Internal Business
Customer’s Performance (CP) Process CP1 CP2 FP1 FP2
CP1 0.9659 0.965
CP2 0.9659 0.954**

Financial Performance (FP)

FP1 0.9603 0.937
FP2 0.9603
Fig. 2. PLS analysis results for the entire sample. *p < 0.05, **p < 0.01.
S. Lee et al. / Information & Management 50 (2013) 553–561 559

Table 6
Test of research hypotheses.

Research hypotheses Entire sample Sample of IT companies Sample of non-IT companies

** **
H1: Learning and Growth ! Internal Business Process Performance 0.746 0.746 0.761**
H2: Learning and Growth ! Customer’s Performance 0.279* 0.107 0.452**
H3: Internal Business Process Performance ! Customer’s Performance 0.610** 0.760** 0.467**
H4: Customer’s Performance ! financial performance 0.672** 0.576** 0.773**
p < 0.05.
p < 0.01.

relationships exist among learning and growth, internal business specific IT services, such as Saas. Further, this study provides an
processes, customer performance, and financial performance. This empirical basis for using BSC as a decision-supporting instrument
implies that for Saas customer companies, the financial category of by illuminating cause-and-effect relationships and time-delay
BSC measurement focuses on generating satisfactory return on elements among measures (these relationships are still elusive
investment and creating shareholder value, while the three other [47]).
categories can be explained as determinants of financial success. This study uses BSCs as a lens for suggesting four measures. The
Customer performance represents value creation for customers intention was to determine the leading and lagging indicators. This
and is important in explaining how customers perceive their determination improves the understanding of leading and lagging
performance. The important way to increase customer service and indicators of BSC categories based on the balance between
market is to have massive numbers of employees servicing financial and non-financial analysis.
customers, or have fewer employees who use time management
more efficiently and who are supported by excellent IT-based 6.2. Implications for practitioners
processes. Creating value for customers only translates into
shareholder value if it is based on effective and efficient key SMEs, which comprise the majority of the firms in the study
internal processes. Finally, the study results indicate that to ensure sample, consider investment in enterprise applications to be costly
that the Saas customer company will be appreciated by future and tend to avoid taking efforts away from their main business
customers and will continue to make excellent use of its processes, area. To focus on main business activities, SMEs are more likely to
the organization and its employees must continue organizational use Saas, as it can provide a viable alternative to costly in-house
learning growth. The learning and growth category should thus information systems. The high use of Saas and the high percentage
group indicators capturing the company’s performance through of its use in core applications show that SMEs use Saas strategically
the use of Saas with respect to learning based on information to fill a gap in IS resources and capabilities, given that Saas provides
sharing and innovation. a cost-efficient and rapid solution to these difficulties. Knowledge
In IT corporations, learning and growth fail to affect customer acquisition on Saas was often informal and ad hoc, usually induced
performance. In contrast, learning and growth significantly by vendor hype rather than through a rigorous evaluation of
influence customer performance in non-IT corporations. Therefore, vendor capabilities. To address this problem, BSCs can provide a
corporation type – IT vs. non-IT – moderates and affects the template for evaluating the performance of application out-
relationship between learning and growth, and customer perfor- sourcing.
mance. For non-IT companies, learning and growth are more The results of this study suggest that BSCs effectively assess four
important for the improvement of customer service and market causally interrelated measures, i.e., learning and growth, internal
than they are for IT companies. Knowledge sharing and Saas use are business processes, customer performance, and financial perfor-
considered natural and taken for granted by IT companies, as these mance. To increase financial measures, customer performance
processes have a lesser influence on customer service and market. should be promoted. This, in turn, requires the enhancement of
For non-IT companies, there exists more opportunity for improve- learning and growth and internal business processes. Financial
ment in customer service and market through learning based on measurement is the most lagging indicator and is effective in
information sharing and use of Saas. summarizing previous financial activities into easily predictable
economic results. Customer performance assesses the extent to
6. Implications which the target market is captured. To increase financial
measures, customer performance should be improved and
6.1. Implications for researchers promoted.
To increase customer performance, employee learning and
Despite rapid growth in the Saas market, no research has been internal business process efficiency should be improved. Internal
conducted on Saas measurement based on BSCs. This study provides business processes are focused on customer satisfaction and
an empirical study on BSC application to Saas and shows the financial objective achievement. Learning and growth assess the
application of comprehensive performance management systems necessary infrastructure constructed for long-term growth and
for SMEs, considering that SMEs have a hard time rationalizing their improvement. In a knowledge economy society, learning knowl-
operational practices and strategic processes. This study indicates edge is the basis for creating new capital, and the purpose of
that significant causal relationships exist among BSC measures in business is to create new knowledge. Thus, continuous learning
the Saas context, which strengthens the justification for BSC and experimentation are necessary to produce new ideas and
application as a decision-supporting instrument, given that the products; it is critical to stress the importance of an organizational
causal relationships among the four BSC measures are the focus of culture supportive of learning. The designers and developers of
BSC assessment. By showing these causal relationships, this paper KMS should focus on developing an integrated KM system that
provides a rationale for SMEs to use BSCs to assess Saas. facilitates knowledge acquisition and application, the two sub-
This study extends previous studies on ASP success in that the processes affecting creative organizational learning. KMS should
causal relationships among Saas financial performance determi- enhance both the efficiency of existing knowledge use by processing
nants are investigated. This study provides an assessment of information and the effectiveness of new knowledge creation [41].
comprehensive performance management systems for SMEs of This study reinforces arguments that KMS developers should
560 S. Lee et al. / Information & Management 50 (2013) 553–561

concentrate on both creation of new knowledge and the use of [16] G.A. Fowler, B. Worthen, The internet industry is on a cloud – whatever that may
mean, Wall Street Journal – Eastern Edition 253 (70), 2009, pp. A1–A10.
existing knowledge. [17] M.L. Frigo, K.R. Krumwiede, Automating the Balanced Scorecard – Maximizing
Corporate Performance Through Successful Enterprise-Wide Deployment, Gentia
7. Conclusion Software plc, 1998.
[18] P. Garengo, S. Biazzo, Unveiling strategy in SME through balanced scorecard
implementation: a circular methodology, Total Quality Management & Business
Because it is impossible, in a competitive environment, to Excellence 23 (1), 2012, pp. 79–102.
assess an organization merely according to financial performance [19] D. Gefen, C.M. Ridings, IT Acceptance: Managing User-IT Group, 2003.
[20] D. Gefen, D.W. Straub, M. Boudreau, Structural equation modeling and regression:
measures, it is necessary to measure the performance of Saas by guidelines for research practice, Communications of the Association for Informa-
using a BSC, which balances leading and lagging factors, and which tion Systems 4 (7), 2000.
also considers financial and non-financial factors. This paper [21] T. Gonzalez-Padron, B. Chabowski, G.T.M. Hult, D.J. Ketchen, Knowledge manage-
ment and balanced scorecard outcomes: exploring the importance of interpreta-
examines the causal relationships among the four BSC categories
tion, learning and internationality, British Journal of Management 21 (4), 2010,
that explain the performance of Saas. pp. 967–982.
Some issues for future research exist. First, practical case [22] M.G. Harvey, F.R. Lusch, Balancing the intellectual capital books: intangible
liabilities, European Management Journal 17 (1), 1999, pp. 85–92.
studies of corporations that have adopted BSC-based performance
[23] J. Hoffecker, C. Goldenberg, Using the balanced scorecard to develop company
monitoring systems are a promising avenue for study. This line of wide performance measures, Journal of Cost Management fall, 1994, pp. 5–17.
research can practically prove that key performance measures of [24] C. Homburg, M. Artz, J. Wieseke, Marketing performance measurement systems:
learning and growth improve other measures, such as customer does comprehensiveness really improve performance? Journal of Marketing 76
(3), 2012, pp. 56–77.
performance and internal business processes, as time passes. [25] H.-C. Huang, Designing a knowledge-based system for strategic planning: A
Second, in analyzing moderating effects, other moderating balanced scorecard perspective, Expert Systems with Applications 36 (January
variables, such as corporate strategy type, market positioning type, (1)), 2009, pp. 209–218.
[26] C.D. Ittner, D.F. Larcker, Are non-financial measures leading indicators of financial
and main service and product type, can be used. Further, it is performance? An analysis of customer satisfaction Journal of Accounting Re-
necessary to carry out future studies to validate the reason that key search Supplement 36, 1998, pp. 1–35.
measures of learning and growth do not affect customer [27] B.-K. Jeong, A.C. Stylianou, Market reaction to application service provider (ASP)
adoption: an empirical investigation, Information & Management 47 (3), 2010,
performance in IT corporations. pp. 176–187.
This study has some limitations. First, due to the insufficient [28] S.D. Johnson, Identification and selection of environmental performance indica-
sample size, it is not possible to further sub-categorize the sample tors: application of the balanced scorecard approach, Corporate Environmental
Strategy 5 (4), 1998, pp. 35–41.
to test moderating effects. More detailed categories must be
[29] R.S. Kaplan, R.D. Norton, Using the balanced scorecard as a strategic management
considered in terms of IT usage or sophistication, rather than just system, Harvard Business Review 1996, pp. 75–85.
two categories (i.e., IT corporations versus non-IT corporations), as [30] R.S. Kaplan, D.P. Norton, The Strategy-Focused Organization: How Balanced
Scorecard Company thrive in the New Business Environment, Harvard Business
used in this study. Second, there could be missing or unconsidered
School Press, 2001.
measurement items for learning and growth, internal business [31] T. Kern, J. Krijger, L. Willcocks, Exploring ASP as sourcing strategy: theoretical
processes, customer performance, and financial performance, even perspecitves, proposions for practice, Journal of Strategic Information Systems 11,
though this study developed or adapted each item based on 2002, pp. 153–177.
[32] G. Kim, E. Kim, An exploratory study of factors influencing ASP (application service
previous literature. Each Saas-adopting corporation has its own provider) success, Journal of Computer Information Systems 48 (3), 2008, p. 118.
main business field and performance characteristics, and this can [33] J.W. Kim, H.S. Han, Y.J. Choi, J.A. Woo, S.C. Park, Evaluation of SME’s ASP Adoption
also affect the response data. .Effects on Economic Performance, National Computerization Agency, Korea, 2006
[35] M. Kumru, A balanced scorecard-based composite measuring approach to asses-
sing the performance of a media outlet, Service Industries Journal 32 (5), 2012, pp.
References 821–843.
[36] H. Kunz, T. Schaaf, General and specific formalization approach for a balanced
[1] F. Altaf, D. Schuff, Taking a flexible approach to ASPs, Communications of the ACM scorecard: an expert system with application in health care, Expert Systems with
53 (2), 2010, pp. 139–143. Applications 38 (March (3)), 2011, pp. 1947–1955.
[2] A. Asosheh, S. Nalchigar, M. Jamporazmey, Information technology project eval- [37] C. Leam, H. Lee, Development of certification and audit processes of application
uation: an integrated data envelopment analysis and balanced scorecard ap- service provider for IT outsourcing, Technovation 24 (1), 2004, pp. 63–71.
proach, Expert Systems with Applications 37 (August (8)), 2010, pp. 5931–5938. [38] H. Lee, J. Kim, J. Kim, Determinants of success for application service provider: An
[3] BBC News, Cloud computing for business goes mainstream, 06 May, 2010 http:// empirical test in small businesses, International Journal of Human–Computer
www.bbc.co.uk/news/10097450. Studies 65 (9), 2007, pp. 796–815.
[4] B.K. Behn, R.A. Riley, Using non-financial information to predict financial perfor- [39] A.H.I. Lee, W.-C. Chen, C.-J. Chang, A fuzzy AHP and BSC for evaluating perfor-
mance: the case of the U.S. airline industry, Journal of Accounting, Auditing and mance of IT department in the manufacturing industry in Taiwan, Expert Systems
Finance 14, 1999, pp. 29–56. with Applications 34 (January (1)), 2008, pp. 96–107.
[5] T.S. Behrend, E.N. Wiebe, J.E. London, E.C. Johnson, Cloud computing adoption and [40] M.G. Lipe, S.E. Salterio, The balanced scorecard: judgmental effects of common
usage in community colleges, Behaviour & Information Technology 30 (2), 2011, and unique performance measures, Accounting Review 75 (3), 2000, pp. 283–
pp. 231–240. 298.
[6] J. Bergman, A. Jantunen, J.M. Saksa, Managing knowledge creation and sharing – [41] Y. Malhotra, Organizational controls as enablers and constraints in successful
scenarios and dynamic capabilities in inter-industrial knowledge networks, knowledge management systems implementation, in: Y. Malhotra (Ed.), Knowl-
Journal of Knowledge Management 8 (6), 2004, pp. 63–76. edge Management and Business Model Innovation, Idea Group Publishing,
[7] A. Butler, S.R. Letza, B. Neale, Linking the balanced scorecard to strategy, Long Hershey, PA, 2001 , pp. 326–336.
Range Planning 30 (2), 1997, pp. 242–253. [42] M. Martinsons, R. Davison, D. Tse, The balanced scorecard: a foundation for the
[8] D. Catteddu, G. Hogben, Cloud computing: benefits, risks and recommendations strategic management of information systems, Decision Support Systems 25 (1),
for information security, European Network and Information. Security Agency 1999, pp. 25–36.
(ENISA) 2009, pp. 1–125. [43] J. Mason, From e-learning to e-knowledge, in: M. Rao (Ed.), KM Tools and
[9] D.R. Compeau, C.A. Higgins, Application of social cognitive theory to training for Techniques: Practitioners and Experts Evaluate KM Solutions, Butterworth-
computer skills, Information System Research 6 (2), 1995, pp. 118–143. Heinemann, Oxford, UK, 2004 , pp. 320–328.
[10] W.L. Currie, A knowledge-base risk assessment framework for evaluating web- [44] A. Mayo, The role of employee development in the growth of intellectual capital,
based application outsourcing projects, International Journal of Project Manage- Personnel Review 29 (4), 2000, p. 2000.
ment 21 (3), 2003, pp. 207–217. [45] C. Mendoza, R. Zrihen, Measuring Up, Financial Up, 2001 pp. 26–29.
[11] M. Cusumano, Technology strategy and management: cloud computing and SaaS as [46] A. Moreo, F.J. DeMicco, L. Xiong, Towards a model to measure the quality of
new computing platforms, Communications of the ACM 53 (4), 2010, pp. 27–29. environmental sustainability: the hospitality environmental scorecard, Journal of
[13] A. DuToit, Knowledge: a sense making process shared through narrative, Journal Quality Assurance in Hospitality & Tourism 10 (1), 2009, pp. 44–58.
of Knowledge Management 7 (3), 2003, pp. 27–37. [47] S. Nielsen, E.H. Nielsen, Discussing feedback system thinking in relation to
[14] M. Epstein, J.-F. Manzoni, Implementing corporate strategy: from tableaux De Bord scenario evaluation in a balanced scorecard setup, Production Planning & Control
to balanced scorecards, European Management Journal 16 (2), 1998, pp. 190–203. 23 (6), 2012, pp. 436–451.
[15] C. Fornell, D.F. Larcker, Evaluating structural equation models with unobservable [48] D. Northcott, T.M. Taulapapa, Using the balanced scorecard to manage perfor-
variables and measurement error, Journal of Marketing Research 18 (1), 1981, pp. mance in public sector organizations: issues and challenges, International Journal
39–50. of Public Sector Management 25 (3), 2012, pp. 166–191.
S. Lee et al. / Information & Management 50 (2013) 553–561 561

[49] N.-G. Olve, J. Roy, M. Wetter, Performance Drivers: A Practical Guide to Using the [70] Y. Yao, E. Watson, B. Kahn, Application service providers: market and adoption
Balanced Scorecard, Wiley, 1999. decisions, Communications of the ACM 53 (7), 2010, pp. 113–117.
[50] R. Petty, J. Guthrie, Intellectual capital literature review: measurement, reporting, [71] R. Yawson, W. Amoa-Awua, A. Sutherland, D. Smith, S. Noamesi, Developing a
and management, Journal of Intellectual Capital 1 (July (2)), 2000, pp. 155–176. performance measurement framework to enhance the impact orientation of the
[51] J. Pfeffer, Changing mental models: HR’s most important task, Human Resource Food Research Institute, Ghana, R&D Management 36 (2), 2006, pp. 161–172.
Management 44 (2), 2005, pp. 123–128. [72] M. Zrivan, N. Pliskin, R. Levin, Measuring user satisfaction and perceived useful-
[52] G. Roos, J. Roos, Measuring your company’s intellectual performance, Long Range ness in the ERP context, Journal of Computer Information Systems 45 (3), 2005,
Planning 30 (3), 1997, pp. 413–426. pp. 43–52.
[53] A. Schwarz, B. Jayatilaka, R. Hirschheim, T. Goles, A conjoint approach to under-
standing IT application services outsourcing, Journal of the Association for Sangjae Lee is a professor at the college of business
Information Systems 10 (10), 2009, pp. 748–781. administration in Sejong University. He received PhD in
[54] K.L. Sim, H.C. Koh, Balanced scorecard: a rising trend in strategic performance Management Information Systems from the Graduate
measurement, Measuring Business Excellence 5, 2001, pp. 18–27. School of management, KAIST. His research interests
[55] B. Smandek, A. Barthel, J. Winkler, P. Ulbig, Balanced score card implementation include decision support systems, electronic com-
for IP rights management in a public research institution, Measuring Business merce, information systems controls, data mining,
Excellence 14 (4), 2010, pp. 65–75. and IS planning. He has published in such journals as
[56] M. Sohn, T. You, S. Lee, H. Lee, Corporate strategies, environmental forces, and Decision Support Systems, Information & Management.
performance measures: a weighting decision support system using the k-nearest
neighbor technique, Expert Systems with Applications 25 (3), 2003, pp. 279–292.
[57] W.E. Stewart, Balanced scorecard for projects, Project Management Journal 32 (1),
2001, pp. 38–53.
[58] A. Susarla, A. Barua, A.B. Whinston, Understanding the service component of
application service provision: An empirical analysis of satisfaction with ASP Sung Bum Park received a master degree in Manage-
services, MIS Quarterly 27 (1), 2003, pp. 91–123. ment Information and a Ph. D. degree in Management
[59] A. Susarla, A. Barua, A.B. Whinston, Transaction cost perspective of the ‘‘software Science from Korea Advanced Institute of Science and
as a service’’ business model, Journal of Management Information Systems 26 (2), Technology. He has been working as a senior researcher
2009, pp. 205–240. for National Information Society Agency since then. His
[60] J. Taylor, C. Baines, Performance management in UK universities: implementing current research interests include performance evalua-
the balanced scorecard, Journal of Higher Education Policy & Management 34 (2), tion of information system and digital content distri-
2012, pp. 111–124. bution.
[61] R.D. Tobia, An Introduction to Least Squares Regression, SAS Institution Inc., 1999.
[63] O. Velcu, Strategic alignment of ERP implementation stages: an empirical inves-
tigation, Information & Management 47 (April (3)), 2010, pp. 158–166.
[64] C.E. Werts, R.L. Lin, K.G. Joreskog, Intraclass reliability estimates: testing struc-
tural assumptions, Educational & Psychological Measurement 34 (1), 1974, pp.
25–33. Gyoo Gun Lim is a professor of MIS at School of
[65] N. Withanachchi, Y. Handa, K.K.W. Karandagoda, P.P. Pathirage, N.C.K. Tennakoon, Business, Hanyang University, Seoul, Korea. He received
D.S.P. Pullaperuma, TQM emphasizing 5-S principles/A breakthrough for chronic PhD in Management Engineering from Korea Advanced
managerial constraints at public hospitals in developing countries, International Institute of Science and Technology (KAIST) of Korea in
Journal of Public Sector Management 20 (3), 2007, pp. 168–177. 2001. His current research interests include Innovative
[66] H. Wold, Partial least squares, in: S. Kotz, N. Johnson (Eds.), Encyclopedia of business model, e-business, IT service, and intelligent
Statistical Sciences, Wiley, New York, NY, 1985, pp. 58–591. systems.
[68] H.-Y. Wu, Constructing a strategy map for banking institutions with key perfor-
mance indicators of the balanced scorecard, Evaluation & Program Planning 35
(3), 2012, pp. 303–320.
[69] H.-Y. Wu, G.-H. Tzeng, Y.-H. Chen, A fuzzy MCDM approach for evaluating banking
performance based on balanced scorecard, Expert Systems with Applications 36
(August (6)), 2009, pp. 10135–10147.