Use of Knowledge Management Systems and The Impact On The Acquisition of Explicit Knowledge

JOURNAL OF INFORMATION SYSTEMS
Vol. 22, No. 2

Fall 2008
pp. 77101
Use of Knowledge Management Systems

and the Impact on the Acquisition
of Explicit Knowledge
Holli McCall
University of Connecticut
Vicky Arnold
University of Central Florida and
University of Melbourne
Steve G. Sutton
University of Central Florida and
University of Melbourne
ABSTRACT: In an era where knowledge is increasingly seen as an organizations most
valuable asset, many firms have implemented knowledge-management systems (KMS)
in an effort to capture, store, and disseminate knowledge across the firm. Concerns
have been raised, however, about the potential dependency of users on KMS and the
related potential for decreases in knowledge acquisition and expertise development
(Cole 1998; Alavi and Leidner 2001b; OLeary 2002a). The purpose of this study, which
is exploratory in nature, is to investigate whether using KMS embedded with explicit
knowledge impacts novice decision makers judgment performance and knowledge
acquisition differently than using traditional reference materials (e.g., manuals, textbooks) to research and solve a problem. An experimental methodology is used to study
the relative performance and explicit knowledge acquisition of 188 participants partitioned into two groups using either a KMS or traditional reference materials in problem
solving. The study finds that KMS users outperform users of traditional reference materials when they have access to their respective systems / materials, but the users of
traditional reference materials outperform KMS users when respective systems / materials are removed. While all users improve interpretive problem solving and encoding
of definitions and rules, there are significant differences in knowledge acquisition between the two groups.
Keywords: knowledge management systems; knowledge management; declarative
knowledge; explicit knowledge; knowledge acquisition; knowledge transfer; ACT-R theory.
We thank Alex Yen for his help in administering the experiments, and Mohamed Hussein, Gim Seow, Ton Cillessen,
Cliff Nelson, Jesse Dwyer, Clark Hampton, Angela Han, Sarfraz Khan, Linda Kolbasovksy, Kate Odabashian,
Sylvia Santiago, and Stan Veliotis for their assistance during the various stages of this research. We also thank
participants at a workshop at the University of Connecticut, the 2006 Mid-Year Meeting of the Information Systems
section, and the 2006 American Accounting Association Annual Meeting, and in particular the editor Brad Tuttle,
an associate editor, and two reviewers for their feedback on previous versions of the manuscript.
77
78
McCall, Arnold, and Sutton

Data Availability: Data availability is limited by Institutional Review Board guidelines
and restrictions.
I. INTRODUCTION
n recent years, organizations have increasingly realized that one of their most valuable
assets is the knowledge that is developed internally and possessed by individuals within
the organization. For instance, Nagle (1999), a knowledge manager at KPMG, noted
that one of the biggest challenges facing the firm was how to capture, store, retain, and
share the knowledge possessed by the firms professionals. Cameron (2000, 3) similarly
noted, Knowledge is power, but without the adequate management of that knowledge, the
consequences for [organizations] could be devastating. Not surprisingly, technology is
viewed as the key enabler to effective knowledge management. Early technological strategies focused on the use of intelligent systems,1 but these strategies have not been terribly
effective (Duchessi and OKeefe 1992). More recently, corporate efforts are focusing on a
class of technologies referred to as knowledge-management systems (KMS) (Leech and
Sutton 2002).
Knowledge management can be defined as the organizational efforts designed to (1)
capture knowledge; (2) convert personal knowledge to group-available knowledge; (3) connect people to people, people to knowledge, knowledge to people, and knowledge to knowledge; and (4) measure that knowledge to facilitate management of resources and help
understand its evolution (OLeary 2002a, 273). Knowledge-management systems (KMS)
focus on bringing together the explicit knowledge that exists in organizations, the knowwhat that can be easily documented and shared (Sambamurthy and Subramani 2005), such
as basic definitional information (e.g., technical terminology), procedures for performing
tasks (e.g., audit checklists), guidelines for interpretation (e.g., GAAP guidance), and previous problem resolution examples (e.g., client memos outlining solutions to issues
raised)information often referred to as three-ring binder knowledge (Dilnutt 2002). As
noted by Alavi and Leidner (2001b), KMS initially contain these types of explicit knowledge and are later expanded upon with a body of tacit knowledge that continues to grow
as users add their interpretations of the explicit knowledge to the systems knowledge base.
Tacit knowledge is the know-how that is difficult to document and emerges from experiences
(Sambamurthy and Subramani 2005). Alavi and Leidner (2001b) further note that access
and/or assimilation of the explicit knowledge in such systems is a necessary precursor to
effective use of the accumulated tacit knowledge in the system. This is consistent with
recent findings in the knowledge-based system (KBS) literature showing that novice users
gravitate toward explicit knowledge support while experienced decision makers gravitate
toward available tacit knowledge support (Arnold et al. 2006).
The use of KMS to support an organizations professionals in their decision making
through organizational knowledge creation is a double-edged sword. The ready availability
of explicit knowledge support in a KMS should allow individuals to improve decision
performance (Gonzalez et al. 2005), but the potential impact on the development of expertise by individuals within the organization remains an unknown. Alavi and Liedner (2001b)
note that some researchers raise questions as to whether KMS users may not develop their
own knowledge while relying on the expertise of others, which may lead to a lack of
expertise development in the next generation of organizational experts and ultimately a
A type of knowledge-based system that is also commonly referred to as expert systems, intelligent decision
support systems, and intelligent decision aids.
Journal of Information Systems, Fall 2008
79
dwindling of human expertise within the firm (e.g., Cole 1998; Powell 1998; OLeary
2002a).
The purpose of this study is to empirically investigate whether a KMS providing explicit
knowledge impacts the decision-making performance and the acquisition of explicit knowledge by novice users. The entire basis for the investment in and development of knowledgemanagement technologies is premised on the belief that an effective KMS should disseminate knowledge throughout the organization and provide the necessary components to
improve decision-making capabilities (Alavi and Leidner 1999). The impact of the use of
KMS on explicit knowledge acquisition is critical given that explicit knowledge provides
the foundation for and is the precursor of tacit knowledge development (Alavi and Leidner
2001b; Anderson 1987; Anderson 1990; Anderson 1993; Anderson et al. 1997; Chi et al.
1989; Roberts and Ashton 2003). As such, acquisition of explicit knowledge is a critical
component in the development and sustenance of expertise (Herz and Schultz 1999). Yet,
prior research provides little insight on the effects of contemporary decision support technologies, such as KMS, where the user initiates search and retrieval of the embedded
knowledge (Alavi and Leidner 2001b).
This study utilizes an experimental methodology to investigate the impact of KMS on
decision-making performance and acquisition of explicit knowledge. A pretest-posttest design is implemented to investigate the acquisition of explicit knowledge focusing on differences between individuals using a KMS (KMS group) versus individuals using traditional
reference materials such as office manuals and textbooks (traditional group). Results indicate that the KMS group outperforms individuals in the traditional group when they have
access to a KMS; however, the advantage disappears when the KMS is removed. Additionally, results indicate that both groups acquire various types of explicit knowledge. The
traditional group tends to encode more rules in memory, while the KMS group tends to
acquire higher-level explicit knowledge (i.e., interpretative problem-solving skills) which is
key to the formulation of tacit knowledge.
This paper contributes to the literature by experimentally examining the impact of KMS
use on the acquisition of explicit knowledge and represents an initial step in addressing the
associated knowledge transfer issues. Little empirical evidence is available on the impact
of KMS on user performance (e.g., Gonzalez et al. 2005). Rather, the extant KMS literature
generally consists of descriptive studies (Davenport et al. 1998; Alavi and Leidner 1999;
Dilnutt 2002; OLeary 2002b), design science studies (Earl 2001), and case studies (Alavi
1997; Baird et al. 1997; Bartlett 1996; Henderson et al. 1998; Thomas et al. 2001; and
Wickramasinghe and Mills 2002). Prior research has not addressed whether knowledge
transfer actually occurs (Grover and Davenport 2001). Alavi and Leidner (2001b) note that
future research needs to address if and to what degree knowledge can be transferred within
the firm. This study focuses on this gap in research and addresses knowledge acquisition
associated with KMS use.
The remainder of this paper is organized into four sections. The first section presents
the theory and associated hypotheses and research questions. The second and third sections
provide an overview of the research method and the results of the experimental study,
respectively. The fourth and final section discusses the implications of the study results and
considers opportunities for future research that could extend the research reported here.
II. BACKGROUND AND THEORETICAL DISCUSSION
Knowledge is defined as a justified true belief (Nonaka 1994, 15) and can be viewed
as a state of mind, an object, a process, a stipulation of having access to information, or a
80
capability (Nonaka 1994). In an organizational setting, knowledge is an asset that enables

firms to obtain a competitive advantage (Alavi and Leidner 1999) and is of limited value
if it is not disseminated to others (Grant 1996). Knowledge management enables organizations to leverage the collective knowledge among members of the organization and sustain
competitive advantage.
A key component of knowledge management is to provide access to stored knowledge
components to improve decision making and to facilitate knowledge acquisition by the user.
ACT-R theory (depicted in Figure 1) provides a conceptualization of the process by which
knowledge is acquired and provides a foundation for understanding how a KMS might
impact individuals knowledge-acquisition processes (Anderson 1990; Anderson 1993;
Anderson et al. 1997). While Anderson presents his theory in terms of declarative and
procedural knowledge, knowledge embedded in a KMS is generally referred to as explicit
and tacit, respectively (Alavi and Leidner 2001b; Sambamurthy and Subramani 2005). In
both literatures declarative and explicit are defined as know-what knowledge, and procedural and tacit are defined as know-how knowledge (Anderson 1993; Sambamurthy and
Subramani 2005).
Act-R theory proposes that an individual encodes or stores definitions, examples, and
rules into long-term memory and utilizes this declarative knowledge in a problem-solving
strategy called interpretive problem solving. Interpretive problem solving is defined as solving problems by analogizing from examples. These examples may come from an external
source (e.g., KMS) or be retrieved from declarative knowledge that has been encoded into
long-term memory. With practice an individual encodes an increasing number of examples,
resulting in declarative knowledge acquisition. After declarative knowledge has been acquired, the individual may then move into the procedural stage of knowledge acquisition
by compiling the analogy process into a production rule (i.e., compilation). Both declarative
encoding and interpretive problem solving must occur prior to compilation (rule creation);
FIGURE 1
Stages of Individual Knowledge Acquisition
Procedural Stage
Declarative Stage
Declarative
encoding
Definitions,
rules,
examples
Declarative
knowledge
acquisition
Compilation
(of production
rules)
Tuning (of
production
rules)
Interpretive
problem
solving
Procedural
knowledge
acquisition
81
however, declarative encoding alone can result in declarative knowledge acquisition. The
production rules (or condition-action pairs likened to if-then statements in a programming
language) created by the compilation process are then tuned (improved or enhanced), which
expands procedural knowledge (i.e., procedural knowledge acquisition). Procedural knowledge is the ability to apply and extend declarative knowledge and is acquired through
experience; thus, it is considered a key antecedent to expertise (Anderson 1993).
Production rules and the linked declarative knowledge are stored in long-term memory.
Long-term memory stores consist of declarative memory and procedural memory. The basic
unit of knowledge in declarative memory is a chunk, while the basic unit of knowledge in
procedural memory is a production rule. Declarative knowledge is factual knowledge that
can be described (definitions, rules, and examples), while procedural knowledge is mechanistic and can only be inferred by behavior (Anderson 1993). For example, declarative
knowledge would entail knowing that a bike has wheels, handlebars, pedals, a seat, and
that the pedals are used to turn the wheels while one is seated and holding on to the
handlebars. However, knowing how to ride a bike would represent procedural knowledge.
Individuals know how to ride a bike, but cannot actually describe all of the processes
required.
The primary sub-process in the declarative stage is declarative encoding of explicit
knowledge which can be described as storing experiencesinstructions, examples, rules,
definitions, and successes and failures of our own attempts. Declarative knowledge is committed to long-term memory by encoding external events or the action side of a conditionaction production pair. Declarative encoding occurs as newly identified explicit knowledge
is first considered by working memory and then may be encoded to declarative long-term
memory in chunks. Once the chunk is in long-term memory, its retrieval is controlled by
its level of spreading activation or the ease with which a chunk of knowledge can be recalled
from memory. A particular chunks level of activation is strengthened as the number of
connections or related chunks increasespreading activation represents how easily and
often the knowledge is retrieved. As the level of activation increases, the chunks can be
retrieved more easily.
When an individual has no applicable production rule instantiations (i.e., procedurallevel knowledge), they look to examples from the past to analogize to solve a problem, a
process referred to as interpretive problem solving. In this second sub-process of the declarative stage, Anderson et al. (1997) find that encoding of declarative knowledge into
long-term memory is not necessary to use interpretive problem solving. The individual
employing interpretive problem solving may simply work from declarative knowledge (definitions, examples, rules, etc.) active within working memory rather than draw upon encoded knowledge (Anderson et al. 1997).
At the core of KMS typically used by accounting firms is top-down knowledge including manuals, directories, and newsletters; work processes knowledge consisting of
working papers, proposals, client correspondence, and other engagement materials; and
customer related knowledge including customer continuity and history information
(OLeary 1998). While these various components can appear to be fairly complex, in reality
they can largely be summed up as definitions (e.g., terminology and explanations of
terminology), rules (e.g., regulations, standards, corporate policies, and interpretations
thereof), and examples (e.g., stories of how problems have been overcome, memos describing problem resolution in a given context, preferred business practices under certain conditions, and summaries of previously researched issues)the three building blocks for
declarative or explicit knowledge.
82
All of these facilities are designed to ease the mental workload and make it easier for
the user to acquire the knowledge necessary to complete the task at hand (Alavi and Leidner
2001b). To accomplish this, the KMS should allow the user to have easy access to explicit
knowledge stored in the system (e.g., definitions, rules, and examples) that can be applied
to solve the problem. The KMS should also improve the ease by which the user can find
a rule and/or example that applies to the current situation and facilitate interpretive problem
solving (Alavi and Leidner 2001b). To a certain degree, the KMS relieves the user of the
need for encoding of explicit knowledge in long-term memory as applicable knowledge
components can be readily accessed by the users active working memory. Easy access to
explicit knowledge via the KMS also reduces the likelihood the user will draw upon encoded explicit knowledge in long-term memory as drawing from long-term memory increases the effort required of the user, thus increasing mental workload (Alavi and Leidner
2001a).
Cognitive load theory suggests that as mental workload decreases, an individual will
have more working memory available for problem solving and will lead to better performance (Sweller 1988; Sweller 1989; Chandler and Sweller 1991; Chandler and Sweller
1996; Sweller and Chandler 1991; Sweller et al. 1998; Paas et al. 2003; van Merrienboer
and Sweller 2005). Thus, having explicit knowledge readily accessible via a KMS should
enhance performance. Accordingly, this leads to the first hypothesis:
H1: A user of a KMS providing access to explicit knowledge required for problem
solving will perform better than an individual using traditional reference materials.
The concern that has been raised is that this easy access to explicit knowledge that
negates the need to encode (and subsequently activate) explicit knowledge in long-term
memory may result in the individual user failing to develop foundational explicit knowledge
that is in turn needed to develop tacit knowledge and expertise (Cole 1998; Powell 1998;
Alavi and Leidner 2001b). On the other hand, the easy availability of knowledge components gives KMS users improved accessibility to a large volume of explicit knowledge (i.e.,
definitions, rules, and examples) and increases the individuals opportunities to encode this
knowledge into long-term memory. The concern is whether this explicit knowledge will be
encoded in long-term memory as effort is considered a key influence on the successful
acquisition of knowledge (Hiltz 1986).
Substantial research with other types of KBS indicate that similar high expectations by
early promoters of intelligent systems for accelerated transfer of knowledge from system
to user (Eining and Dorr 1991) did not come to fruition. Rather, explorations of the impact
on acquisition of explicit knowledge through explanation provision in intelligent systems
provide mixed evidence. Bransford et al. (1982), Franks et al. (1982), Stein et al. (1982a),
and Stein et al. (1982b) found that the precision of an explanation embedded within an
intelligent system positively affects the development of explicit knowledge. Intelligent system explanations embedded with explicit knowledge have been shown to successfully increase long-term memory storage of explicit knowledge (Smedley and Sutton 2004). Alternative research indicates that users of intelligent systems may acquire and encode less
explicit knowledge than users of other reference materials (Brody et al. 2003; Marchant et
al. 1991; Murphy 1990). Odom and Dorr (1995) also find evidence indicating that precise
explanations embedded within an intelligent system with examples actually decreased acquisition of explicit knowledge.
Prior research using intelligent systems may or may not be applicable to the class of
KBS falling under the KMS domain. In an intelligent systems environment, most of the
83
experimental studies have explanations automatically provided at relevant points in the

decision-making process. This provides the explicit knowledge for potential storage in longterm memory and a context in which to help activate the knowledge, but does not require
any effort to be exerted by the user. In these settings, the user is unlikely to store and
activate the explicit knowledge unless they also exhibit behavior consistent with intentional
learning (Anderson 1993; Smedley and Sutton 2007)behavior that requires both motivation and effort (Hiltz 1986).
In a KMS environment, two factors may counter the likelihood of encoding explicit
knowledge. First, the volume of information and multiple ways of accessing information
available in a KMS could potentially increase mental workload while retrieving the informationa factor found to impact knowledge acquisition (Rose and Wolfe 2000; Rose
2005). However, the searching capability embedded in KMS arguably makes accessibility
easier rather than more difficult and should ease mental workload. The second factor is the
perceived ease of accessibility. If the system does facilitate easy access, a user focused on
problem solving may very likely choose to retrieve available explicit knowledge from the
KMS, move this knowledge into active working memory to support interpretive problem
solving, but not feel any motivation to actually encode the knowledge (Alavi and Leidner
2001b).
These competing effects imply that knowledge acquisition will differ between users of
a KMS and those that use traditional reference materials, but it is unclear which group is
likely to acquire more knowledge. This knowledge-acquisition effect is investigated through
three research questions related to the different types of knowledge components that are
available to the decision maker. These research questions are stated in the alternative form
as follows:
RQ2: Will there be a difference in definition recall between individuals using a KMS
embedded with definitions and individuals accessing definitions through traditional reference materials?
RQ3: Will there be a difference in rule recall between individuals using a KMS embedded with rules and individuals accessing rules through traditional reference
materials?
RQ4: Will there be a difference in example recall between individuals using a KMS
embedded with examples and individuals accessing examples through traditional
reference materials?
RQ24 relate to encoding of explicit knowledge and levels of increase in encoded
knowledge. The other dimension of explicit knowledge acquisition that is of interest is
the interpretive problem solving ability. For the same reasons that there are concerns that the
KMS may impede encoding of explicit knowledge, interpretive problem solving should
strongly increase for users of the KMS. A KMS is designed to make it relatively efficient
and easy for a user to retrieve the explicit knowledge components that are needed to solve
a problem. Consistent with cognitive load theory (Sweller 1988; Sweller 1989; Chandler
and Sweller 1991; Chandler and Sweller 1996; Sweller and Chandler 1991; Sweller et al.
1998; Paas et al. 2003; van Merrienboer and Sweller 2005), this ease of effort allows the
user to easily view the knowledge components as desired while maintaining low levels of
mental workload, but at the same time the ease of access does not necessarily motivate the
user to encode the data. However, once the requisite explicit knowledge has been accessed,
the user can focus efforts on honing interpretive problem-solving skills that allow the user
84
to match available solution examples with similar decision-making tasks faced by the user
(Anderson 2000). The easy access to rules and examples should facilitate the user becoming
quite adept at using these rules and/or examples to solve similar problemsi.e., interpretive
problem solving. The hypothesis is stated as:
H5: A user of a KMS embedded with explicit knowledge (i.e., definitions, rules, and
examples) will acquire more interpretive problem-solving abilities than an individual not using a KMS.
As suggested by Andersons (1993) ACT-R theory, acquisition of explicit knowledge
is a necessary precursor to the development of tacit knowledge which is fundamental to
the establishment of expertise. Ultimately, the ability to support knowledge acquisition
by the individual user through the use of a KMS is critical to an organizations overall
knowledge-management efforts. Yet, with the questions raised earlier, one should be concerned with the direction of the change in knowledge acquisition as predicted in RQ24.
Failing to encode explicit knowledge at a level equal to or greater than that achieved through
traditional approaches should be of serious concern to knowledge managers. While there
may be variances between types of knowledge that are encoded more effectively by KMS
users than users of traditional reference materials, the cumulative effect is perhaps of
greatest concern. Variances in knowledge acquisition will also lead to variances in performance when a KMS is not available. The research question is stated as:
RQ6: Will an individual who acquires knowledge through solving problems with the
assistance of a KMS achieve a different level of unassisted problem-solving ability than will an individual who acquires knowledge through solving problems
with the assistance of traditional reference materials?
An increase in problem-solving abilities by users of a KMS would be consistent with
the belief that the rich repository of explicit knowledge components combined with easy
access facilitates a user in experiencing a greater breadth and depth of knowledge. On the
other hand, a decrease in problem-solving abilities by users of a KMS may be indicative
of technology dominance by the KMS where the user allows the system to dominate the
decision process and takes a passive role in problem resolution (Arnold and Sutton 1998),
thereby reducing knowledge acquisition by the user.
III. RESEARCH METHOD
To simulate an environment in which novice decision-makers would make a business
decision, we needed (1) participants with little or no prior knowledge to make the decision,
(2) a simple decision task that entry-level accountants might make, and (3) appropriate
reference materials to support that task. We chose a managerial accounting task where
participants were asked to make three different, but related, decisions that an entry-level
accountant might be asked to complete. In a business environment, entry-level accountants
might refer to previous examples, company manuals, or textbooks for guidance. Alternatively, they might refer to the companys KMS with these items embedded if one was
available. Students enrolled in a managerial accounting class were chosen to insure that
they had not previously acquired the requisite explicit or tacit knowledge to make the
decisions. Practicing professionals would not be feasible as they might already have
the knowledge to complete the task.
85
In order to study the effect of KMS use, a three-stage experiment was conducted to
compare the acquisition of explicit knowledge components by KMS users (KMS group)
to individuals using traditional, paper-based reference materials (traditional group). The
experiment covered three consecutive 75-minute class periods with each stage corresponding to the three class periods. During the first stage of the experiment, participants were
given an introductory lecture accompanied by a set of lecture notes. The introductory lecture
provided the participants with a base level of explicit knowledge, including definitions,
rules, and examples.
During the second stage of the experiment, participants completed a pretest recall instrument designed to assess the amount of explicit knowledge the participants had encoded
prior to treatment. Next, participants completed a series of treatment casessome participants had access to a KMS, while others had access to traditional paper-based materials
including a textbook, handouts, and lecture notes. The treatment cases required the participants to access explicit knowledge already encoded in long-term memory or available
through either the KMS or traditional reference materials. The knowledge needed to complete the treatment cases was beyond the scope of the introductory lectures; thus participants
had to access the materials provided by either the KMS or traditional reference materials
to complete the task. By completing the cases and accessing the KMS or traditional reference materials, participants had an opportunity to encode the explicit knowledge into
long-term memory.
During the third stage of the experiment, which was conducted in the subsequent class
session, the participants completed a posttest recall instrument. The time lag between sessions allowed us to assess improvements due to long-term knowledge acquisition. The
posttest recall assessed how much explicit knowledge individuals had encoded as a result
of the treatment. In addition, the participants solved a series of posttest cases. The posttest
cases provided a measure of the participants interpretive problem-solving abilities. Figure
2 provides an overview of the three experimental sessions.
Experimental Procedure
The experiment was designed to examine participant encoding of definitions, rules, and
examples, and to measure interpretive problem-solving abilities. To enable such measurements, a decision-making task requiring the use of elementary knowledge of definitions,
rules, and examples was used. The decision-making task consisted of three managerial
decisions: (1) special order, (2) sell at split-off or process further, and (3) product/department elimination.
The experiment consisted of an explicit knowledge recall instrument, three treatment
cases, and two posttest decision-making cases. The recall instrument for explicit knowledge
was developed based on common rules and definitions found in traditional reference materials. In addition, the recall instrument contained common examples that would be used
to demonstrate the various types of explicit knowledge. The recall instrument consisted of
30 multiple-choice questions. The first ten questions examined knowledge of definitions.
Each question defined a term and called for the selection of the correct term among five
answers. The next ten questions assessed knowledge of rules. The questions called for the
completion of the rule with the appropriate information from a set of five answers. The last
ten questions tested recall of examples and stated a complete example, including the answer.
Each participant was asked to recall whether he/she had seen the example, not seen the
example, or did not remember.
The recall instrument was administered twice to achieve a pretest-posttest design as
shown in Figure 2. The recall instrument was initially administered during the second
86
FIGURE 2
Overview of Experiment
EXPERIMENTAL SESSION 2
EXPERIMENTAL
SESSION 1
KMS Group
Complete:
Pretest Recall (definitions, rules,
examples)
Three Treatment Cases (while using
KMS)
Introductory Lecture
Lecture Notes
Complete:
Demographic Questionnaire
Goal Orientation Survey
Consent Form
Traditional Group
Complete:
Pretest Recall (definitions, rules,
examples)
Three Treatment Cases (while using
a textbook and lecture notes)
EXPERIMENTAL SESSION 3
Complete:
Posttest Recall (definitions, rules,
examples)
Two Posttest Cases (while referring
to a set of examples)
experimental session and was again administered after completing the treatment cases. The
posttest recall score less the pretest recall score provided a measure for the encoding of
explicit knowledge that occurred as a result of the treatment.
The cases were developed based on a cost/managerial accounting test manual
(Schoenebeck 2003). The three treatment cases were modified slightly, by including supplemental questions relating to the case, in order to incorporate all of the rules and definitions that were tested in the recall instrument. This was done to ensure that participants
needed to use either the materials that were provided in the KMS or the textbook and
lecture notes to complete the treatment cases. Researching within the materials provided to
the groups to complete the cases should improve recall in posttesting. Posttest Case 1 was
identical to a portion of Treatment Case 1, which contained supplemental questions not
included in Posttest Case 1. The purpose of Posttest Case 1 was to measure interpretive
problem-solving abilities and was embedded in both the KMS and class notes. Thus, the
participants had seen the case at least twice before completing Posttest Case 1. Posttest
Case 2 is similar but not identical to Treatment Case 3. The participants had been exposed
to the explicit knowledge required to solve this case, but had not worked this particular
case previously. The purpose of the posttest cases was to provide a measure of improvement
of interpretive problem solving resulting from the treatment.
The recall instrument, treatment cases, and posttest cases were pilot tested by 32 undergraduate managerial accounting students and five PhD students. As a result of the pilot
tests, minor grammatical changes were made and one of the supplemental questions was
replaced.
87
Participants and Incentives

In order to examine whether acquisition of explicit knowledge by a KMS user would
be greater than the user of traditional reference materials, novice-level users with minimal
knowledge of the tasks were needed as participants. While participants needed sufficient
baseline knowledge to have the ability to understand the lecture and notes in experimental
session 1, they needed to be novice enough to allow for acquisition of new explicit knowledge. As noted earlier, prior research has shown that novice users pursue explicit knowledge
explanations while solving problems (Arnold et al. 2006). As a result, undergraduate managerial accounting students were selected to participate in this experiment as they had a
base level of knowledge enabling an understanding of the task materials, but did not have
the necessary level of explicit knowledge necessary to successfully complete the experimental materials prior to the treatment. The participant pool consisted of 222 business
students enrolled in six sections of a sophomore-level managerial accounting course. The
188 participants who attended all three class sessions and completed all materials formed
the final sample, with the other 34 participants being dropped for missing one or more
sessions.
The experiment was conducted as part of the regular class material and covered three
consecutive 75-minute class periods. In order to induce participants to attend all three sessions and take the task seriously, performance-based and participation-based class bonus
points were awarded toward their final grade. In experimental session 1, points were
awarded for attending the lecture, completing the demographic questionnaire, and completing the goal-orientation survey. In experimental session 2, participants could earn points
for performance on pretest recall and for completing the treatment cases. In experimental
session 3, participants could earn points for performance on posttest recall and posttest
cases. In addition, the material covered during these three class sessions represented approximately 20 percent of the material covered on the courses final examination, providing
additional participant motivation to attend to the task.
Demographic information on the participants is provided in Table 1. Participants were
randomly assigned to one of the two treatment groups.2 The KMS group consisted of 87
participants, while the traditional group consisted of 101 participants; t-tests were conducted
and no significant differences were found for any of the demographic data.
Experimental KMS
WebCT, an Internet-based course portal, was used to implement the KMS in the experiment for several reasons. First, the WebCT interface is very similar to Lotus Notes
databases, which are applied as the software architecture in most KMS (OLeary 2002b).
Second, the participants were proficient with WebCT and had used it in the same course
to access materials and grades. Alavi and Leidner (2001b) note the importance of ease of
use in getting users to actually use a KMS. Third, WebCT provides a robust facility for
tracking student activity, enabling analysis of participant interaction. Finally, WebCT allowed the course administrator to allow and deny participant access, facilitating control
over the KMS.
The KMS used in the experimental sessions was restricted to the inclusion of explicit
knowledge components as this was the aspect of KMS of most interest in this study. Alavi
and Leidner (2001b) note that a KMS includes both explicit and tacit knowledge. However,
2
The experiment was administered as part of regular class time so participants in each class were randomly
assigned to report to either the regular classroom or to a computer laboratory. Due to seating limitations in the
computer laboratory, more participants were assigned to the traditional group, resulting in unequal sample sizes.
88
TABLE 1
Participant Information
Variable
KMS Group
Number of Participants
GPA
Age
Hours a Week Using a Computer
Majors
Accounting
MIS related
Class
Sophomore
Junior
Senior
Employed
Use a KMS at Work
Have used a KMS
Traditional Group
Combined
87
3.27
20.06
9.56
101
3.30
19.56
8.73
188
3.29
19.79
9.12
27.59%
4.60%
30.69%
6.93%
29.20%
5.85%
71.26%
24.14%
4.60%
47.13%
10.34%
14.94%
75.76%
21.22%
2.02%
49.50%
6.93%
11.88%
73.68%
23.12%
3.23%
45.70%
8.51%
13.30%
the explicit knowledge is first provided in the system as it is the most easily definable,
then the tacit knowledge which makes sense of the explicit knowledge is added by users
to create a shared-knowledge environment. Users lacking adequate prior encoded explicit
knowledge must access and make sense of explicit knowledge before they can effectively
use tacit knowledge (Alavi and Leidner 2001b). Further, prior studies of KMS implementations find that novice users application of explicit knowledge has largely been successful
(e.g., Alavi et al. 2006; Butler 2003; Gonzalez et al. 2005), but that attempts to capture
and effectively use tacit knowledge have often failed to gain traction (Alavi et al. 2006;
Butler 2003; Gonzalez et al. 2005). Both Butler (2003) and Alavi et al. (2006) found in
their studies of organizations that without strong interventions, users focused on the explicit
knowledge embedded in KMS and relied on routinized solutions to less complex problems.
The KMS in this study focused on provision of explicit knowledge and routinized
solutions by providing the ability to search through definitions, rules, examples, and templates. The KMS group used these facilities while solving the treatment cases in experimental session 2. While solving the cases, the participants used the KMS to search for
unknown terms or rules. In addition, the user had the ability to look at examples to facilitate
problem solving. The user also had access to templates that illustrated how to set up
the problem solution. The intent was to create a KMS environment that would contain the
types of materials that are commonly available to KMS users in a business environment.
The KMS organized definitions, rules, examples, and templates by both the knowledge
category (definitions, rules, examples, templates) and the decision type (special order, drop
a product or department, sell at split-off or process further decision associated with joint
products). The KMS main menu displayed icons and links to definitions, rules, examples,
templates, special order, drop a product, department, or division, and split-off versus process
further. This allowed the user to navigate to either a specific decision or to a specific type
of knowledge. In addition, the main menu included a search facility. The search facility
provided the user with all occurrences in the KMS of the search term entered by the user.
The user could select a link to any occurrence.
The navigation of the KMS was based on the category selected by the user, and operated as follows based on the selection:
89
Definition category took the user to a list of terms. These terms, when clicked, linked
to the respective definition.
Rules category took the user to a list of rules. These rules, when clicked, took the
user to the respective rule. The rules were categorized by decision type.
Templates category took the user to a list of templates, which were linked to the
respective template. The templates were categorized by decision type.
Examples category took the user to a list of examples categorized by decision type.
The examples were linked to the respective example.
Every term in the rules, templates, and examples was hyperlinked to the respective
definition in the KMS. Each template contained a link to the respective rule used to generate
the template and vice versa (each rule was linked to the respective template). Each example
contained links to the respective templates and rules. This interconnectivity allowed the
user to navigate easily from anywhere in the KMS. In addition, from any page within
the KMS, the user could select any of the main menu options from the navigation bar.
Traditional Reference Materials
Participants in the traditional reference materials treatment were allowed access to their
textbook, lecture notes, and handouts. These materials contained all of the same information
as the KMS, but the information was embedded within the materials and required typical
manual search techniques to find relevant explicit knowledge components. These materials
are representative of the types of materials available in pre-KMS environments where operations manuals, workpaper templates, audit guides and checklists, accounting standards
(e.g., GAAP) guides, and so forth are all available through printed books, three-ring binder
grouping, and/or .pdf documents with limited search capability. Accordingly, such materials
require a more effortful search strategy to identify relevant supporting knowledge components and increase the mental workload associated with pulling together problem solutions.
Measurement and Design
The independent variable was treatment group (KMS or traditional) for testing all
hypotheses and research questions. For H1, the dependent variable was the score attained
on Treatment Cases 1 through 3. For the definitions research question (RQ2), the dependent
variable was the posttest with pretest definition recall used as a covariate. For the rules
research question (RQ3), the dependent variable was posttest rule recall with pretest rule
recall included as a covariate. For the examples research question (RQ4), the dependent
variable was posttest example recall with pretest example recall included as a covariate.
For H5, which posits that individuals using a KMS will improve interpretive problemsolving skills more than individuals using traditional reference materials, the dependent
variable was the difference in percentage score of Posttest Case 1 and Posttest Case 2.
Treatment Case 1, which constituted a measure of initial interpretive problem-solving skills,
was used as a covariate. For RQ6, the dependent variables were the score attained on
Posttest Cases 1 and 2.
Three covariates were used in the analysis of all six hypotheses/research questions
ability, performance goal orientation, and learning goal orientation. Research indicates ability is an antecedent to knowledge acquisition. Furthermore, ability and knowledge affect
performance (Libby 1995; Libby and Luft 1993). Ability is a critical prerequisite to knowledge acquisition and enhances performance; accordingly, it was included as a covariate in
all models as participants ability may affect recall and performance. Participants GPA was
used as a proxy for ability.
90
Participants orientation toward goals could affect the level of recall and level of performance. Anderson (2000) refers to this as attention to learning. If an individual is not
oriented toward learning, they are less likely to encode knowledge even when encountered
during problem resolution. Smedley and Sutton (2007) differentiate intentional learners
from other participants in their study of procedural knowledge acquisition from use of a
KBS. They found that intentional learners showed significantly greater knowledge acquisition and responded better to system explanations that presented knowledge components. In this study, we control for such effects by monitoring participants goal orientation
across two potentially conflicting goalsperformance goal orientation and learning goal
orientation. Performance goal orientation leads an individual to strive for high performance
or avoid low performance, while learning goal orientation leads an individual to understand
something new or increase level of performance in a given activity (Button et al. 1996).
Participants exhibiting performance goal orientation are likely to attempt to achieve high
performance and outperform participants not exhibiting performance goal orientation. Participants exhibiting learning goal orientation strive to acquire knowledge and may acquire
more knowledge than individuals not exhibiting learning goal orientation. Either type of
goal orientation could affect performance and knowledge acquisition and is accordingly
controlled in our analyses of participants task-related responses. Both types of goal orientation were included as covariates in all models and were measured using the work
preference inventory scale developed and validated by Button et al. (1996).
IV. RESULTS
Data collected during the three experimental sessions provide the basis for testing the
six hypotheses/research questions. Table 2 provides the mean results of the performance
for pretest recall, posttest recall, treatment cases, and posttest cases. The data indicate a
notable difference in performance on the treatment cases between the two groups; the KMS
TABLE 2
Mean Performance Results
Pretest Scoresa
Definition Recall
Rule Recall
Example Recall
Posttest Scores
Definition Recall
Rule Recall
Example Recall
Treatment Casesb
Case 1
Case 2
Case 3
Posttest Cases
Case 1
Case 2
a
b
KMS Group
Mean (Std. Dev.)
Traditional Group
Mean (Std. Dev.)
Overall
Mean (Std. Dev.)
7.37 (1.78)
5.33 (2.34)
5.86 (1.99)
7.68 (1.43)
5.97 (2.21)
6.15 (2.38)
7.54 (1.61)
5.68 (2.29)
6.02 (2.21)
8.24 (1.49)
6.31 (2.26)
5.20 (2.19)
8.40 (1.23)
7.45 (1.91)
5.73 (2.19)
8.32 (1.35)
6.92 (2.15)
5.48 (2.20)
87.4% (33.4%)
61.3% (36.0%)
78.7% (34.6%)
72.8% (44.5%)
47.5% (36.4%)
65.3% (42.9%)
79.5% (40.3%)
53.9% (36.8%)
71.5% (37.7%)
89.7% (29.7%)
85.1% (27.6%)
93.1% (24.5%)
93.1% (20.0%)
91.5% (27.0%)
89.4% (24.1%)
Maximum score for each pretest and posttest recall is 10.

Treatment cases and posttest cases are measured on a scale of 0 to 100 percent.
91
group performed consistently better than the traditional group when the KMS was available,
but consistently poorer on the posttest cases when it was not available. (The table also
shows that the KMS group scored a greater difference between the pretest and posttest
definition recall (increase of .87) than the traditional group (increase of .72), meaning they
acquired more definitional knowledge than the traditional group. On the other hand, the
traditional group had a greater increase from the pretest to the posttest scores for rule recall
(increase of 1.48) than the KMS group (increase of .98) indicating that the traditional group
acquired more rule-type knowledge than the KMS group. Interestingly, example recall severely degraded for both groups as exhibited by decreases in scores from the pretest to the
posttest.
To determine whether participants acquired explicit knowledge as a result of the treatment, t-tests were performed on differences between the posttest and pretest recall scores
for definitions, rules, and examples for the KMS group, traditional group, and overall. As
indicated in Table 3, all t-tests are significant. A t-test of all participants indicates knowledge
acquisition occurred for definitions (p .001) and rules (p .001), but example recall
degraded (p .001). Separate analysis of the KMS group shows significantly improved
definition recall (p .001) and rule recall (p .001), but degraded example recall (p
.002); while separate analysis of the traditional group followed the same pattern with
significantly improved definition recall (p .001) and rule recall (p .001), and worse
example recall (p .020).
KMS Performance Effects
Hypothesis 1 focuses on the performance effects of using the experimental KMS to
solve problems, hypothesizing that users of a KMS will have a higher level of performance
than those who have access to traditional reference materials. Hypothesis 1 is first tested
using a MANCOVA with the scores on Treatment Cases 1, 2, and 3 as the dependent
variables, group as the independent variable, and GPA, performance goal orientation, and
learning goal orientation as the covariates. The MANCOVA results indicate that there was
a significant difference between groups (p .004). As noted previously, the mean values
reported in Table 2 show that on average the KMS group outperformed the traditional group
for all three treatment cases. Separate ANCOVAs are performed for each of the three
TABLE 3
Results of t-Tests of Recall Differences
Group
Overall
Definition Recall
Rule Recall
Example Recall
KMS
Definition Recall
Rule Recall
Example Recall
Traditional
Definition Recall
Rule Recall
Example Recall
Pretest Score
Posttest Score
t-Statistic
Significance
Level
7.54
5.68
6.02
8.32
6.92
5.48
7.106
8.880
3.903
0.000
0.000
0.000
7.37
5.33
5.86
8.24
6.31
5.20
4.902
4.625
3.140
0.000
0.000
0.002
7.68
5.97
6.15
8.40
7.45
5.73
5.161
7.980
2.364
0.000
0.000
0.020
92
treatment cases in order to isolate the specific sources of effects captured within the
MANCOVA analysis. As shown in Table 4, each of the treatment cases results in a significant difference in performance between the KMS and traditional groups (Treatment Case
1, p .012, Treatment Case 2, p .003, and Treatment Case 3, Panel C, p .014). The
results consistently support the expected relationship as tested through H1.
Encoding of Explicit Knowledge
The purpose of the research questions related to encoding of explicit knowledge is to
assess the participants ability to recall explicit knowledge components. The explicit knowledge research questions, RQ2, RQ3, and RQ4 measure the participants recall of definitions,
TABLE 4
Results for Performance during Treatment Cases (H1)
Panel A: MANCOVA Results
Source of Variance
F-value (Wilks criterion)
p-value
4.674
.004
8.452
0.309
1.731
.000
.819
.162
Independent Variable:
Group
Covariates:
GPA
Performance Goal Orientation
Learning Goal Orientation
Panel B: ANCOVA Results
Source of Variance
Treatment Case 1
Model
Group
Covariates:
GPA
Error
Treatment Case 2
Model
Group
Covariates:
GPA
Error
Treatment Case 3
Model
Group
Covariates:
GPA
Error
Type III SS
df
F-value
p-value
2.034
3.285
.013
1.002
6.474
.012
0.169
0.016
0.767
28.332
1
1
1
183
1.092
.101
4.954
.297
.751
.027
3.640
7.739
.000
1.072
9.118
.003
2.659
0.067
0.003
21.518
1
1
1
183
22.612
0.573
0.027
.000
.450
.870
2.151
3.595
.008
0.929
6.208
.014
1.292
0.000
0.008
27.374
1
1
1
183
8.637
0.002
0.055
.004
.969
.815
93
rules, and examples, respectively. The test of RQ2 examines whether individuals using a
KMS embedded with explicit knowledge components exhibit a difference in definition recall
than individuals accessing traditional reference materials. To test for the significance of the
difference in definition recall between the KMS and traditional groups, an ANCOVA is
used with posttest definition recall as the dependent variable, group as the independent
variable, and pretest definition recall, GPA, learning goal orientation, and performance goal
orientation as the covariates. Pretest definition recall is used as a covariate to control for
the knowledge that participants had before completing the treatment.3 Even though the
average improvement in definition recall was better for the KMS group than the traditional
group (.87 compared to .73), the results of the ANCOVA shown in Table 5, Panel A, do
not show a significant difference in definition recall between the two groups (p .942).
The test of RQ3 examines whether individuals using a KMS embedded with explicit
knowledge components exhibit a difference in rule recall than individuals accessing traditional reference materials. To test for differences in rule recall between the two groups, an
ANCOVA is used with the posttest rule recall as the dependent variable, group as the
independent variable, and pretest rule recall, GPA, performance goal orientation, and learning goal orientation as covariates. The traditional group had greater improvement in rule
recall than the KMS group (1.475 compared to .98), which is significant (p .002) supporting RQ3 (see Table 5, Panel B).
The test of RQ4 examines whether individuals using a KMS embedded with explicit
knowledge components will exhibit a difference in example recall than individuals accessing
traditional reference materials. To test for differences in example recall, an ANCOVA was
used with posttest example recall as the dependent variable, group as the independent
variable, and pretest example recall, GPA, performance goal orientation, and learning goal
orientation as covariates. While the ability to recall examples actually declined in both
groups, the KMS group had a greater mean decrease in example recall than the traditional group (.67 versus .42), the differences between the groups are not statistically
significant (p .134), as shown in Table 5, Panel C. Thus, RQ4 is not supported.
Interpretive Problem Solving
Hypothesis 5 focuses on the interpretive problem-solving aspect of explicit knowledge
acquisition, hypothesizing that users of a KMS will improve their interpretive problemsolving skills substantially more than individuals who do not have access to a KMS. To
isolate participants utilizing interpretive problem solving skills, only those with a Posttest
Case 1 score greater than or equal to Posttest Case 2 score were included in the analysis.
Posttest Case 1 was identical to Treatment Case 1, a case embedded in the KMS, and a
case in the class notes. According to Anderson et al. (1997), If participants were performing better for those original examples, then it would be evidence that they were solving
these problems by means of retrieving the study example and utilizing interpretive problem
solving (Anderson et al. 1997, 934935). Better performance on Posttest Case 2 indicates
that participants had developed tacit knowledge and were not using explicit knowledge. As
a result, 15 participants that performed better on Posttest Case 2 were removed from the
sample to examine H5, which is specifically focused on a type of explicit knowledge
acquisition.
3
Using posttest score as the dependent variable and pretest score as the covariate is a better measure of change
than the difference score between posttest and pretest. Using the difference ignores the relative value of the
change that occurred as a result of the treatment.
94
TABLE 5
Results of ANCOVA for Recall (RQ24)
Source of Variance
Type III SS
df
F-value
p-value
Panel A: Dependent Variable: Posttest Definition Recall

Model
Group
Covariates:
Pretest Definition Recall
GPA
Error
92.590
13.448
.000
.007
0.005
.942
71.218
6.752
4.984
.242
250.618
1
1
1
1
182
51.719
4.903
3.619
.176
.000
.028
.059
.675
375.419
28.096
.000
25.930
9.703
.002
251.989
1.661
5.014
4.448
486.384
1
1
1
1
182
94.292
.621
1.876
1.665
.000
.432
.172
.199
382.368
26.532
.000
6.544
2.270
.134
364.319
3.307
.045
.523
524.584
1
1
1
1
182
126.397
1.147
.016
.181
.000
.286
.901
.671
Panel B: Dependent Variable: Posttest Rule Recall

Model
Group
Covariates:
Pretest Rule Recall
GPA
Error
Panel C: Dependent Variable: Posttest Example Recall

Model
Group
Covariates:
Pretest Example Recall
GPA
Error
Hypothesis 5 is tested by examining the difference in interpretive problem solving

between the KMS and traditional groups when completing Posttest Case 1. The mean
performance increase for the KMS group was .05 and for the traditional group was .00,
indicating that on average the KMS group demonstrated greater interpretive problemsolving ability than the traditional group. An ANCOVA was performed with interpretive
problem solving (Posttest Case 1 score less Posttest Case 2 score) as the dependent variable,
group as the independent variable, and GPA, performance goal orientation, learning goal
orientation, and Treatment Case 1 as covariates. Treatment Case 1 is a covariate in order
to isolate the improvement in the participants interpretive problem-solving skills. Since
Treatment Case 1 was identical to Posttest Case 1, Treatment Case 1 functions as the initial
level of interpretive problem-solving skills. As shown in Table 6, a significant difference
in improvement of interpretive problem-solving skills is found between the groups (p
.048), providing evidence supporting the hypothesized relationship (H5).
95
TABLE 6
Results of ANCOVA for Interpretive Problem Solving (H5)Dependent Variable:
Interpretive Problem Solving Score
Source of Variance
Model
Group
Covariates:
GPA
Treatment Case 1 Score
Error
Type III SS
df
F-value
p-value
.674
2.870
.016
0.186
3.964
.048
0.098
0.081
0.014
0.311
7.846
1
1
1
1
167
2.094
1.729
0.289
6.615
.150
.190
.591
.011
Knowledge Acquisition from a KMS, Performing Without It

Research Question 6 focuses on concerns over the potential impact on future performance capabilities of individuals not having access to a KMS after solving problems only
through access and use of a KMS. Acquisition of explicit knowledge is a critical step in
the development of expertise and the impact of KMS usage on individuals acquisition of
explicit knowledge and problem-solving skill development within a critical decision domain
has major implications to KMS designers and implementers. As noted earlier, the mean
performance values reported in Table 2 show that on average the traditional group outperformed the KMS group on both posttest cases.
A MANCOVA was first used to test for the overall differences between the two groups.
The MANCOVA included Posttest Case 1 and Posttest Case 2 scores as the dependent
variables, group as the independent variable, and GPA, performance goal orientation, and
learning goal orientation as the two covariates. The MANCOVA results indicates that group
is marginally significant (p .062) indicating that the traditional group outperformed the
KMS group.
The results from the posttest cases were further examined using separate ANCOVAs.
As shown in Table 7, the difference between the two groups was not significant for Posttest
Case 1 (p .373), but was significant for Posttest Case 2 (p .022) thus the overall results
from the MANCOVA was driven by the differences in Posttest Case 2. In aggregate the
results provide partial support for RQ6, indicating that use of the KMS may have a deleterious effect on the development of problem-solving skills in absentia of a KMS. The lack
of significance in the Posttest Case 1 analysis may also be influenced by the dependence
on interpretive problem solving which the KMS group showed a stronger ability to apply.
V. DISCUSSION AND CONCLUSION
Anecdotal information has suggested that KMS can effectively improve decision making by relatively novice users. Additionally, KMS may impact the knowledge acquisition
of the useralthough there are conflicting theoretical views on whether this is likely to be
a positive or a negative impact. Given the widespread adoption of KMS in professional
environments, research examining the impacts of KMS adoption on user performance and
expertise development is imperative to fully understand the consequences of KMS use. The
study reported in this paper has provided initial evidence on the impact of a KMS on user
performance and acquisition of explicit knowledge.
96
TABLE 7
Analysis of Posttest Cases (RQ6)
Panel A: MANCOVA Results
Source of Variance
F-value (Wilks
criterion)
p-value
2.827
.062
5.999
2.282
2.583
.003
.105
.078
Group
Covariates:
GPA
Panel B: ANCOVA Results
Source of Variance
Posttest Case 1
Model
Group
Covariates:
GPA
Error
Posttest Case 2
Model
Group
Covariates:
GPA
Error
Type III SS
df
F-value
p-value
1.220
4.494
.002
.054
.796
.373
.763
.271
.251
12.418
1
1
1
183
11.250
3.997
3.697
.001
.047
.056
.543
2.406
.051
.303
5.364
.022
.106
.062
.049
10.329
1
1
1
183
1.885
1.098
.877
.171
.296
.350
The results indicate that there are both positive and negative consequences associated
with the use of KMS by novice decision makers. From a performance standpoint, KMS
users were found to perform better than users of traditional reference materials in solving
structured problems. However, the users of traditional reference materials were found to be
more proficient at the task when supporting materials were subsequently not available. From
a knowledge-acquisition standpoint, the implications are less clear. As expected, the ready
availability of examples did significantly improve KMS users interpretive problem-solving
skills in comparison to the level of skill acquired from the use of traditional reference
materials. However, the impact of KMS use on the other component of explicit knowledge
acquisition, encoding of explicit knowledge, was less clear. Use of traditional reference
materials yielded significantly greater encoding of decision rules, but no significant differences in encoding of knowledge related to definitions. The encoding of knowledge related
to examples was most concerning in that both users of a KMS and users of traditional
reference materials actually experienced decreases in encoding.
The increases in KMS users performance (i.e., when the KMS is available) helps
validate the anecdotal data that has been reported in the business press and the speculation
97
of academics working in the research domain. The poorer performance when the KMS is
not available, however, provides evidence supporting the de-skilling effects that have been
theorized from a technology dominance standpoint (Arnold and Sutton 1998; OLeary
2002a). These combined results present a dilemma to professional organizations using
KMS. In the short term, the use of a KMS appears beneficial, but in the long term, there
are concerns about the development of domain expertise by KMS users.
The results from a knowledge-acquisition perspective raise warnings that will necessitate additional research to better understand the overall impact of KMS use. First, the
results indicate that KMS users show a greater improvement in interpretive problem-solving
skills. KMS users should continue to enhance their interpretive problem-solving skills as
they continue to use the KMSinterpretive problem solving is well supported by a KMS
and reduces the cognitive effort required to solve problems. However, Anderson (1993)
theorizes that when interpretive problem solving requires less cognitive effort, even users
that have the necessary procedural-level knowledge will regress to using interpretive problem solving for problem resolution. Projecting from this behavior, novices with easy access
to examples to support decision making may continue to use interpretive problem solving
for problem resolution and not necessarily be motivated to begin developing higher-level
tacit knowledge.
On the other hand, individuals using traditional reference materials exhibited better
encoding of rules. The individuals may be progressing to the procedural (tacit) stage of
knowledge acquisition and developing production rules. Anderson and Fincham (1994)
argue and find that procedural knowledge can be developed by solving as little as one
problem. It is quite plausible that the users of traditional reference materials may have
incurred greater cognitive effort during the treatment stage and began to develop tacit
knowledge while solving the treatment cases. As a result, the associated rule recall improved
by using developed production rules (a component of tacit knowledge). This phenomenon should be investigated further in future research examining the linkages between
explicit knowledge and tacit knowledge acquisition in KMS-supported decision-making
environments.
The degrading of knowledge encoding of examples also raises concerns. The most
likely driver of this effect is information overload. When the information load exceeds the
information-processing capacity, information overload occurs (Schick et al. 1990). Schroder
et al. (1967) describe the limitations of human information processing by modeling the
association between information load and information-processing complexity. The model
suggests that individuals encountering information overload will respond by increasing
information-processing complexity from low-level (concrete) to high-level (abstract). More
cues are identified, finer distinctions are drawn from them, and the cues are integrated in
an increasing number of methods. This process continues until the cognitive structure is so
inundated with integration of cues that the facilities formerly used to identify cues are
compensating for the integration; and hence, the identification of cues declines (Schroder
et al. 1967). This phenomenon may have caused the decline in example recall as participants
are provided with a multitude of examples.
As with any study, there are limitations to the research that should be considered when
making inferences about the results. There is a risk that some participants may have studied
the material outside of the experimental sessions. While the lecture notes were collected
at the end of each session and access to the KMS was deactivated, any of the participants
may have studied the textbook. Additionally, although the participants were instructed not
to talk to one another about the experiment, whether they did discuss the experiment with
one another outside of the experimental sessions is indeterminable. Hence, complete control
98
was not available, but there is no reason to believe that this would impact one treatment
group more than the other.
An additional limitation is use of student participants completing a fairly simple decision task. While use of students was desirable for control purposes in this study with the
focus on acquisition of explicit knowledge, students may not be completely representative
of professional behavior. On the other hand, it would be exceedingly difficult to obtain
professional participants lacking baseline explicit knowledge needed to perform their jobs.
Future research should move beyond explicit knowledge to investigate the impact of KMS
on the development of tacit knowledge, as this study provides initial evidence that tacit
knowledge acquisition may be hindered when individuals use KMS. In addition, research
should investigate long-term effects of KMS such as de-skilling and reduction of firm
expertise as suggested by prior research (Arnold and Sutton 1998; OLeary 2002a; Mascha
and Smedley 2007; Dowling and Moroney 2008).
While the research reported in this study examines whether the use of a KMS impacts
explicit knowledge acquisition, future research should examine why that difference occurs.
For instance, by utilizing an enhanced design such as eye-tracking software that compares
handouts on screen to the KMS using eye-tracking software, we could gain insight into
how the use of a KMS impacts knowledge acquisition.
REFERENCES
Alavi, M. 1997. KPMG Peat Marwick U.S.: One Giant Brain. (Case 9-379-108) Boston, MA: Harvard
Business Publishing.
, and D. L. Leidner. 1999. KMS: Issues, challenges, and benefits. Communications of the
Association for Information Systems (February): 137.
, and . 2001a. Research commentary: Technology-mediated learningA call for greater
depth and breadth of research. Information Systems Research 12 (1): 110.
, and . 2001b. Review: Knowledge management and knowledge management systems:
Conceptual foundations and research issues. MIS Quarterly 25 (1): 107137.
, T. R. Kayworth, and D. L. Leidner. 2006. An empirical examination of the influence of
organizational culture on knowledge management practices. Journal of Management Information Systems 22 (3): 191224.
Anderson, J. R. 1987. Skill acquisition: Compilation of weak-method problem solutions. Psychological Review 94 (2): 192210.
. 1990. The Adaptive Character of Thought. Hillsdale, NJ: Lawrence Erlbaum Associates.
. 1993. Rules of the Mind. Hillsdale, NJ: Erlbaum.
, and J. M. Fincham. 1994. Acquisition of procedural skills from examples. Journal of Experimental Psychology: Learning, Memory, and Cognition 20 (6): 13221340.
, , and S. Douglass. 1997. The role of examples and rules in the acquisition of a
cognitive skill. Journal of Experimental Psychology: Learning, Memory, and Cognition 23 (4):
932945.
. 2000. Learning and Memory. New York, NY: John Wiley and Sons, Inc.
Arnold, V., and S. G. Sutton. 1998. The theory of technology dominance: Understanding the impact
of intelligent decision aids on decision makers judgments. Advances in Accounting Behavioral
Research 1: 175194.
, N. Clark, P. A. Collier, S. A. Leech, and S. G. Sutton. 2006. The differential use and effect
of knowledge-based system explanations in novice and expert judgment decisions. MIS Quarterly 30 (1): 7997.
Baird, L., J. C. Henderson, and S. Watts. 1997. Learning from action: An analysis of the center for
army lessons learned. Human Resource Management 36 (4): 385396.
99
Bartlett, C. 1996. McKinsey & Company: Managing Knowledge and Learning. (Case 9-396-357)
Boston, MA: Harvard Business Publishing.
Bransford, J. D., B. S. Stein, N. J. Vye, J. J. Franks, P. M Auble, K. J. Mezynski, and G. A. Perfetto.
1982. Different approaches to learning: An overview. Journal of Experimental Psychology:
General (4): 390398.
Brody, R. G., T. K. Kowalczyk, and J. M. Coulter. 2003. The effect of a computerized decision aid
on the development of knowledge. Journal of Business and Psychology (Winter): 157174.
Butler, T. 2003. From data to knowledge and back again: Understanding the limitations of KMS.
Knowledge and Process Management 10 (3): 144155.
Button, S. B., J. E. Mathieu, and D. M. Zajac. 1996. Learning goal and performance goal orientation.
Organizational Behavior and Human Decision Processes (July): 2648.
Cameron, P. 2000. Managing the wealth. CMA Management 74 (9): 4649.
Chandler, P., and J. Sweller. 1991. Cognitive load theory and the format of instruction. Cognition and
Instruction 8 (4): 293332.
, and . 1996. Cognitive load while learning to use a computer program. Applied Cognitive Psychology 10 (2): 151170.
Chi, M. T. H., M. Bassok, M. W. Lewis, P. Reimann, and R. Glaser. 1989. Self-explanations: How
students study and use examples in learning to solve problems. Cognitive Science 13 (2): 145
182.
Cole, R. E. 1998. Introduction. California Management Review 40 (3): 1521.
Davenport, T. H., D. W. DeLong, and M. C. Beers. 1998. Successful knowledge management projects.
Sloan Management Review 39 (2): 4357.
Dilnutt, R. 2002. Knowledge management in practice: Three contemporary case studies. International
Journal of Accounting Information Systems 3 (2): 7582.
Dowling, C., and R. Moroney. 2008. (Forthcoming.) Auditor performance variation: Impact of subspecialty knowledge differences between industry specialists. Advances in Accounting Behavioral Research 11.
Duchessi, P., and R. M. OKeefe. 1992. Contrasting successful and unsuccessful expert systems.
European Journal of Operations Research 61: 122134.
Earl, M. 2001. Knowledge management strategies: Toward a taxonomy. Journal of Management
Information Systems 18 (1): 215234.
Eining, M., and P. Dorr. 1991. The impact of expert system usage on experiential learning in an
auditing setting. Journal of Information Systems 5 (1): 116.
Franks, J. J., N. J. Vye, P. M. Auble, K. J. Mezynski, G. A. Perfetto, J. D. Bransford, B. S. Stein,
and J. Littlefield. 1982. Learning from explicit versus implicit texts. Journal of Experimental
Psychology: General 4: 414422.
Gonzalez, L. M., R. E. Giachetti, and G. Ramirez. 2005. Knowledge management-centric help desk:
Specification and performance evaluation. Decision Support Systems 40: 389405.
Grant, R. M. 1996. Prospering in dynamically competitive environments: Organizational capability as
knowledge integration. Organization Science 7 (4): 375387.
Grover, V., and T. H. Davenport. 2001. General perspectives on knowledge management: Fostering a
research agenda. Journal of Management Information Systems 18 (1): 523.
Henderson, J. C., S. W. Sussman, and J. Thomas. 1998. Creating and exploiting knowledge for fastcycle organizational response. Advances in Applied Business Strategy 5: 103128.
Herz, P. J., and J. J. Schultz. 1999. The role of procedural and declarative knowledge in performing
accounting tasks. Behavioral Research in Accounting 11: 127.
Hiltz, S. R. 1986. The virtual classroom: Using computer-mediated communication for university
teaching. Journal of Communication 36 (2): 95104.
Leech, S. A., and S. G. Sutton. 2002. Knowledge management issues in practice: Opportunities for
research. International Journal of Accounting Information Systems 3 (2): 6974.
Libby, R., and J. Luft. 1993. Determinants of judgment performance in accounting settings: Ability,
knowledge, motivation, and environment. Accounting, Organizations and Society 18 (5): 425
450.
100
. 1995. The role of knowledge and memory in audit judgment. In Judgment and Decision
Making Research in Accounting and Auditing, edited by R. H. Ashton, and H. Ashton, 176
206. Cambridge, U.K.: Cambridge University Press.
Marchant, G., J. Robinson, U. Anderson, and M. Schadewald. 1991. Analogical transfer and expertise
in legal reasoning. Organizational Behavior and Human Decision Processes 480 (2): 272290.
Mascha, M. F., and G. Smedley. 2007. Can computerized decision aids do damage? A case for
tailoring feedback and task complexity based on task experience. International Journal of Accounting Information Systems 8 (2): 7391.
Murphy, D. S. 1990. Expert system use and the development of expertise in auditing: A preliminary
investigation. Journal of Information Systems 4 (3): 1835.
Nagle, C. 1999. Research opportunities in knowledge management. Presented at the Mid-Year Meeting
of the Auditing Section of the American Accounting Association (January 14).
Nonaka, I. 1994. A dynamic theory of organizational knowledge creation. Organization Science 5
(1): 1437.
Odom, M. D., and P. B. Dorr. 1995. The impact of elaboration-based expert system interfaces on deskilling: An epistemological issue. Journal of Information Systems 9 (1): 118.
. 1998. Enterprise knowledge management. Computer (March): 5461.
. 2002a. Knowledge management in accounting and professional Services. In Researching
Accounting as an Information Systems Discipline, edited by V. Arnold, and S. G. Sutton, 273
283. Sarasota, FL: American Accounting Association.
. 2002b. Knowledge management across the enterprise resource planning systems life cycle.
International Journal of Accounting Information Systems 3 (2): 99110.
Paas, F., A. Renkl, and J. Sweller. 2003. Cognitive load theory and instructional design: Recent
developments. Educational Psychologist 38 (1): 14.
Powell, W. 1998. Learning from collaboration: Knowledge and networks in the biotechnology and
pharmaceutical industries. California Management Review 40 (3): 228240.
Roberts, M. L., and R. H. Ashton. 2003. Using declarative knowledge to improve information search
performance. Journal of the American Taxation Association 25 (1): 2138.
Rose, J. M., and C. J. Wolfe. 2000. The effects of system design alternatives on the acquisition of
tax knowledge from a computerized tax decision aid. Accounting, Organizations and Society
25: 285306.
. 2005. Decision aids and experiential learning. Behavioral Research in Accounting 17: 175
189.
Sambamurthy, V., and M. Subramani. 2005. Special issue on information technologies and knowledge
management. MIS Quarterly 29 (1): 17.
Schick, A. G., L. A. Gordon, and S. Haka. 1990. Information overload: A temporal approach.
Accounting, Organizations and Society 15 (3): 199220.
Schoenebeck, K. 2003. Test Item File: Cost Accounting: A Managerial Emphasis, 11th Edition. Upper
Saddle River, NJ: Pearson Education.
Schroder, H. M., M. J. Driver, and S. Streufert. 1967. Human Information Processing. New York,
NY: Holt, Rinehart & Winston.
Smedley, G. A., and S. G. Sutton. 2004. Explanation provision in knowledge-based systems: A theorydriven approach for knowledge transfer designs. Journal of Emerging Technologies in Accounting 1: 4161.
, and . 2007. The effect of alternative procedural explanation types on procedural
knowledge acquisition during knowledge-based systems use. Journal of Information Systems 21
(1): 2751.
Stein, B. S., J. D. Bransford, J. J. Franks, R. A. Owings, N. J. Vye, and W. McGraw. 1982a. Differences in the precision of self-generated elaborations. Journal of Experimental Psychology: General 4: 399405.
, , , N. J. Vye, and G. A. Perfetto. 1982b. Differences in judgments of learning
difficulty. Journal of Experimental Psychology: General 4: 406413.
101
Sweller, J. 1988. Cognitive load during problem solving: Effects on learning. Cognitive Science 12:
257285.
. 1989. Cognitive technology: Some procedures for facilitating learning and problem solving
in mathematics and science. Journal of Educational Psychology 81 (4): 457466.
, and P. Chandler. 1991. Evidence for cognitive load theory. Cognition and Instruction 8: 351
362.
, J. van Merrienboer, and F. Paas. 1998. Cognitive architecture and instructional design. Educational Psychology Review 10 (3): 251296.
Thomas, J. B., S. A. Watts, and J. C. Henderson. 2001. Understanding strategic learning: Linking
organizational learning, knowledge management, and sensemaking. Organizational Science 12
(3): 331345.
van Merrienboer, J., and J. Sweller. 2005. Cognitive load theory and complex learning: Recent developments and future directions. Educational Psychology Review 17 (2): 147177.
Wickramasinghe, N., and G. L. Mills. 2002. Integrating e-commerce and knowledge management
What does the Kaiser experience really tell us. International Journal of Accounting Information
Systems 3 (2): 8398.

Use of Knowledge Management Systems and The Impact On The Acquisition of Explicit Knowledge

Загружено:

Сведения о документе

Исходное описание:

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Use of Knowledge Management Systems and The Impact On The Acquisition of Explicit Knowledge

Загружено:

Авторское право:

Доступные форматы

JOURNAL OF INFORMATION SYSTEMS

Vol. 22, No. 2

Use of Knowledge Management Systems

McCall, Arnold, and Sutton

Journal of Information Systems, Fall 2008

Use of Knowledge Management Systems

Journal of Information Systems, Fall 2008

McCall, Arnold, and Sutton

capability (Nonaka 1994). In an organizational setting, knowledge is an asset that enables

Journal of Information Systems, Fall 2008

Use of Knowledge Management Systems

Journal of Information Systems, Fall 2008

McCall, Arnold, and Sutton

Use of Knowledge Management Systems

experimental studies have explanations automatically provided at relevant points in the

McCall, Arnold, and Sutton

Use of Knowledge Management Systems

McCall, Arnold, and Sutton

Journal of Information Systems, Fall 2008

Use of Knowledge Management Systems

Participants and Incentives

Journal of Information Systems, Fall 2008

McCall, Arnold, and Sutton

Use of Knowledge Management Systems

McCall, Arnold, and Sutton

Maximum score for each pretest and posttest recall is 10.

Journal of Information Systems, Fall 2008

Use of Knowledge Management Systems

Journal of Information Systems, Fall 2008

McCall, Arnold, and Sutton

F-value (Wilks criterion)

Use of Knowledge Management Systems

Journal of Information Systems, Fall 2008

McCall, Arnold, and Sutton

Panel A: Dependent Variable: Posttest Definition Recall

Panel B: Dependent Variable: Posttest Rule Recall

Panel C: Dependent Variable: Posttest Example Recall

Hypothesis 5 is tested by examining the difference in interpretive problem solving

Use of Knowledge Management Systems

Knowledge Acquisition from a KMS, Performing Without It

McCall, Arnold, and Sutton

Use of Knowledge Management Systems

McCall, Arnold, and Sutton

Use of Knowledge Management Systems

McCall, Arnold, and Sutton

Journal of Information Systems, Fall 2008

Use of Knowledge Management Systems

Journal of Information Systems, Fall 2008

Вам также может понравиться