Montazemi, A. R. (2006). The Effect of Video Presentation in a CBT Environment.

Educational Technology & Society, 9 (4),

123-138.

The Effect of Video Presentation in a CBT Environment

Ali Reza Montazemi
DeGroote School of Business, McMaster University, 1280 Main St. West, Hamilton, ON L8S 4M4, Canada
Tel: +1 905 529 7070 Ext. 27434
Fax: +1 905 521 8995
montazem@mcmaster.ca
http://www.business.mcmaster.ca/msis/profs/montaz

ABSTRACT
Multimedia is increasingly used to enhance human-computer interaction in a variety of formats (text,
graphics, audio, animation, and video). The challenge is how to use the most effective presentation format
that would result in performance improvement. This paper addresses the added value of video
presentations in a technology-mediated learning environment for a basic Management Information System
course at MBA level. We postulate two hypotheses that take into account students’ mastery learning and
satisfaction with regard to the usefulness of video presentation, moderated by students’ self-interest to learn
the subject matter (i.e., intrinsic motivation). Our analysis, on the basis of empirical research, shows that
the subjects find a learning environment supported with video presentations is more useful. Their intrinsic
motivation to learn the subject matter had a significantly positive effect on their satisfaction with the
availability of video presentations. Video presentations had no significant effect on their performance on
exams, however.

Keywords
Video presentation, Multimedia, CBT

Introduction
The most significant applications of multimedia are in (1) education and training, (2) communications,
coordination, and collaboration, and (3) entertainment. With the growth of the new market for multimedia, end
users, application developers, and content providers face the challenge of how to effectively manage, organize,
and access the vast amount of information they accumulate. The focus of this paper is on the effect of video
presentation for education and training in the context of technology-enabled learning media. There is an
increasing expectation that information technology will enhance the learning process (Benbunan-Fich, 2002).
This expectation is based on the notion that information technology, by supporting interactive instruction, will
encourage students to be more responsible for their own learning. In this endeavor, the design of information
technology should be based on well-grounded theories of learning processes (Alavi and Leidner, 2001; Hannafin
and Rieber, 1989; Leidner and Jarvenpaa, 1995). For example, Hannafin and Rieber (1989) caution, “The
emergence of computer-driven ‘hybrid’ technologies … has spawned unprecedented interest, yet advances in
technological capability alone no more improve instruction than sharpened pencils improve prose.” More than a
decade later, a research essay by Alavi and Leidner (2001) calls for greater depth and breadth of investigation
into technology-mediated learning. It seems that in spite of the large number of reported research findings, we
know little about the effective means of applying and managing information technology in support of the
learning process. This is mainly due to the intricacy of the task at hand, for the dynamic nature of the interaction
between the environmental factors (i.e., instructional strategy and information technology) that affect the
learning process is complex (Alavi and Leidner, 2001; Carroll, 1997; Leidner and Jarvenpaa, 1995).

For example, let us consider the usefulness of dynamic visuals such as video presentations when used in support
of learning. Video presentation is used routinely in support of classroom lectures to enhance the learning
process. Inclusion of video in support of learning is expensive, and this means that we must make sure that it
benefits the learner. This paper presents the findings from an empirical investigation into the effectiveness of
video in a computer-based tutoring environment.

Computer-based tutoring (CBT) systems have been used in diverse learning environments to enable the learner
to self-pace in acquiring the pertinent materials. In such systems, feedback is a critical part of effective learning,
and it is expected that active involvement will lead to more effective learning than will passive involvement.
CBT systems are generally based on the stimulus-response-feedback views of learning that are associated with
the objectivist model of learning (Leidner and Jarvenpaa, 1995; Montazemi and Wang, 1995-a; Retalis and
Papasalours, 2005; Stone, 2001). Thus, CBT can be viewed as enhancing the cognitive information processing
of learners by tailoring the learning process to individual needs (Leidner and Jarvenpaa, 1995; Montazemi and
Wang, 1995-b). The learner is guided by the CBT system so that he or she reaches the predefined learning

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
123
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.
materials built into the system. Therefore, CBT systems can be developed for learning environments with a
well-defined structure, pre-defined learning objectives and possible solutions to the learner’s enquiries.
Multimedia technology can enrich the learning experience by making it possible to present the learning materials
in different formats. The question raised in this paper is whether inclusion of video presentations in a CBT
system is useful to the learner. To this end, the following two hypotheses are postulated and empirically tested:

HYPOTHESIS 1: Inclusion of video in a CBT system has no effect on the learner's perception regarding the
usefulness of the system.

HYPOTHESIS 2: Inclusion of video in a CBT system has no effect on the learner's mastery of the material to be
learned.

The remainder of paper is structured as follows: Section Two provides a review of research findings related to
the usefulness of dynamic visuals to the learning process. In Section Three, the basic theories on multimedia
learning and human perception are used to postulate above hypotheses in regard to the usefulness and
effectiveness of dynamic visuals in support of the learning process. A CBT system called E-Tutor is used to test
these hypotheses. Section Four outlines the methodology adopted in this investigation. Section Five presents
the results. A discussion and an overview of the implication for practice and future research close the paper.

On the Usefulness of Dynamic Visuals on Learning

Effective presentation of instructional materials, whether it be in the form of text, picture, charts, graphs,
diagrams, animation, videos, etc. is an important facet of the learning process. As a result, multimedia
educational systems are rapidly growing in popularity because they enable us to provide a variety of presentation
modes in support of the learning process (Anderson et al., 2000; Heller et al., 2001; Pilkington and Grierson,
1996). In particular, interactive multimedia systems with dynamic visual presentations are becoming popular
because they can provide an enriched learning environment by facilitating the acquisition of pertinent materials.
Dynamic visuals that have been embedded in CBT for improved learning can be classified as movies and
simulations. We can distinguish movies and simulations in the following way: animations, videos, time-lapse
photography and other motion pictures that are the same each time they are viewed are classified as movies; and
graphics, tables, or other output generated by running a computer program under student control are classified as
simulations (Pane, 1994). In the latter, the visual results are subject to any changes that the student makes to the
simulation program, and thus may be different each time they are viewed.

A number of investigators have examined the effect of dynamic visuals on learning (Pane et al., 1996; Park and
Hopkins, 1993; Rieber, 1990; Velayo, 2000). Park and Hopkins (1993) identified five important instructional
roles of dynamic visuals:
1. As an attention guide, the dynamic visual can serve to guide and direct the subject's attention.
2. As an aid for illustration, dynamic visuals can be used as an effective aid to represent the structural and
functional relations among components in a domain of knowledge.
3. As a representation of domain knowledge, movement and action can be used to effectively represent certain
domain knowledge.
4. As a device model for forming a mental image, dynamic visuals can be used to represent system structures
and functions that are not directly observable (e.g., blood flowing through the heart).
5. As a visual analogy or reasoning anchor for understanding abstract and symbolic concepts or processes,
dynamic visuals can make abstract and symbolic concepts (e.g., velocity) become more concrete and
directly observable.

The question arises whether dynamic visual has any effect on students' learning. To answer this, Park and
Hopkins (1993) produced a research summary of 25 studies and concluded that:
The research findings do not consistently support the superior effect of dynamic visual displays.
The conflicting findings seem to be related to the different theoretical rationales and
methodological approaches used in various studies (Park and Hopkins, 1993, p. 427).

Traditionally, designers of instructions have had two categories of outcomes in mind: those directed toward
cognitive goals, and those related to the attitude of the learner. Student achievement related to cognitive goals
has been the paramount objective of most instructional activities. However, it may also be important to recognize
the need for establishing attitudinal goals and for planning activities that are designed to facilitate the learning
process. Thus, one of the major goals of instruction involving media should be the development of positive

124
attitudes (Venkatesh, 1999; Simonson, 1985; Simonson and Maushak, 1995). Although the strength of the
relationship between attitudes and achievement is unclear (Zimbardo and Leippe, 1991), it seems logical that
students are more likely to remember information, seek new ideas, and continue studying when they react
favorably to an instructional situation or when they like a certain content area. Nonetheless, despite its
importance in the learning process, we know little about assessment of student perception on the usefulness of
dynamic visual media and their effect on learning. This vital research question is addressed in this paper.

The Objective of this Investigation

The basic question raised is "How do dynamic visuals in the form of videos embedded in a CBT system enhance
learning?" As mentioned above, previous investigations have contradictory findings concerning improvement in
students' learning outcomes (i.e., improved exam scores) when dynamic visuals were added to the CBT systems.
Missing from these studies is some consideration of the internal psychological processes through which learning
occurs (Alavi and Leidner, 2001). The contention of Alavi and Leidner in regard to technology-mediated
learning (TML) is that:
To be useful, TML research questions need to be formulated in terms of the way in which
technology features can engage psychological processes of learning that will in turn result in the
desired learning outcomes (Alavi and Leidner, 2001, p. 4).

Dynamic visuals can be embedded in CBT systems in the form of multimedia learning instructions. Multimedia
learning occurs when students use information presented in two or more formats – such as text, figures, and
videos. Figure 1 summarizes a dual-coding theory of multimedia learning (Mayer and Sims, 1994). It offers a
three-process account of how visually and verbally presented material might be integrated in the learner’s
working memory. On the top left portion of the figure, a verbal explanation, such as an oral narration, is
presented to the learner. Within the working memory the learner constructs a mental representation of the
system described in the verbal explanation. The cognitive process of going from an external to an internal
representation of the verbal material is called building a verbal representational connection (or verbal encoding)
(Mayer and Sims, 1994). On the bottom left portion of the figure, a visual explanation is presented to the
learner, such as video. Within the working memory the learner constructs a mental representation of the visually
presented system. The cognitive process of going from an external to an internal representation of visual
information is called building a visual representational connection (or visual encoding) and is indicated by the
second arrow. The third arrow denotes the construction of referential connections between the two mental
representations, that is, the mapping of structural relations between the two representations of the system. Long-
term memory enables the learner to interpret and assimilate the content of working memory to interact with the
decision environment. It is expected that the added value of multimedia enhance the decision performance of the
learner. For example, in understanding an explanation of how intranet and extranet work, the learner can build
referential connections between visual and verbal representation of essential parts, actions, relations and
principals in the communication system.

Figure 1: A dual coding model of multimedia learning (Mayer and Sims, (1994)

125
This theory of multimedia learning has been the basis of previous investigations on the effectiveness of dynamic
visuals in support of the learners’ improved learning performance. However, as indicated earlier, missing from
these investigations is internal psychological processes of the learner. Students must be motivated to use
strategies to regulate their cognition and effort (Pintrich, 1988 & 1989). Research findings suggest that students
with a motivational orientation involving goals of mastery learning, as well as a belief that the task is interesting
and important, will engage in more metacognitive activity, more cognitive strategy use and more effective effort
management (Ames and Archer, 1988; Meece et al., 1988; Nolen, 1988). There are two main classes of
motivation: extrinsic and intrinsic (Davis et al., 1992). Extrinsic motivation relates to the drive to perform a
behavior to achieve specific goals/rewards, while intrinsic motivation relates to perception of pleasure and
satisfaction from performing the behavior.

In this research, our measurement for CBT system's learning outcome is the learner’s performance and affective
reactions to the usefulness of videos. We use exam scores as a measure of the student’s extrinsic motivation to
achieve mastery learning of the pertinent learning materials. A student’s affective reaction to the usefulness of
videos is based on the Technology Acceptance Model (TAM) (Davis, 1989; Davis et al., 1989, 1992; Taylor and
Todd, 1995). In TAM, extrinsic motivation and the associated instrumentality are captured by the perceived
usefulness construct (Davis et al., 1992). In this research, affect of dynamic visuals on extrinsic motivation of
the learner and subsequent learning outcome is of interest. To this end, based on previous research findings, the
following two null-hypotheses are postulated:

HYPOTHESIS 1: Inclusion of video in a CBT system has no effect on the learner's perception regarding the
usefulness of the system.

HYPOTHESIS 2: Inclusion of video in a CBT system has no effect on the learner's mastery of the material to be
learned.

An effective assessment of these hypotheses requires us to take into account the effect of the learner’s intrinsic
motivation and self-regulated learning behavior, in form of covariates, which are important aspects of learning
and academic performance (Corno and Mandinach, 1983; Corno and Rohrkemper, 1985). The theoretical
framework adopted in this research for conceptualizing student intrinsic motivation is an adaptation of a general
expectancy-value model of motivation (Pintrich, 1988, 1989; Pintrich and DeGroot, 1990). The model
proposes that there are three motivational components that can be linked to the components of self-regulated
learning: (a) an expectancy component, which includes the student’s belief in his or her ability to perform a task
(i.e., self-efficacy), (b) a value component, which includes the student’s goals and beliefs about the importance
and interest of the task (i.e., intrinsic value), and (c) an affective component, which includes the student’s
emotional reactions to the task (i.e., test anxiety).

Self-regulated learning adopted in this research as covariate to assess the stated hypotheses includes the student’s
metacognitive strategies for planning, monitoring, and modifying cognition (Corno, 1986; Weinstein and Mayer,
1986; Zimmerman, 1988). Different cognitive strategies have been found to foster active cognitive engagement
in learning and to result in higher levels of achievement (Weinstein and Mayer, 1986). Self-regulation, on the
other hand, refers to the student’s ability to manage and control the handling classroom academic tasks. For
example, capable students who persist at a difficult task or block out distractions (i.e., noisy classmates) maintain
their cognitive engagement in the task, and so are able to perform better (Corno, 1986; Corno and Rohrkemper,
1985).

Methodology
A CBT system called E-Tutor, developed in support of a Management Information Systems (MIS) textbook was
adopted to test the stated hypotheses (see O’Brien, 2001). The structure of E-Tutor was similar to the model
proposed by Retalis and Papasalours (2005). E-Tutor enables each student to test his or her knowledge of the
lessons to be learned. Textbook materials are embedded in the E-tutor in the form of hypermedia that presents
text, pictures, figures and videos. We developed two versions of E-Tutor referred here as “without-video” and
“with-video”: both versions had the basic textbook materials (i.e., text, figures, tables and pictures). However,
one version had 80 videos in form of AVI files, called here E-Tutor (with video), and the other without videos,
called here E-Tutor (without video). The videos that were embedded in the E-Tutor (with video), provided
further definition or examples to support various parts of the text of the lessons being studied. These videos are
expected to add value to the instructor’s explanation of the materials. Thus, McGraw Hill has included them in
the instructor's CD ROM that accompanies their MIS textbooks. The question posed through our hypotheses is

126
whether such learning materials in the form of videos are useful to the learning process when embedded in a
CBT system.

Experimental Design
We used a 2x2 factorial design with one between-subjects factor and two repeated trials to assess the stated
hypotheses. The between-subject factor was the presence or absence of the AVI videos embedded in a CBT
system. Two dependent variables used were the usefulness of the video presentations in support of the learning
materials and students’ test score gained from two exams. Students’ intrinsic motivation to learn the subject
matter and self-regulated learning behavior, assessed through two structured questionnaires, were used as
covariates.

Subjects

Fifty-one students registered in a basic Management Information Systems course at the MBA level were asked to
participate in this investigation by completing the following steps for 5% extra credit towards their final grade.
Student participation was optional and they could drop out at will. Forty-eight students completed the
experiments. None of the subjects were disabled.

Procedures

There were four exams, three weeks apart, for the students to take. Each exam, which consisted of 50 multiple
choice questions with medium-level of difficulty, was to cover 3-4 chapters of the textbook materials. The
results of the first two exams were used to assess random assignment of the students to two groups (See
Appendix A). Assessment of the stated hypotheses was based on exams 3 and 4.

A 44-item questionnaire, adopted from Pintrich and DeGroot (1990), to assess the learner’s “motivation” and
“self-regulated learning” (Appendix B) was administered when the class had covered half the textbook and taken
the first two exams. By covering about half of the textbook materials and completing two exams, students were
expected to have a better understanding of their interest in the subject matter.

Next, students were randomly placed into two groups (Table 1). Each student in one group received a CD that
contained E-Tutor (with video), and each subject in the other group received a CD that contained E-Tutor
(without video). Students participating in this investigation were required to use E-Tutor to prepare for the two
scheduled exams 3 and 4. This repeated trials made it possible to assess the effects of videos embedded in the E-
Tutor within and between groups. A hands-on demonstration of E-Tutor use was done to make sure that all
students were familiar with the way the system functioned.

Table 1: Assignment of the subjects to the two test groups

Treatment Trial 1 Trial 2
Subject with access to E-Tutor (with Video) 25 23
Subjects with access to E-Tutor (without video) 23 25

Trial 1: Subjects were required to complete the pertinent chapters in E-Tutor in order to prepare for the
scheduled third exam. In addition, subjects who had access to E-Tutor (with video) were required to view all the
pertinent videos. E-Tutor kept track of each student’s progress through the lessons and saved it in a “working
file”. To make sure that all the subjects completed the assigned tasks, they were required to upload their
working files to a site. This upload included captured data about the date that the assigned chapters were
completed, as well as the video files—for students with E-Tutor (with video)—that were viewed. Two weeks
after receiving the E-Tutor CDs, students took a scheduled exam that consisted of 50 multiple-choice questions
covering the assigned chapters of the textbook (Appendix C). Prior to viewing the questions, they completed a
questionnaire to assess the usefulness of E-Tutor in support of related learning materials for the exam (Appendix
D).

Trial 2: The procedure followed in this trial was similar to trial 1, except that subjects who used E-Tutor (with
video) in trial 1 exchanged it for E-Tutor (without video). Those with E-Tutor (without video) exchanged it with

127
E-Tutor (with video). Two weeks after exchanging the E-Tutor CDs, students took the fourth exam. Prior to
viewing the questions on this exam, as in trial 1, subjects completed a questionnaire designed to assess their
perceived usefulness of videos.

Results

We used MANOVA to assess within and between subject difference in regard to the E-Tutor’s usefulness with
and without videos in support of learning and improved performance. Students’ “motivation” and “self-
regulated learning” were included in the analysis as covariates. Levene’s test of equality of error variance of the
dependent variables showed insignificant inequality across the groups (for usefulness F=0.49 (p=0.69) and for
the tests 3 & 4 F=0.93 (p=0.43)). In addition, Box’s test of equality of covariance matrices showed insignificant
inequality across groups (F=0.78 (p=0.63)). Hypothesis 1 states, “Inclusion of video in a CBT system has no
effect on the learner's perception regarding the usefulness of the system”. The average score for the E-Tutor
usefulness (Table 2) in the Trial 1 was: with video = 30.24 and without video = 27.64, and for the Trial 2: with
video = 30.97 and without video = 28.39. The results of MANOVA test demonstrated that there was a
significant difference (F=11.43), p<0.01) about the perceived usefulness of videos embedded in E-Tutor among
the groups (Table 3). Subsequent pairwise comparison of the variables showed that the group with access to
videos perceived E-tutor to be more useful than did the other group in both trial 1 and trial 2 (Table 4). There
was no significant difference in regard to the usefulness of E-Tutor when both groups had access to E-Tutor with
the same capability (i.e., both had access to E-Tutor with video or without videos). Analyses also showed that
subjects’ motivation to learn had a significant interaction effect on their perception of the E-Tutor usefulness to
enable them learn the pertinent reading materials (Table 3). However, subjects’ self-regulated learning had no
interaction effect on their perceived usefulness of E-Tutor.

The above analysis provides sufficient evidence that the difference in the perceived usefulness of the E-Tutor is
due to the presence or absence of videos. Therefore, based on the results of this experiment, the stated
hypothesis cannot be supported.

Table 2: Descriptive Statistics of the Research Variables

Subjects in Group 1 Subjects in Group 2
Variables Mean, (Variance) Mean, (Variance)
Min. – Max. Min. – Max.
Exam 1 79.13%, (7.30%) 77.04%, (6.30%)
68% - 88% 68% - 82%
Exam 2 78.80%, (10.02) 77.80%, (10.99%)
61% - 92% 60% - 94%
Exam 3 85.92%, (5.64%) 85.82%, (4.82%)
74% - 96% 72% - 96%
Exam 4 87.60%, (5.32%) 85.92%, (6.96%)
72% - 94% 70% - 96%
Motivational beliefs 91.44, (21.62) 93.35, (16.14)
44 - 119 63 - 127
Self-Regulated Learning Strategies 102.96, (18.42) 101.65, (20.96)
65 - 127 51 - 133
Usefulness of the E-Tutor for Trial 1 30.24, (2.42) 27.64, (2.12)
25 - 33 25 - 30
Usefulness of the E-Tutor for Trial 2 28.39, (1.92) 30.91, (2.65)
25 - 33 25 - 35

Table 3: MANOVA Tests of Between-Subjects with and without Video Presentation

Source Dependent Variable df Mean Square F-Ratio Sig.
Factor Usefulness 3 55.86 11.43 0.00
Score for exams 3&4 4.66 0.56 0.64
Motivation Usefulness 1 20.93 4.28 0.04
Score for Exams 3&4 3.57 0.43 0.51
Self-regulated learning Usefulness 1 0.06 0.01 0.91
Score for Exams 3&4 0.42 0.05 0.82

128
Our hypothesis 2 states; “Inclusion of videos in a CBT system has no effect on the learner's mastery of the
material to be learned.” The results of MANOVA test (Table 3) showed that there was no significant difference
between the scores on exams 3 and 4 with and without video (F = 0.56, p = 0.64). This indicates that although
availability of video was perceived to be useful in students’ learning processes, it had no effect on their mastery
of the material to be learned based on their exam scores. Thus, hypothesis 2 cannot be rejected.

Table 4: Pairwise Comparisons of Usefulness of E-Tutor with and without Video

Dependent Variable: Usefulness Mean Difference Sig. (p)
(I) FACTOR vs. (J) FACTOR (I-J)
Group with video in trial 1 vs. group without video in trial 1 2.60 0.00
Group with video in trial 2 vs. group without video in trial 2 test 2.52 0.00
Group with video in trial 1 vs. group with video in trial 2 -0.61 0.35
Group without video in trial 1 vs. group without video in trial 2 -0.69 0.29

Discussion
Inspired by the advent of new information and communication technologies, use of technology-mediated
learning environments is on the rise. The number of online courses offered through academic and professional
institutions is evidence of this. Although research related to technology-based learning has been with us for
more than 30 years, we are still at an early stage of knowing how to effectively apply technology in support of
learning. In this endeavor, textual material is supplemented by still images, animation, video sequences, and/or
sound sequences. The objective is to produce an enhanced learning environment in which students will be able
to learn and understand more effectively. This research shows that students find a learning environment with
video presentation more useful. Furthermore, it shows that students’ intrinsic motivation to learn the subject
matter had a significant positive effect on their satisfaction with the availability of video presentation. Based on
this, we can conclude that students with higher interest in the subject matter would have found the added
information gleaned from video presentation more satisfying. However, added functionality of video
presentation had no significant effect on performance in exams. That is, the addition of the video presentation to
the text and pictures did not produce significant gains in students’ ability to answer questions on it. Of course,
these findings are limited by the characteristics of our environment. E-Tutor was designed to support text-based
explicit knowledge to improve student mastery learning of basic MIS materials. However, the effect of video
presentation on learning might be different when used in support of tacit knowledge (e.g., analysis of complex
case studies) (Lim and Benbasat, 2000). Tacit knowledge relate to experiential learning, whereas explicit
knowledge deals with objective and technical knowledge. Thus, future research should assess the effectiveness
of video in support of learners in need of acquiring tacit knowledge.

Future research should find it interesting to assess the long-term effect of video on students learning and student
motivation to pursue the subject matter further. Learners’ cognitive style may also affect effectiveness of video
presentation (Montazemi and Wang, 1989). In this endeavor, we need to develop methodologies to distinguish
students as visual-spatial learners or auditory-sequential learners: Visual-spatial learner is believed to think in
pictures and have a different brain organization than auditory-sequential learner (Silverman, 2002). It is possible
that visual-spatial learns benefit more from video presentation. We should also search for different means of
using dynamic visuals in support of improved learning performance. Learning performance, as depicted in
Figure 1, is contingent on internal (long-term memory) and external (e.g., video) mental representation of the
learner. Dynamic visual based on animation is a possible medium to satisfy the individual learning requirements
of the learner. We can exploit the rapidly evolving graphics capabilities of information systems to provide
learners with engaging three-dimensional worlds (Badler et al., 2002). For example, to learn procedural tasks, a
three-dimensional learning environment could enable students to perform the task directly in the environment.
Hence, rather than memorizing an abstract procedure, students learn by interacting with rich three-dimensional
models representing the subject matter. Lifelike pedagogical agents can play a key role in a central problem
posed by task-oriented three-dimensional learning environment; detecting and correcting misconception. While
the pertinent technologies are new, they are beginning to mature, and we have already begun to see the
emergence of techniques for real-time behavior sequencing in response to rapidly changing learning contexts
(Badler et al., 2002; Funge, 2000; Light and Maybury, 2002).

Information technology can play a significant role in support of learning if the embedded functionality of the
system is perceived to be useful by the learner. We believe, as others do (Alavi and Leidner, 2001; Carroll,
1997; Ohlsson, 1991; Tong and Angelides, 2000), that system “hacking” is not an effective means of improving
the state of the art in technology-mediated learning (TML). Instead, well-grounded learning theories need to be

129
empirically tested in TML environments. Only then will system designers be provided with the much-needed
theoretical foundation for implementing more perfect TML systems.

Acknowledgement
The Author would like to thank the three anonymous reviewers for their comments on the earlier version of this
manuscript. This research has been supported in part by grants from Indiana University Ameritech Fellow of
Information Technology Program, and Natural Sciences and Engineering Research Council of Canada.

References
Alavi, M. & Leidner, D. E., (2001). Research Commentary: Technology-Mediated Learning-A Call for Greater
Depth and Breath of Research. Information Systems Research, 12 (1), 1-10.

Ames, C. & Archer, J., (1988). Achievement Goals in the Classroom: Student Learning Strategies and
Motivation Processes. Journal of Educational Psychology, 80 (4), 260-267.

Anderson, A. H., Smallwood, L., MacDonald, R., Mullin, J., Annemarie, F. & O’Malley, C., (2000). Video Data
and Video Links in a Mediated Communication: What Do Users Value? International Journal of Human-
Computer Studies. 52 (1), 165-187.

Badler, N. I., Erignac, C. A. & Liu, Y., (2002. Virtual Humans for Validating Maintenance Procedures.
Communications of the ACM. 45 (7), 57-63.

Benbunam-Fich, A, (2002). Improving Education and Training with IT,” Communications of the ACM. 45 (5),
94-99.

Carroll, J. M., (1997). Human-Computer Interaction: Psychology as a Science of Design. International Journal
of Human-Computer Studies. 46 (4), 501-522.

Corno, L., (1986). The Metacognitive Control Components of Self-Regulated Learning. Contemporary
Educational Psychology, 11 (4), 333-46.

Corno, L. & Mandinach, E., (1983). The Role of Cognitive Engagement in Classroom Learning and Motivation.
Educational Psychologist, 18 (4), 88-100.

Corno, L. & Rohrkemper, M., (1985). The Intrinsic Motivation to Learn in Classrooms,” In Ames, C. & Ames,
R. (Eds.), Research in Motion, Vol. 2. The Classroom Milieu, Academic Press, New York, NY, 53-90.

Davis, F. D., (1989). Perceived Usefulness, Perceived Ease of Use and User Acceptance of Information
Technology. MIS Quarterly, 13 (3), 319-39.

Davis, F. D., Bagozzi, R. P. & Warshaw, P. R., (1992). Extrinsic and Intrinsic Motivation to Use Computers in
the workplace. Journal of Applied Social Psychology, 22 (4), 1111-32.

Davis, F. D., Bagozzi, R. P. & Warshaw, P. R., (1989). User Acceptance of Computer Technology: A
Comparison of Two Theoretical Models. Management Science, 35 (8), 982-1002.

Funge, J., (2000). Cognitive Modeling for Games and Animation. The Communications of the ACM. 43 (7), 41-
48.

Hannafin, M. J. & Rieber, L. P., (1989). Psychological foundations of instructional design for emerging
computer-based instructional technologies. Part II. Educational Technology Research and Development, 7, 102-
14.

Heller, R. C., Martin, C. D., Haneef, N. & Gievska-Krliu, S., (2001). Using a Theoretical Multimedia Taxonomy
Framework. ACM Journal of Educational Resources in Computing, 1 (1), 1-22.

130
Kerlinger, F. N., (1973). Foundations of Behavioral Research. Holt, Rinehart, and Winston Inc., Chicago, IL.

Leidner, D. E. & Jarvenpaa, S. L., (1995). The Use of Information Technology to Enhance Management School
Education: A Theoretical View. MIS Quarterly, 19 (3), 265-91.

Light, M. & Maybury, M. T., (2002). Personalized Multimedia Information Access. Communications of the
ACM, 45 (5), 54-59.

Lim, K. H. & Benbasat, I, (2000). The Effect of Multimedia on Perceived Equivocality and Perceived
Usefulness of Information Systems. MIS Quarterly, 24 (3), 449-471.

Mayer, R. E. & Sims, V. K., (1994). For Whom Is a Picture Worth a Thousand Word? Extensions of a Dual-
Coding Theory of Multimedia Learning. Journal of Educational Psychology, 86 (3), 389-401.

Meece, J., Blumenfeld, P. & Hoyle, R., (1988). Students’ Goal Orientations and Cognitive Engagement in
Classroom Activities,” Journal of Educational Psychology, 80 (4), 514-23.

Montazemi, A. R. & Wang, F., (1995-a). CBT in Support of Mastery Learning. Journal of Educational Computing
Research, 13 (2), 185-205.

Montazemi, A.R. & Wang, F., (1995-b). On the effectiveness of Neural Network for Adaptive External Pacing.
Journal of Artificial Intelligence in Education, 6 (4) 379-404.

Montazemi, A. R. & Wang, S., (1989). The Effects of Modes of Presentation on Decision Making: A Review and
Meta-Analysis. Journal of Management Information Systems, 5 (3), 111-127.

Nolen, S. (1988). Reasons for Studying: Motivational Orientations and Study Strategies,” Cognition and
Instruction, 5 (4), 269-287.

O’Brien, J. A., (2001). Management Information Systems; Managing Information Technology in the E-Business
Enterprise. New York, NY, McGraw-Hill.

Ohlsson, S., (1991). System Hacking meets learning Theory: Reflection on the Goals and Standards of Research
in Artificial Intelligence and Education. Journal of Artificial Intelligence in Education, 2 (3), 5-18.

Pane, J. F., (1994). Assessment of the ACSE Science Learning Environment and the Impact of Movies and
Simulations,” Carnegie Mellon University, CMU-CS-94-162.

Pane, J. F., Corbett, T. A., & John, B. E., (1996). Assessing Dynamics in Computer-Based Instruction. In
Proceedings of the Conference on Human Factors and Computing Systems, April 13-18, Vancouver, Canada,
197-204.

Park, O. & Hopkins, R., (1993). Instructional Conditions for Using Dynamic Visual Displays: A Review.
Instructional Science, 21 (6), 427-499.

Pintrich, P. R., (1988). A Process-Oriented View of Student Motivation and Cognition. In Stark, J. S. &Mets, L.
(Eds.), Improving Teaching and Learning Through Research. New Directions for Institutional Research, San
Francisco, CA, Jossey-Bass, 55-70.

Pintrich, P. R., (1989). The Dynamic Interplay of Student Motivation and Cognition in the College Classroom.
In Ames, C. & Maehr, M. (Eds.), Advances in Motivation and Achievement: Vol. 6. Motivation Enhancing
Environments, Greenwich, CT, JAI Press, 117-160,.

Pintrich, P. R. & DeGroot, V. E., (1990). Motivational and Self-Regulated Learning Components of Classroom
Academic Performance. Journal of Educational Psychology, 82 (1), 33-40.

Pilkington, R. & Grierson, A., (1996). Generating explanation in a simulation-Based Learning Environment.
International Journal of Human-Computer studies, 45 (5), 527-551.

131
Rieber, L. P., (1990). Animation in Computer-Based Instruction. Educational Technology Research and
Development, 38 (1), 77-86.

Retalis, S. & Papasalours, A., (2005). Designing and Generating Educational Adaptive Hypermedia
Applications, Educational Technology and Society, 8 (1), 69-79.

Silverman, L. K., (2002). Upside-down Brilliance: The Visual-Spatial Learner, DeLeon Publishing, Denver, CO.

Simonson, M., (1985). Persuasive Films: A Study of Techniques Used to Change Attitudes. Journal of Teaching
and Learning Technologies. 1 (2), 39-48.

Simonson, M. & Maushak, N., (1995). Situated Learning, Instructional Technology, and Attitude Change. In
McLellan, H. (Ed)., Perspectives on Situated Learning, Cliffs, NJ, Englewood, 46-79.

Stone, R., (2001). Virtual Reality for Interactive Training: An Industrial Practitioner’s View Point. International
Journal of Human-Computer Studies. 55 (4), 699-711.

Taylor, S. & Todd, P., (1995). Understanding Information Technology Usage: A Test of Competing Models.
Information Systems Research. 6 (2), 144-176.

Tong, A. K. Y. & Angelides, M. C. (2000) An Empirical Model for Tutoring Strategy Selection in Multimedia
Tutoring Systems. Decision Support Systems. 29 (1), 31-45.

Velayo, R. S., (2000). How do Presentation Modality and Strategy Use Influence Memory for Paired Concepts.
Journal of Instructional Psychology. 27 (2), 126-133.

Venkatesh, V., (1999). Creation of Favorable User Perceptions: Exploring the Role of Intrinsic Motivation. MIS
Quarterly, 23 (2), 239-260.

Weinstein, C. E. & Mayer, R. E., (1986). The Teaching of Learning Strategies. In Wittrock, M. (Ed.), Handbook
of Research on Teaching, Macmillan, New York, NY, 315-327.

Zimbardo, P. & Leippe, M., (1991). The Psychology of Attitude Change and Social Influence. Temple
University Press, Philadelphia, PA.

Zimmerman, B. & Pons, M., (1988). Construct Validation of a Strategy Model of Student Self-Regulated
Learning. Journal of Educational Psychology, 80 (3), 284-290.

132
Appendix A
Questionnaire Validity and Reliability

One measure of construct validity is the extent to which each item of a questionnaire correlates with the total
score (Kerlinger, 1973). The correlation between the score for each item (question) and the total usefulness
score was 0.65 to 0.84 (all significant at p<0.001) (Table 1A). Correlation between the overall score for
motivation with 22 pertinent items showed that items 1 and 22 were not significantly correlated (alpha >0.05).
We removed these two items from the subsequent analysis. Correlation between overall score for self-regulated
learning and the 22 pertinent items (i.e., items 23–44 as depicted in Appendix B) showed that three items 26, 27
and 37 were insignificant. Thus, these three items were removed from the subsequent analysis.

Table 1A: Validity and Reliability of the Questionnaires

Validity Reliability
Questionnaire Correlation Coefficient Factor Analysis Cronbach alpha
(all significant at alpha < 0.01) (Item communality range)
Usefulness for Trial 1 0.65 – 0.84 0.84 – 0.88 0.83
Usefulness for Trial 2 0.68 – 0.90 0.86 – 0.95 0.88
Motivation 0.51 - .88 0.65 – 0.89 0.91
Self-regulated learning 0.50 – 0.77 0.59 – 0.76 0.86

The data were further analyzed using factor analysis (principal component analysis using Varimax rotation
method with Kaiser normalization) to establish convergent and discriminant validity (Table 1A). The factor
analysis for usefulness questionnaire scores in exam 3 and exam 4 showed one orthogonal factor with
eigenvalues above 1.0 accounting for 97.18% and 99.18% respectively, with item communality ranging between
0.84 and 0.95 (Table 2A). The factor analysis for “motivation and self-regulated learning” showed two
orthogonal factors with eigenvalues above 1.0, together accounting for 76.4% of the variation, with item
communality ranging between 0.59 and 0.89.

The Cronbach alpha reliability score for the three questionnaires was 0.83 to 0.91 (Table 1A). These values are
well within the thresholds suggested by Nunnally (1978). Thus, the questionnaires can be used as acceptable
instruments for subsequent analysis.

Table 2A: Factor Analysis for Perceived Usefulness

Item Factor loading Item Communality
Q1-Trial 1 0.932 0.880
Q1-Trial 2 0.932 0.891
Q2-Trial1 0.897 0.871
Q2-Trial2 0.876 0.863
Q3-Trial1 0.898 0.853
Q3-Trial2 0.917 0.885
Q4-Trial1 0.896 0.835
Q4-Trial2 0.946 0.949
Q5-Trial1 0.911 0.846
Q5-Trial2 0.946 0.949

Test for Random Assignment

To test for random assignment of subjects to the two groups, the following analysis was performed. The
ANOVA test was used to assess subject performance for the first and second multiple-choice exams (Table 4) to
find out if there were any differences between the two groups regarding their domain knowledge. The result of
the ANOVA test demonstrated no significant difference between the two groups: F = 1.104 (p = 0.299) for the
first exam and F = 0.648 (p = 0.425) for the second exam. We also used ANOVA to test possible differences
between the two groups in regard to the subjects’ “motivational beliefs” and “self-regulated learning strategies”.
The results of ANOVA test showed no significant difference between the two groups: F = 0.079 (p = 0.971) for
the motivational beliefs and F = 0.035 (p = 0.991) for the self regulated learning strategies.

References
Nunnally, J. C. (1978). Psychometric Theory, New York, NY: McGraw-Hill.
133
Appendix B
The following 44 items represent the Motivated Strategies for Learning Questionnaire (MSLQ) used in this
study to measure students’ motivational beliefs and self-regulated learning. The number next to the items
reflects the item’s actual position on the questionnaire.

All the items were assessed on the following 7-point Likert-type scale:

1 2 3 4 5 6 7
not at all true of me |-------------|-------------|-------------|-------------|-------------|-------------|------------| Very true of me

Motivational Beliefs
2. Compared with other students in this class I expect to do well.
6. I’m certain I can understand the ideas taught in this course.
8. I expect to do very well in this class.
9. Compared with other students in this class, I think I am good student.

I am sure I can do an excellent job on the problems and tasks assigned for this class.
13. I think I will receive a good grade in this class.
16. My study skills are excellent compared with others in this class.

Compared with other students in this class I think I know a great deal about the subject.
19. I know that I will be able to learn the material for this class.
1. I prefer class work that is challenging so I can learn new things.
4. It is important for me to learn what is being taught in this class.
5. I like what I am learning in this class.
7. I think I will be able to use what I learn in this class in other classes.

I often choose paper topics I will learn something from even if they require more work.
14. Even when I do poorly on a test I try to learn from my mistakes.
15. I think that what I am learning in this class is useful for me to know.
17. I think that what we are learning in this class is interesting.
21. Understanding this subject is important to me.
3. I am so nervous during a test that I cannot remember facts I have learned.
12. I have an uneasy, upset feeling when I take a test.
20. I worry a great deal about test.
22. When I take a test I think about how poorly I am doing.

Self-Regulated Learning

When I study for a test, I try to put together the information from class and from the book.
24. When I do homework, I try to remember what the teacher said in class so I can answer the
questions correctly.
26. It is hard for me to decide what the main ideas are in what I read.
28. When I study I put important ideas into my own words.

I always try to understand what the teacher is saying even if it does not make sense.
30. When I study for a test I try to remember as many facts as I can.
31. When studying, I copy my notes over to help me remember material.

When I study for a test I practice saying the important facts over and over to myself.

I use what I have learned from old homework assignments and the textbook to do new assignments.
39. When I am studying a topic, I try to make everything fit together.
41. When I read material for this class, I say the words over and over to myself to help me
remember.
42. I outline the chapters in my book to help me study.

When reading I try to connect the things I am reading about with what I already know.
134
 25. I ask myself questions to make sure I know the material I have been studying.
27. When work is hard I either give up or study only the easy parts.
32. I work on practice exercise and answer end of chapter questions even when I do not have to.

Even when study materials are dull and uninteresting, I keep working until I finish.

35. Before I begin studying I think about the things I will need to do to learn.
37. I often find that I have been reading for class but do not know what it is all about.
38. I find that when the teacher is talking I think of other things and do not really listen to what is
being said.
40. When I’m reading I stop once in a while and go over what I have read.
43. I work hard to get a good grade even when I do not like a class.

135
Appendix C

A Sample of Exam Questions used in Trials 1 & 2

End user resistance can be minimized through:
A) Proper end user education and training.
B) Improved communications with IS professionals.
C) End user involvement in the development and implementation of new systems.
D) All of the above.

Management involvement in information systems can take the form of:

A) Top management participation in an executive information technology committee.
B) Management participation in steering committees for systems development projects.
C) End user management participation in the systems development process and the management of
information systems.
D) All of the above.

Your text defines change management as managing the process of implementing major changes in information
technology, business processes, organizational structures, and job assignments to reduce the risks and costs of
change, and optimize its benefits. A number of factors (dimensions) of change management are illustrated and
discussed. Of these factors, which one is the major focus of organizational change management?
A) Process
B) People
C) Technology.
D) Knowledge management.

Your text states that the E-business planning process ahs three major components - strategy development,
resource management, and technology architecture. Strategy Development involves:
A) Developing E-business and E-commerce strategies that support a company's E-business vision that
focuses on customer and business value.
B) Developing strategic plans for managing or outsourcing a company's IT resources.
C) Developing strategic IT choices that reflect information technology architecture designed to support a
company's E-business and E-commerce initiatives.
D) None of the above applies to defining strategy development.

Prototyping involves:
A) The execution of the standard systems development cycle using CASE tools.
B) A rapid generation of systems by information systems professionals without the need for end user input.
C) The use of a fail-safe development process designed to ensure that an information system meets all user
requirements without revision.
D) An iterative and interactive development process with extensive end user involvement.

Organizations are increasingly using encryption of data and firewall computers as methods to protect computer
network resources. Firewalls are described as:
A) The transmission of data through telecommunications lines in "scrambled" form.
B) A "gatekeeper" system that protects a company's intranets and other computer networks from intrusion
by providing a filter and safe transfer point for access to and from the Internet and other networks.
C) The act of ensuring the accuracy, integrity, and safety of all E-business processes and resources.
D) None of the above apply.

A DBMS query language is designed to:

A) Support information systems professionals in the development of complex application software.
B) Support end users that wish to interrogate the database using English-like or natural language
commands.
C) Provide efficient processing of the database in the batch mode.
D) Specify the content, relationships, and structure of a database.

In the data planning and modeling process described in your text:

A) Database development starts with a bottom-up data planning process.
136
B) Subject area databases are consolidated to form the enterprise model.
C) User views are generated by identifying the key data elements in the subject area databases, which are
needed to perform specific business activities.
D) The enterprise model is mapped directly into the physical database design.

The logical structure of information in a database is contained in the

A) Data dictionary.
B) Data manipulation subsystem.
C) Data administration subsystem.
D) Data definition subsystem.

A network that contains one or more host computers that provide some type of service to the other computers in
a network is a:
A) Peer-to-peer network
B) Client/server network
C) Local area network.
D) Wide area network.

137
Appendix D

Perceived Usefulness Questionnaire

All the items were assessed on the following 7-point Likert-type scale:

Likely |________|________|________|________|________|________|________| Unlikely

extremely quite slightly neither slightly quite extremely

1. Using E-Tutor enables me to learn the MIS course materials more quickly

2. Using E-Tutor improve my performance in the pertinent tests.

3. Using E-Tutor enhance my learning effectiveness

4. Using E-Tutor make it easier to learn the subject matter

5. I find E-Tutor useful in learning subject matter of MIS course

138