A Comparison of Laboratory and Field Research in the Study of Electronic Meeting Systems

ALAN R. DENNIS, JAY F. NUNAMAKER. JR.. and DOUGLAS R. VOGEL

ALAN R. DENNIS is a doctoral candidate in MIS at the University of Arizona. He received a Bachelor of Computer Science from Acadia University and an M.B.A. from Queen's University in Kingston, Ontario, and was a winner of the AACSB National Doctoral Fellowship. Prior to entering the Arizona doctoral program, he spent three years as a faculty member of the Queen's University School of Business. He has published several conference proceedings, book chapters, and journal articles (in, among others, MIS Quarterly, Information & Management, Data Base, Decision Support Systems, Computers & Graphics, IEEE Transactions on Systems, Man and Cybernetics, and International Journal of Man-Machine Studies). His current research interests include electronic meeting systems, decision support systems, and business graphics.

is Assistant Professor of MIS. College of Business and Public Administration, University of Arizona, Tucson. He has been involved with computers and computer systems in various capacities for over 20 years. He received his MS in computer science from UCLA, and his Ph.D. in MIS from the University of Minnesota, where be was also coordinator for MIS Research Center. His current research interests bridge the business and academic communities in addressing questions on the impact of management information systems on aspects of interpersonal communication, group decision making, and organizational productivity.
DOUGLAS R . VOGEL

Research into the use of Electronic Meeting Systems (EMS) has rapidly increased over the past few years. However, EMS laboratory experiments have often drawn very different conclusions about the effects of EMS use than have EMS field studiesconclusions that at first appear inconsistent. By examining the differences in the design of prior EMS studies, we attempt to better understand tbe factors that affect the use of EMS technology. It is our contention that these differences in findings are not inconsistent, but rather they reflect different situations that researchers have studied. We identify 24 potentially important differences in organizational contexts, group characteristics, tasks, and EMS environments. In planning future research, EMS researchers need to make explicit design decisions for each of these aspects, and to consider how those choices may affect research results. An analysis of these differences suggests several approaches to enhance the design of future laboratory experiments and field studies.
ABSTRACT:

An earlier version of this paper was originally published in the Proceedings of the Twenty-Third
Hawaii International Conference on System Sciences (IEEE Computer Society Press, 1990).
Journal of Management Information Systems /Winter 1990-91, Vol. 7, No. 3, pp. 107-135. Copyright M. E. Sharpc, Inc., 1991.

tO8

DENNIS, NUNAMAKER. AND VOGEL

KEY WORDS AND PHRASES; electronic tneeting systems, decision support systems, research methodologies. ACKNOWLEDGMENT We would like to thank Terry Connolly, Joey George, Scott Poole, Daryl Sabers, Joe Valacich, Gary Wagner, Mary Anne Winniford, and Bob Zmud for helpful comments on earlier drafts of this paper. This research was partially stjpported by the IBM Management of Information Systems Program, with additional funding from the Social Sciences and Humanities Research Council of Canada.

Introduction
the use of information technology lo support group work. As the scope of problems supported and the nature of this technology has evolved, it has been known by many names. We prefer the term Electronic Meeting System (EMS) [9]. Currency, the results from different types of EMS research tell us different things. As shown in the next section, many experiments in laboratory settings have found that, while the use of F.MS software can improve the quality of meeting outcomes, it can also require slightly longer meeting times and lead to decreased participant satisfaction. In contrast, many field studies of EMS use have found that the use of EMS produces higher quality outcomes in less time, with highly satisfied participants. At first glance, Lhese results appear inconsistent. However, it is our contention that they are not inconsistent, but rather reflect the different contexts, groups, tasks, and EMS that researchers have studied. The purpose of this paper is to examine previous laboratory experiments and field studies to try to understand what may account for differences in findings. By examining the differences in contexts, groups, tasks, and EMS among prior studies, we can identify some of the factors that potentially influence the effects of EMS use. This analysis should provide four benefits. First, it will help researchers and practitioners understand the bounds to which specific research findings can be generalized. Second, it will encourage researchers to document critical aspects of their studies, so that results from different studies can be better compared. Third, it will highlight factors that need to be addressed in the future empirical evaluation and development of EMS technology. Finally, it will begin to lay the foundation for research leading to the development of theories of EMS. The next section briefly reviews previous EMS laboratory and field studies. The third section highlights the key design differences between these two classes of study, and considers how these differences could account for the contrasting fmdings. The fmal section discusses implications for researchers. Previous EMS Research
IN TfflS PAPER, WE FOCUS EXCLUSIVELY ON PUBUSHED RESEARCH (as of the Summer
IN RECENT YEARS, THERE HAS BEEN A GROWING INTEREST in

of 1990) thatevaluatestheeffectsof using EMS meeting rooms where groups meet at the same time and place. Obviously, this excludes some valuable dissertations and papers that have not yet reached publication [19], papers that examine asynchronous or nonproximate EMS-supported meetings [62], and papers whose primary purpose

LABORATORY VS. FIELD RESEARCH ON EMS

109

was not the evaluation of EMS technology [31]. Our focus on EMS meeting rootns is not to suggest that all such meetings are similar; indeed, as we shall sec in the next section, such meetings have varied considerably among studies. The purpose of this section is to provide a very brief review of the findings from previous research (see also [9,41, 53]). There are a variety of research methodologies available to researchers in the Management Information Systems (MIS) area, as recent discussions in MIS joumals have highlighted [cf. 3,16,20,34]. To date, laboratory experiments and field studies have been the most commonly used methodologies in the study of EMS. The applicability of conclusions from laboratory studies to the organizational use of Mis has been challenged by field researchers, as the laboratory setting, the use of students as subjects, and the nature of experimental tasks are often different from those encountered in the organizational use of MIS. Galliers and Land note that "although the experimental design of such is research may well be academically acceptable and intemaliy consistent, all too often it leads to inconclusive or inapplicable results" [20, p. 900]. The question is how to interpret behavior exhibited in laboratory settings and apply conclusions from it to behavior in natural settings. That is, while laboratory studies may have intemal validity, they may lack extemal validity. In contrast, the difficulty in attributing the observed outcomes to the use (or nonuse) of MIS technology in field settings has caused the use of field studies to come under fire. The different standards of evidence used in field studies have been challenged by experimentalists: "The ideal of science is the controlled experiment. . . . The main reason for the preeminence of the controlled experiment... is that researchers can have more confidence that the relations they study are the relations they think they are" [40, p. 293]. Field studies typically involve the examination of open systems; that is, systems that permit other factors not related to the research design to affect the observed outcomes [7]. Separating the effects of the factors of interest from those of other factors can he difficult. That is, while field studies may have extemal validity, they may lack intemal validity. In this paper, we define EMS field studies as studies of the use of EMS technology by organizational groups (public or private sector) addressing problems of their own choosing at an EMS facility at their organization or another institution, where data are collected to develop theory or evaluate research propositions. Laboratory experiments involve groups using EMS technology to address tasks prescribed by a researcher for the purpose of examining the effects of the use of the technology, often using student subjects. The field experiment by Jarvcnpaa et al. [36] largely parallels this design and is grouped with the laboratory experiments.

Laboratory Experiments
Experimental research conducted in EMS meeting rooms has produced a variety of research findings for effectiveness, efficiency, and group member satisfaction (see Table 1). Studies that did not measure one or more of these constructs are indicated with an "n/a." (Studies that did not directly compare EMS-supported to non-EMS-supported groups list an "n/a" for all three constnicts.) The findings have been mixed.

110

DENNIS. NUNAMAKER. AND VOGEL

Table 1

Outcomes (Compared to No EMS Support) Effectiveness Efficiency decreased rt/a n/a n/a n/a n/a decreased n/a n/a no change decreased decreased n/a n/a n/a n/a n/a nAi n/a decreased decreased n/a n/a n/a n/a Efficiency increased
iiKTeased

Experiments

Member satisfaction in/decreased n/a n/a n/a n/a n/a increased n/a no change decreased no change decreased n/a no change n/a n/a n/a n/a n/a n/a increased n/a n/a decreased n/a Member satisfaction increased increased increased in/decreased increased increased increased increased increased increased

[4] Bui increased [5] Chidambarum n/a [6] Connolly n/a [8] Dennis n/a [12] Dennis n/a [13] DeSanctis n/a [17] Easton increased n/a [18] Baston [21] Gallupe no change [22] Gallupe increased [23] Gallupe no change no change [24] George [30] Ho n/a [34] Jarvenpaa increased n/a [37] Jessup n/a [38] Jessup [43] Um n/a [44] Loy n/a [58] Sambamunhy n/a [59] Sharda increased [63] Steeb no change [64] Valacich n/a [65] VanShaik no change [68] Watson n/a [69] Zigurs n/a Field studies [2] [10] [11] [47] [50] [51] [52] [64] [66] [67] Adelman Dennis Dennis McCartt Nunamaker Nunamaker Nunamaker Valacich Vogel Vogel Effectiveness increased increased increased in/decreased increased increased increased increased increased increased

increased in/decreased increased increased increased increased increased increased

Some experimental studies found the use of EMS to improve meeting effectiveness (e.g., decision qtiality) [4,17,22,34,59], while others found no effect on effectiveness [21,23,24,63,65]. Some studies found EMS use to decrease efficiency (i.e., increase the time required to complete the task) [4, 17, 23, 24, 59, 63]. One found no effect

LABORATORY VS. HELD RESEARCH ON EMS

UI

[22]. Some studies found that EMs use increased group member satisfaction [4, 17. 63] or decreased satisfaction [4,22,24,68] or had no effect [21,23,34]. It is difficult to see any clear pattem in these findings.

Field Study Research

Comparatively fewer field studies have yet reached pubhcation (see Table 1) [2,10, 11,47,50,51,52,64,66,67]. None of these studies have directly compared EMS- and non-EMS-supponed groups, although most have asked participants to compare the two indirectly. Many of these field studies could also be tenned action research, as the researchers were also participants (i.e., the facilitator). In these cases, the researcher(s) had two objectives: to develop a better understanding of the effects of EMS technology, and to assist the group in performing its task. An exception to this are two studies reported by Nunamaker etal. [52] and Vogel etal. [66] in which researchers designed the data collection plan and instruments, but played no role in the meetings. In vinually all cases (some exceptions are noted in McCant and Rohrbaugh [47]), participants found the use of EMS technology improved meeting effectiveness, efficiency, and member satisfaction. In most cases, these time improvements were postmeeting perceptions, but in one case [52] they were differences between premeeting time estimates and actual meeting times. The pattem here is clear: field studies indicate improvements in effectiveness, efficiency, and satisfaction for EMS use.

Key Differences in Design

I N T i n s SECTION, WE ffiGHUGHT DESIGN DIFFERENCES IN PREVIOUS RESEARCH. A few

of the differences we identify are inherent and caused by the choice of research methodology, whether laboratory or field. Many other differences are not inherent, but rather reflect design decisions quite amenable to change in futtire studies. For example, in the following sections we note that previous laboratory experiments typically studied the use of EMS technology by small groups addressing operationallevel decisions, while fteld research typically studied larger groups addressing strategic-level problems. There is no inherent reason for this focus; future studies could quite easily have a different focus. As we will show below, there is reason to expect that altemative focus may yield different fmdings. We have argued that the processes and outcomes of group woric depend on the interactions among four groups of variables: organizational context, group characteristics, task, and EMS technology [9]. What variables within these four groups have significant effects remains an open question. We have selected 24 variables; clearly there are many others that could be considered. However, the number of variables also precludes meta-analysis as there are too many variables for the number of studies. Our selection was motivated both by theoretical arguments (i.e., there is some a priori reason to suspect that the variable might affect processes and outcomes) and empirical fmdings (i.e., there have been differences among studies that have differed in these characteristics).

112

DENNIS, NUNAMAKER. AND VOGEL

Table 2

Organizational Context Organizational culture student student student student student student student student student student student student Asian student researcher student private sector Asian student student student student student student student student student Organizational culture military private sector public & private public sector public & private public & private private sector public & private private sector public sector Incentives no yes Same objectives
yes yes yes yes

Experiments [4] [5] [6] [8] [12] [13] [17] [18] [21] [22] [23] [24] [30] [34] [37] [38] [43] [44] [58] [59] [63] [64] [65] [68] [69] Bui Chidambarum Connolly Dennis Dennis DeSanctis Easton Easton Gallupe Gallupe Gallupe George Ho Jarvenpaa Jessup Jessup Lim Loy Sambamurthy Sharda Steeb Valacich VanShaik Watson Zigurs

Interrelated problems
no no no no

no
no

no no no no
no yes no no no no no no no no no yes no no no no
no

yes yes & no yes yes yes yes yes yes yes & no yes yes yes yes & no yes yes yes yes yes yes yes & no yes Same objectives yes& no yes & no yes & no yes & no yes & no yes & no yes & no yes & no yes & no yes & no

no no no no no no no no no no no no no no no no no no no no
no

Eield studies [2] [10] [11] [47] [50] [51] [52] [64] [66] [67] Adelman Dennis Dennis McCartt Nunamaker Nunamaker Nunamaker Valacich Vogel Vogel

Incentives
yes yes yes yes

Interrelated problems
yes yes yes yes yes yes

yes
yes yes yes yes yes

yes yes yes yes

LABORATORY VS. FIELD RESEARCH ON EMS

113

Organizational Context
"The embedding organization is a crucia] part of any work group's context, and the pattem of mutual inter-depcndence between the group and the surrounding organization is a crucial part of the substantive phenomena to be explored" [49, p. 385]. We note four contextual factors that may be important (see Table 2). First, organizational culture and behavior norms serve as a guide to the meeting process for organizational groups in field studies. Norms may be lacking in laboratory groups formed for the purpose of an experiment; an assembled group of individuals may be a group in body, but not in spirit. In preexisting experimental groups, contextual norms may simply be different from the norms of organizational groups. For example, in one laboratory experiment using preexisting groups, the norms of one group were such that it choose not to use the EMS [69]. Second, in organizational groups, group members have incentives to perform. Accomplishing the task successfully means recognition and reward for the group. In some experiments, where the tasks are such that performance can be measured objectively, this has been provided by pay and incentives based on experimental performance [e.g., 22]. Third, members of organizational groups may not always have consistent goals and objectives; there may be political elements such that the "best" outcome for some group member(s) is different from that for other group member(s). Tasks in experiments have traditionally presumed the rational model, "where organizational decisions are consequences of organizational units using information in an intendedly rational manner to make choices on behalf of the organization" [32, p. 3], although some [e.g., 68] have involved what are essentially bargaining tasks that have no right answers. In field studies, objectives have not always followed the rational model. They have often had a political component, "where organizational decisions are consequences of the application of strategies and tactics by units seeking to infiuence decision processes in directions that will result in choices favorable to them" [32, p. 3]. Thus the goals (or incentives) for participants in field studies have not always been congruentorthogonal or in direct competition on occasionin sharp contrast to the goals of participants in most laboratory studies, whose goal has often been to make the "best" decision for the organization in the experimental case. Finally, for organizational groups, issues and problems are interrelated. "Basically, every real world problem is related to every other real world problem" [46, p. 4, original emphasis]. Thus every time an organizational group attempts to resolve a particular problem, it needs to consider the problem's potential relationship with all other problems. This generally has not been a concem for groups in laboratory experiments. In summary. Table 2 shows that most previous laboratory experiments have examined student-related organizational cultures without perfonnance incentives but with common objectives without interrelated problems. Most field studies have examined public and/or private sector cultures with incentives and interrelated problems, but with necessarily the same objectives. We note that the three laboratory

114

DENNIS. NUNAMAKER, AND VOGEL

experiments finding EMS use to improve performance provided incentives to subjects or studied a nonstudent organizational culttire.

Group Characteristics
There have been many differences between the groups used in experimental research and those in the field that may account for differences in fmdings; we focus on seven (see Table 3). First, most experimental groups have been composed of students, while organizational groups have been composed of managers and professionals. Individual characteristics of the two populations may be different. For example, one individual characteristic of interest is prior experience with group meetings; most managers and professionals will have had more experience with group meetings than undergraduates. While the use of senior and junior undergradtiate students in experimental research is a common practice, studying one population (e.g., undergraduates) may produce different results than studying another (e.g., managers and professionals) [7]. Based on a review of 32 (non-Mis) studies comparing undergraduates to managers, Gordon, Slade, and Schmitt [26] conclude that there are signiftcant differences between undergraduates and managers. In MIS, Remus [55, 56] has also found that there can be important differences between managers and undergraduates in decisionmaking processes and performance. Of course, all groups of students or groups of managers and professionals are not the same; some undergraduates may have prior work experience, while participants in field studies may be recent graduates with little or no experience. Reporting subject demographics becomes important. A second potential factor is the familiarity of group members with the task. Members of organizational groups typically have had more experience with the task, as, in general, they address the tasks faced in the ongoing management of the organization. In contrast, experimental groups have often had less familiarity with the task they have been assigned. Third, experimental groups have typically been ad hoc, formed for the sole purpose of the experiment, and have no past history or foreseeable future. Field studies have typically used established groups, for whom the meeting under study is just one in a long series of meetings. Participants in field studies need to associate with each other and live witb the meeting outcome long after the meeting is over, in contrast with many experimental groups. Differences have been observed between ad hoc and esiablisbed/ongoing groups in non-EMS-supported research [e.g., 28] and in EMS research [8]. Fourth, groups in experimental studies have generally been peers. While some experiments have studied the effects of an emerging leader or have temporarily assigned a leader for the duration of the experimental session [e.g., 24], this form of leadership can be different from that in organizations. Groups in previous field studies have generally had a distinct hierarchy and/or differing social status among members; the leader was the leader before, during, and after the meeting. The leader(s) promotes and rewards participants. His/her presence was felt during the meeting, although he/she may not always have displayed leadership during the meeting. These power and status differences may lead to different meeting processes and different group

LABORATORY VS. FIELD RESEARCH ON EMS

115

needs than groups in which participants have similar power and status. Different group structure and leadership has been shown to affect non-EMS-supported meetings [60]. Fifth, participants in experiments have often been fyst-time users of the EMS technology. Participants in field studies have also often been fu-st-time users, but by the end of the study they have often logged many hours of use; they are moderately experienced EMS users. Observations drawn from the study of inexperienced users of EMS technology may be useful, but may apply only to inexperienced users. Generalizing the results of these studies to the ongoing organizational use of EMS may not be appropriate. Sixth, researchers have speculated that EMS may prove more effective for larger rather than smaller groups [9,14]. Most experimental research to date has focused on small groups (often 3-6 members). In contrast, most field study groups have been larger (typically 10 or more members). Group size, of course, has been shown to have significant impacts on non-EMS-supported group work [e.g., 29]. Finally, we consider the logical size of the group in addition to the physical size of the group. Groups can be considered logically small if there is a high overlap in the participants' domain knowledge and skill; while there may be many participants, their combined knowledge and skills do not extend very far beyond those of the most knowledgeable and skilled person in the group. Logically, large groups have less ovCTlap in knowledge and skills; thus the group has a wider range of knowledge and skills than any one person in the group. The overlap or lack of overlap of skills, traits, and abilities has been shown to have different effects in studies of non-EMS-supported group work [e.g., 60] particularly in the popensity to participate [15]. In organizational group meetings, participants have typically come from different areas. They bring different skills and domain knowledge that can be used by the group in resolving the issue at hand. Organizational groups have therefore been logically large. In contrast, experimental groups have often been logically small, having been drawn from one population, typically members of an MIS course or other course in the business school. In summary. Table 3 shows that the majority of previous laboratory experiments have studied physically and logically small ad hoc groups of undergradtiate students, without a formal hierarchy, who were unfamiliar with the task domain and the EMS. The majority of field studies have examined physically and logically medium- and large-sized established groups of managers, with a fonnal hierarchy, who were familiar with the task domain and had more experience with the EMS.

The Task
The task performed by groups is "an especially important variable, often accounting for as much as 50 percent of the variation in group performance" [54, p. 88]. We note eight potentially important differences in task (see Table 4). The fu^t is the type of task. There are many useful ways of classifying type of tasks [e.g., 48], such as choice (choosing or allocating among alternatives), generation (the development of ideas, alternatives, information), planning, etc. There have been significant differences in

116

DENNIS, NUNAMAKER. AND VOGEL

the type of tasks between experimental groups and organizational groups that may have had a major impact on the conclusions drawn from different studies. Second, task complexity has been shown to influence group member participation in non-EMS-supported group work [e.g., 15]. Experimental tasks must be appropriate for the experimental group and the time for which they have agreed to participate. Most tasks last one sessionat most a few hours. In contrast to the tasks in experimental sessions, tasks faced by organizational groups have often been "wicked problems" [46,57]. Wicked problems cannot be exhaustively formulated and written down on paper. They have no end; the task is never complete. They cannot be reproduced and are essentially unique. They do not have an enumerable set of potential solutions or operations that could be included in the solution plan. In many cases, the problem cannot be stated separately from the solution, as understanding the task is synonymous with solving it. Finally, there is no immediate and no ultimate test of the "goodness" of a solution. While these tasks occur in field studies, they have typically not been studied in experimentsfor obvious reasons! Third, this complexity is often reflected in the time required to complete the task. Tasks faced by organizational groups in prior field studies have often required several sessions to address, while those in the laboratory have typically required several hours at most. Fourth, the clarity of the task has often varied among studies. In many cases, participants in organizational groups, particularly logically large groups, do not have a common understanding of the problem or may hold different assumptions about it. A group member's assumptions about, or conception of, the situation may differ significantly from the actual situation and from that of other group members [1,46], which can have a significant effect on the outcomes. However, in laboratory experiments, participants have typically read the same task instructions, and thus have been more likely to share common assumptions and conceptions. Fifth, similar differences have been common in the symmetry of information pertaining to the task. Members of organizational groups in the field have often had asymmetrical (even conflicting) task information. Experimental groups, who in previous EMS studies have generally worked from a common description of the experimental task, would have been more likely to have a common set of infonnation. Non-EMS-supported research has shown that interventions into the group process have different effects, depending upon whether all group members have the same information or different infonnation [e.g., 27]. Sixth, the infonnation management needs of experimental groups and organizalional groups typically have been different. Experimental tasks have tended to be "clean," well-defined tasks that arc neatly packaged and small in scope. All information is generally provided in one source, a case package. In contrast, the tasks used by organizational groups are usually ill-defined and "messy" [1,46]. "Messiness" in this sense refers to the degree that the needed information comes from many fragmented sources with varying degrees of objectivity and accuracy. Data may be intemal documents, extemal reports, or subjective assessments held by the participants. Accessing and integrating data from different sources is a task in its own right. The

LABORATORY VS. FIELD RESEARCH ON EMS

117

necessary reporting of informatioti to experimental participants removes their need to access these messy data from multiple sourcesa need often keenly felt by organizational groups, and thus effects on processes and outcomes may have differed. Seventh, with organizational groups there is a need to share information beyond the group to other individuals and groups in the organization. Virtually all organizational groups have taken minutes of meetings for future reference. As a rule, experimental groups don't. The EMS helps ensure key information that will form the basis of the meeting's minutes is recorded error free, without requiring one of the participants to devote time to the recording processand potentially lose opportunities to participate. As the EMS records this infonnation, it also promotes greater accuracy; a person recording information from which the minutes will be prepared has a greater chance to mis-hear or misinterpret spoken comments. Finally, the information needs of organizational groups have usually included information from previous meetings, whether EMS-supported or not. As more than one meeting may be needed to address the task, it becomes more important to manage the information arising in sessions to ensure that it is accessible in subsequent sessions. It must be available for members to refer to, and it must also be available to members who have missed prior meetings to ensure they are briefed on the missed session(s). These information management needs have typically not been addressed by experimental tasks requiring one meeting. In summary. Table 4 shows that the majority of laboratory experiments have examined low-complexity, high-clarity, choice tasks with high information symmetry and few information sources, requiring one to two hours to perform, and no information to be shared from previous meetings or to subsequent meetings. The majority of field studies have examined medium-to-high complexity generation and planning tasks of varying clarities with low to medium information symmetry and many information sources requiring 1-2 days to perform, with no information shared from previous meetings and to subsequent meetings. The laboratory studies finding greater effectiveness with EMS use were more likely to have used tasks including some generation activities of higher complexity that required longer to perform.

EMS Environment
If we examine the EMS environments used in previous studies, we see at least four differences reflecting the different EMS software used at various research institutions (see Table 5). First, an EMS environment can promote at least three different styles of meeting, each of which may have different effects: a chauffeuredprocess, a supported process, and an interactive process. In using these terms, we refer to styles of meeting processes that combine differing amounts of verbal and electronic communication. Each style can be combined with the others and with non-EMS-supported verbal discussion at different stages of any meeting. With a chauffeured process, only one person uses the EMS, either a group member or a meeting leader/facilitator. A workstation is connected to a public display screen, providing an electronic version of the traditional blackboard. The group verbally discusses the issues, with the electronic blackboard used as a group memory to record

118

^ .2 E E

f
8

g g S

a:-I.

'Si o ^ 0 < S o O 0 0 0 O O < U O O O ( S e 3 rt>sE3Bc:eccn>iBBe^>-.

I
u :2

I
E.2 E.2-2-2.2.2.2.2.2 E E-2-2.2
V) in a>
^4 M4
^ > ^

-a
w
K

M
^H

uj
N-

2
Dfl

^1

^1

a -^
!3 U

U)

bJD

dO

Ofi

&J)

OU

bJ)

119

u <u on 00
TO TO

I
lA .^ cn EA

MOD

O C

O C

O O C d

^ A W

M " J "M "M W

:a
S s s

:a

ra

t.

TO

1 c
o

del

ats

OH

BBBBBBEBBB
B

120

II
II
g o o o o o o o o o o p g o S o p o o p g o

i; !-i

E fe E

Ea

Ea

^ .c^ .^ .g^ -o ^ 4J ^ ^ 2 :a 3 E 3 E

w:
is -a 31 i f

E E
(NO '

'E 'E

-r '^

o .S ,J P E E

S .E

"^
CTv

O
" ^
' --H

5 -Si

o o o o o

:i

ro m m

s E

S.2
S
u
M

E
o

u
IA

u
lA

u
n

O
FM

O
F^

O
"C

O
"C

O
rC

II

.a

o o u

I
I
C U Q U

3^

3J

3^

(^

E- & g- w) = ;^ a 5
fl CO eo O O

O O O O ffi

121

c c c c

4^5

E E E E E E E E E E

1.
o U j= j= D 00 O j= 00

T3

T) U

TS

-o O

-o O

-a O

T3 4J

E X 2 S

.2.2.2.2.2.2.3.2.2.3

f f f f f ? ^
oi) oi) oj) 00 00 oi) oi)

-q

j=

x:

ES.2.2.2.2.2.2.S E

.s
o

5 5 j ^

- ^ - ^ f O - ^ - ^ f S O O f O O O

I
2
u '=4j
Q JS

E g g

g Oi)

.G U

c -

1
a

Z 2 > > >

122

"B
l
tA M IA

11 11
t/1

11
a, c c c

o
^^

-p
.L^

o - p - p
^K4

^.^

Jfc^

J? a u o-a q -a
9

at

>

>

to V

>

w a s a .S .5 .5 .S

123

I
a-SiS-SiS
a -a
o -p o

s s s s
o o p o

ao

BO

BO

a 3 s

3
J5

icti

>

a
M <^

x:
3 J=

o. a.
(A

3 o

a .s

.5

.s .s .s

y 13

QJ o

4J o

2 2 2 2 > > > F" o" ^ c ^ ^ r^ T Tf V^ W^

>/l

*O

>C

\O

124

DENMS, NUNAHAKER. AND VOGEL

and structure information. A supported process is similar to a chauffeured meeting process, but differs in that each member has a computer workstation that provides a parallel, anonymous electronic communication channel. The electronic blackboard is still used to present and structure information, but with members able to add items electronically from their workstations. The meeting proceeds using a mixture of verbal and electronic interaction. With an interactive process, the parallel, anonymous electronic communication channel is used for almost all group communication. Virtually no one speaks. While an electronic blackboard may be provided, the group memory is typically too large to fit on a screen, and thus it is maintained so that all members can access it electronically at their workstations. Second, the degree of structure provided by the EMS in terms of the process and, third, the task, has also varied. Some studies have provided high process structure, in which the groups followed a predetermined agenda using predetermined software, while in others group members have been free to select their own agenda and tools. Some studies have provided high task structure (e.g., providing mathematical models or conceptual techniques) to assist the group in resolving the task, while others have provided no task structure. One final difference has been facilitation. Laboratory experiments have typically used EMS environments without facilitators or with only a technical facilitator whose role was essentially passive (fearing possible confounding effects of the facilitator on treatments). Field studies have typically examined environments in which the facilitator took an active role as an integral part of the normal EMS processas important to the outcome of the meeting as any other aspect of the EMS technology, such as individual workstations or electronic blackboards. Indeed, some EMS environments are based on the premise that the facilitator is more important to the normal operation of the environment than individual workstations [e,g.,47]. The perceptions of participants suggest that use of a facilitator can affect meeting outcomes at least as much as any other component in the EMS environment. In summary. Table 5 shows that the majority of laboratory experiments used highly structured processes with more verbal communication (i.e., supported or chauffeured), with less task structure, and with a passsive facihtator or no facilitator. The majority of field studies used moderately or highly structured processes with a more even mix of verbal and electronic communication (i.e., interactive, supported, and chauffeured processes) with a variety of task structure and an active process facilitator. Implications is that it is important to acknowledge explicitly the limitations of conclusions drawnfromEMS research. Results from both experiments and field studies should not be freely generalized to all organizational contexts, all groups, all tasks, all F-MS environments. Research must report sufficient contextual, group, task, and EMS environment information to enable readers to judge the limitations of the conclusions, and researchers should endeavor to bound clearly the scope to which their results apply.
THE FIRST AND MOST BASIC IMPUCATION PROM THESE DIFFERENCES

LABORATORY VS. HELD RESEARCH ON EMS

12S

Second, we have proposed 24 differences that may (or may not) have important impacts on the processes and outcomes of EMS-supported meetings. The importance of each of these differences remains an empirical question. Do any or all have significant impacts on effectiveness, efficiency, or satisfaction in EMS-supported group work? The need (or, conversely, the opportunity) for future research is enormous. Finally, we beUeve that both laboratory and field researchers need to consider explicitly these issues in designing future research, as these factors have the potential to change research fmdings. The key for experimental researchers is to think of the laboratory experiment as a challenge, as an opportunity to mode! the "real" world of their chosen target environment as closely as possible, in order to maximize the ability to generalize findings to the organizational use of EMS. While it has been argued that such "extemal validity" issues are unimportant in theoretically motivated research [7], we disagree. Instead, we have argued that processes and outcomes depend upon the interaction among many variables. Such an interactionist perspective argues that support for a theory developed in one environment provides no evidence that that theory will hold in a different environment; indeed, the opposite may be true (e.g., classical/Newtonian physics versus subatomic/quantum theory). In our opinion, developing such context-free theories is inappropriate, as few such universal "truths" exist. In contrast, the inherent problem in field research is the lack of control, and the lack of precision and accuracy in measurement. The challenge for EMS researchers conducting field study research is to improve these issues, e.g., by the increased use of random assignment and "hard" performance measures rather than "perceived" measures. Several recent papers have presented insightful observations about the design of MIS field studies [42, 45, 70]. Rather than take the general perspective of these papers, we will focus on issues particularly relevant to EMS field studies. In the rest of this section, we present the implications for design decisions in future research (see Table 6). These decisions are listed under seven headings, corresponding to some degree with the areas identified previously: building groups, established/ongoing groups, group size, task, information management, incentives, and EMS environment. We expect that other researchers with different interests will draw other conclusions and choose different altematives for these research design decisions. The important point is that these design decisions should be made explicit and that the choices are appropriate for the goals of the researcher within the constraints faced.

Building Groups
Identifying a pool of subjects to form experimental groups is often an early step in the research design of a laboratory experiment Ideally, subjects will be the same managers and professionals who could use EMS in the organizational environment. Unfortunately, this is not usually a practical altemative; of necessity, Uie use of student subjects is a common practice. Work by Gordon et al. [26] and Remus [55, 56] suggests that, if students are to be used, the use of more experienced, more mature students (e.g., graduate students) is strongly preferred to less experienced, less mature students (e.g., typical undergraduates). Care should be taken in forming experimental groups from a common pool of

126

DENNIS, NUNAMAKER, AND VOGEL

Table 6

Key Design Decisions and Some Options Field studies Attempt to find and study "contior' groups Select groups with desired characteristics Understand group members' backgrounds

Laboratory experiments Building groups Use a diverse subject pool Use more mature graduate students Build logically large groups Establishedlongoing groups Use existing groujK that have a past and a future of working together Group size Study larger groups Task Use preexisting tasks with which the subjects are familiar Use very information-rich cases Information management Provide multiple information sources Provide different information to different subjects

Study established/ongoing projects as well as individual meetings

Study smaller groups Examine fit of the group, task, technology

Track task characteristics Examine participants'task comprehension Monitor participants'interest Provide information integration and monitor usage

EMS environment Use environments appropriate for organizations Monitor the type of meeting process. degree of task and process stmcture, and the level of facilitation Incentives Give incentives to motivate subjects Give different incentives to different subjects

Examine differences in technology across sites Monitor the type of meeting process, degree of task and process structure, and the level of facilitation

Understand organizational incentives

subjects. StudyinggroupsofundergraduateMlSorbusiness Students, for example, tnay not lead to the same conclusions as studying diverse groups whose training, experience, and interests are not exclusively in MIS or business. Drawing subjects from a narrowly defined population may also lead to the formation of groups whose members have the same basic skills and knowledge domain; in other words, it may lead to the formation of logically small groups, not the logically large groups often found in organizations. One solution may be the use of stratified random sampling from several populations, rather than simple random sampling from one population. For example, groups could be built by randomly selecting one subject from an Mis course, one

LABORATORY VS. FIELD RESEARCH ON EMS

127

subject from an organizational behavior course, one from an engineering course, and so on. This has the added benefit of increasing the generalizability of the study [7], but it also has the potential to increase the error variability of statistical measures [40]. Another step toward providing an experimental context more similar to that of organizations is to give each subject different information about the case, to better reflect the differing views held by the actors or groups in the case. The selection of groups to study in field studies is equally important. The desirability of simultaneously studying randomly selected treatment and control groups is obvious. However, such a luxury is not common in fteld studies. Instead, opportunistic analysis of naturally occurring organizational events is more likely. It becomes important to understand why groups have chosen to use EMS technology, as the motives, needs, and expectations of early innovators may differ from those of subsequent groups. However, opportunities to select groups randomly may arise more commonly than anticipated. For example, resource constraints can make it impossible for all possible groups in an organization to use EMS technology or, alternately, only part of an organization may be targeted for the initial installation of EMS technology. In these cases, it may be possible to study otherwise similar treatment and nontreatment groups addressing similar tasks in similar contexts [7]. Other opportunities also exist. Markus [45] argues that in order to disconfum a theory, it is necessary to define a set of conditions in which the theory is most likely to hold. By selecting a case that satisfies these conditions, and then proceeding to disconfirm a theory, tbe theory will be "quite decisively disconfirmed" (p. 10). Therefore, in the study of single groups using EMS, is is necessary to define the characteristics of a desired group, and then identify a group that meets those characteristics. Field studies can also benefit from data gathered on the background of group members. Such information is useful in determining the appropriate technology and tools, as well as assisting in developing theories to explain why seemingly similar groups have dissimilar EMS experiences. Hypotheses from such observations can then be tested experimentally. The combined knowledge base of experimental and field experience can encourage the development of empirically based guidelines for appropriately matching technology to task and group characteristics.

Established/Ongoing Groups
In order to better model the established/ongoing groups typically found in organizations (who have a past and future together), Zigurs et al. [69], for example, used preexisting groups of students. These groups were taken from an MIS course that required groups of students to work together for the duration of the course. The experimental session (unrelated to the course material) occurred midway through the term. The groups had no prior experience with EMS, which Zigurs et al. conclude was a major factor in one group's refusal to use the EMS; they did not want to change their existing group work pattems. This points out the benefits and hazards of using established/ongoing groups. The groups had established norms and work pattems.

128

DENNIS, NUNAMAKER. AND VOGEL

making them similar to organizational groups. However, these norms can increase error, thus requiring more groups to achieve statistical significance [8]. Field studies as well as laboratory studies should focus not only on the meeting session as a unit of analysis, but also on the project as a unit of analysis, where a project may consist of a number of meetings. Group membership and linked use of information across sessions should be monitored. Particular attention should be given to tracking changes in group member perceptions and performance over a series of sessions. Comparisons can be made on a longitudinal basis within a project or series of sessions as well as between projects.

Group Size
A host of non-EMS-supported research has shown that the process and outcome of group meetings is different among groups of different sizes [e.g., 29]. Important differences have also been observed among different sized EMS-supported groups [12]. As organizational groups have tended to be larger than experimental groups in previous studies, one obvious suggestion is to increase the size of groups used in experimental studies, and for field studies to examine the effects of EMS use on smaller groups. Field studies, as well, can benefit from careful tracking of group size as a function of impact on the process and outcome of sessions. Much can be leamed by taking advantage of situations that suggest various group sizes, as well as situations in which there appears to be a mismatch of the group size with the task. One goal is to compare equivalent group sizes in experimental and field settings.

The Task
Selecting the task is arguably the most important pan of the experimental research design. The task should be both appropriate for the subjects, and similar to the tasks addressed by organizational groups in whatever target environment the researcher has chosen. While most laboratory experiments have studied decision-making tasks (i.e., choice tasks or generate and choose tasks), most field studies have examined generate tasks and planning tasks. This suggests that future experiments should seek to investigate generate/planning tasks, while future field studies should examine decision making. As well as providing an issue to be addressed by the group, the task in the laboratory must provide the backdrop for the group discussion. The task domain should be appropriate for the subjects, and, ideally, familiar to them [26]for example, the parking problem common to most universities [6]. Such tasks are readily understood, and provide a rich meeting environment. Participants bring assumptions to the meeting. Different perceptions are likely. It may even be possible to measure the improved understanding of the issues gained by each individual. If such preexisting tasks are not used, sufficient background information must be provided in the case packages provided to subjects to enable them to understand the variety of issues involved. Organizational culture and norms should be provided.

LABORATORY VS. FIELD RESEARCH ON EMS

129

Subjects should be actively encouraged to consider all relevant components of the issues at hand, as would be the case in the tasks facing organizational groups. Field studies should likewise be concemed with task issues from several perspectives. Different measures of the nature of the task should be collected, to help researchers understand ihe effects of EMS for different types of task. Such measures include task difficulty, familiarity with the task, etc. [60]. In many cases, there may be a lack of consensus on what the task is; group members may not share a common understanding of the goals and objectives of the group. These issues need to be considered, possibly by comparing processes and outcomes when a group does a "real" task instead of an "artificial" one developed by the researchers that differs in these aspects.

Infonnation Management
Information for experimental sessions could be provided in a variety of forms. Rather than being neatly condensed into one case package, it could be spread across a variety of "organizational" reports and databases, as it could be with organizational groups. The information can then be made available in either paper or electronically accessible copy during the group's deliberations. Different information (either conflicting or complementary) could be provided to different participants. The end result is a setting more typical of the way organizational groups use EMS. Further, the availability (or lack thereoO of information can be used to reinforce the group's charter, as well as evaluate aspects of the impact of introduction of information extemal to the group during the decision-making process. Again, such information dynamics are typical of organizational group use of EMS. From a field research perspective, it is important to understand how the differences in information held by the participants affect the processes. Links to corporate databases and integration into organization infonnation systems are also important, and thus systematically monitoring and tracking the use of extemal information becomes possible. Provision should also be made for integrating information ^ross group sessions and among groups to further provide a measure of effective group support and provide a foundation for evaluation.

Incentives
Performance incentives should be provided to experimental subjects to better ensure that their participation behavior is appropriately motivated; without incentives, researchers can be less sure that the behavior exhibited is "real," as subjects have no stake in the outcome [62]. Subjects in groups achieving "better" meeting outcomes can be rewarded, even for tasks that do not provide "objective" outcomes. In organizational groups, incentives are not necessarily tied to the "best" outcome for the organization as whole, and thus similar incentives need not be provided to each subject. To model the vested interests and political nature of organizational group meetings, each subject's reward could be be determined on a different basis, such as increasing resources for a specific constituency. An important function of rigorous field research is to understand the nature and

130

DENNIS, NUNAMAKER. AND VOGEL

administration of performance incentives in the organization being studied and the degree of organizational politics. Members may also be more or less motivated to participate in the stated task. Although this is true of field groups with or without provision of automated support, it becomes particularly important to track group member perceptions of the task to facilitate comparisons across groups and tasks. Preand postmeeting questionnaires, as well as systematic tracking of group member interaction during EMS sessions, is helpful to evaluate incentive issues. Additional information can be obtained in interviews with group leaders, facilitators, managers, and executives knowledgeable in the incentive structure and politics of the organization to determine the degree to which they could affect processes and outcome.

EMS Environment
As each type of meeting process (chauffeured, suf^rted, or interactive) may have different effects on meeting outeomes, it is essential to document the processes groups use at each point in the meeting. Some processes are likely more effective than others for certain activities, so experiments comparing different processes for the same tasks are needed. Research data can be collected unobtrusively ftom a variety of sources. EMS environments can be designed to provide more detailed process information to gain insight into the EMS-supported meeting process. Software can provide detailed activity logs. Audio-video recordings of sessions can be used for detailed analysis. Groups c ^ be more carefully observed before and after the EMS-supported meeting(s) to provide a better understanding of exactly how the EMS altered the group work process. EMS is a state-of-the-art technology. Unlike other technologies in MIS research (such as end-user computing or system development techniques), it cannot yet be widely studied in the hands of practitioners. Only a few organizations use EMS technology as researchers know it today [41]. This has several implications. First, researchers must develop their own EMS (or use EMS technology developed by other researchers). While it is tempting to develop only the minimum EMS environment necessary for laboratory research, this lessens the generalizability of findings from these environments to EMS environments in organizations. Second, a major concem of field studies lies in comparison of results across sites. Rarely are sites identical. Rather, each site, for historical or practical reasons, or a conscious desire to evaluate different physical configurations, tends to have a certain distinctiveness. This can seriously confound the ability to compare results across sites, even when ignoring cultural and other organizational differences that may also vary significantly, even within the same organization. The site characteristics should be carefully recorded and analyzed to better field research results leading to the development of new theories. Finally, a major research contribution that EMS researchers can make is to develop guidelines for the successful implementation of EMS technology in organizations. Practitioners in the early 1980s were relatively unsuccessful in their attempts to implement EMS [41]. However, more recent research has shown EMS technology to make important contributions to group work in organizations [52]. One of our tasks

LABORATORY VS. FIELD RESEARCH ON EMS

131

is to guide practitioners in the successful adoption of EMS technology, which requires the use of EMS environments similar to those required by organizations.

T o w a r d a More Perfect World

WE BELIEVE THAT, WHILE FINDINGS FROM PREVIOUS EMS laboratory

experiments and field studies have been different, they are not inconsistent Findings have been different because EMS use in previous field studies has been distinctly different from EMS use in previous laboratory experiments. We noted 24 specific differences in four areas. To paraphase Huber [33, p. 571], while we believe that each of these differences has merit, even if only one or two has merit, the conclusion has meritthat the use of EMS in experiments has been sufficiently different from the use of EMS by organizational groups to suggest that researchers have studied different contexts, groups, tasks, and EMS, thus findings are not necessarily inconsistent. This also suggests that previous laboratory results shotild not be generalized to current field use of EMSand that previous field studies should not be generalized to current laboratory environmentsalthough they could be generalized to similar environments. In our opinion, these issues do not lessen the value of previous experimental or field study research. Both have been key components in advancing the state of knowledge of EMS-supported group work. However, each has been, and will continue to be, only one component. Many of these past differences reflect research design issues that can be addressed in futtire studies. For example, future field studies can examine smaller groups while future laboratory studies can examine larger groups. Other differences, however, are more fundamental to these research methodologies, and will not be completely ameliorated (e.g., task complexity). Regardless of researchers' attempts, there will continue to be important differences between the laboratory and the field. In a more perfect world, these fundamental differences would be widely recognized. We would realize that neither laboratory experiments nor field studies provide complete insight into the use of EMS, and that an overreliance on one approach could be misleading. We would strive to gain firsthand understanding from both perspectives in order to better integrate findings. In a more perfect world, field rese^chers would recognize and accept that ihe laboratory is not the Field, and that certain aspects of laboratory experiments may never be "real." Likewise, laboratory researchers would recognize and accept that the field is not a laboratory, and that different standards of evidence are used to assign cause and effect. In a more perfect world, we would strive to understand the effects that these differences have on the processes and outcomes of group meetings. Is EMS more beneficial in politically charged contexts, to larger groups, for generation tasks, or when it provides a highly structured process? We could then use this understanding to interpret and apply the conclusions of experiments to the use of EMS by business organizations, and to interpret and apply observations in the field to the development and testing of theories in the laboratory. From this understanding, we would develop

132

DENNIS. NUNAMAKER. AND VOGEL

new research designs: field studies with greater rigor and experiments with greater relevancy to the organizational use of EMS. In a more perfect world, we would recognize that these design decisions directly affect the processes and outcomes observed and the subsequent conclusions drawn about the effects of EMS use. Such decisions would be made only after a careful consideration of their potential impact on the final conclusions and in a manner that best suited the research questicms. We have presented several approaches to enhance the design of EMS experiments and field studies. These suggestions certainly do not apply to all research programs, as different programs have different objectives. Nor are these suggestions without cost. However, as suggested by Javenpaa, Dickson, and DeSanctis [35] with regard to another research area (management graphics), a lesser attention to such details will not advance our understanding of EMS; it will only serve to cloud the issues.
NOTES

1. Although Kerlinger is speaking about the social sciences in general, we and others [e.g., 25,39] feel thai this view is aJso relatively prevalent in the MIS domain. 2. In using the tenn "organizational groups" throughout this paper, we are referring to the groups from both the public and the private sector who have used EMS technology as part of their ongoing organizational activities.
REFERENCES

1. Ackoff, R. L. Creating the Corporate Future. New York: John Wiley and Sons, 1981. 2. Adelman, L. Real time computer support for decision analysis in a group setting. Interfaces, 14, 2 (1984), 75-83. 3. Benbasat, I.; Goldstein, D. K.; and Mead, M. The case research strategy in studies of information systems. M/^guorrer/^i, 11, 3 (1987), 369-386. 4. Bui, T., and Sivasankaran, T. R. Relation between GDSS use and group task complexity. Proceedings of the Twenty-Third Hawaii International Conference on Systems Sciences (1990), ni: 69-78. 5. Chidambarum, L.; Bostrom, R. P.; and Wynne, B. E. An empirical investigation of the impact of computer support on group development. Proceedings of the Twenty-Third Hawaii International Conference on Systems Sciences (1990), III: 3-12. 6. Connolly, T.; Jessup, L. M.; and Valacich, J. S. Idea generation in a GDSS: effects of anonymity and evaluative tone. Management Science, forthcoming. 7. Cook, T. D., and Campbell, D. T. Quasi-Experimentation. Boston: Houghton Mifflin, 1984. 8. Dennis, A. R.; Easton, A. C ; Easton, G. K.; George, J. F.; and Nunamaker, J. F., Jr. Ad hoc versus established groups in an electronic meeting system environment. Proceedings of the Twenty-Third Hawaii International Conference on Systems Sciences (1990), HI: 23-29. 9. Dennis, A. R.; George, J. F.; Jessup, L. M.; Nunamaker, J. F. Jr.; and Vogel, D. R. Infonnation technology to support group work. MIS Quarterly, 12 (1988), 591-624. 10. Dennis, A. R.; Heminger, A. R.; Nunamaker, J. F., Jr.; and Vogel, D. R. Bringing automated support to large groups: the Burr-Brown experience. Information & Management, 18, 3 (1990). 111-121. 11. Dennis, A. R.; Tyran, C. K.; Vogel, D. R.; and Nunamaker, J. F., Jr. An evaluation of electronic meeting systems to support strategic management. ICIS, 1990, forthcoming. 12. Dennis, A. R.; Valacich, J. S.\ and Nunamaker, Jr., J. F. An experimental investigation

LABORATORY VS. HELD RESEARCH ON EMS

133

of group size in an electronic meeting system environment. IEEE Transactions on Systems, Man, and Cybernetics, forthcoming. 13. DeSanctis, G.; D'Onofrio, M. J.; Sambamtirlhy, V.; and Poole, M. S. Comprehensiveness and reslrictiveness in group decision heuristics: effects of computer support on consensus decision making. ICIS (1989), 131-140. 14. DeSanctis, G., and Gallupe, R. B. A foundation for the study of group decision support systems. Management Science, 33 (1987), 589-609. 15. Diehl, M., and Stroebe, W. Productivity loss in bralnstorming groups: towaid the solution of a riddle. Journal of Personality and Social Psychology, 53 (1987), 497-509. 16. Dutton, W. H. Letter to the editor. MIS Quarterly, 12,4 (1988), 521. 17. Easton, A. C ; Vogel, D. R.; and Nunamaker, J. F., Jr. Stakeholder identification and asumption surfacing in small groups: an experimental study. Proceedings of the Twenty-Second Hawaii International Conference on Systems Sciences (1989), III: 344-352. 18. Easton, G.; George, J. F.; Nunamaker, J, F., Jr.; and Pcndergast, M. O. Using two different electronic meeting system tools for the same task: an experimental comparison. Journal of Management Information Systems, 1990. 19. Fellers, J. W. The Effect of Group Size and Computer Support on Group Idea Generation for Creativity Tasks: An Experimental Evaluation Using a Repeated Measures Design. Unpublished Ph.D. dissertation, Indiana University, 1989. 20. Galiiers, R. D., and Land, F. F. Choosing ^jpropriate information systems research methodologies. Viewpoint, Communicat'ionsoftheACM, 30, 11 (1987), 900-902. 21. Gallupe, R. B. Suppressing the contribution of the group's best member: isGDSS use appropriate for all group tasks? Proceedings of the Twenty-Third Hawaii International Conference on Systems Sciences {\^^\^S1\ Yh-ll. Tl. Gallupe, R. B.; DeSanctis, G.; and Dickson, G. W. Computer-based support for group problem finding: an experimental investigation. MIS Quarterly, 12, 2 (1988), 277-296. 23. Gallupe, R. B., and McKeen, 3. D. Beyond computer-mediated communication: an experimental study into the use of a group decision support systems for face-to-face versus remote meetings. Information &. Management, 18, 2 (1990). 24. George, J. F.; Easton, G. K.; Nunamaker, J. F., Jr.; and Northcraft, G. B. A study of collaborative group work with and without computer based support. Information Systems Research, forthcoming. 25. Goldstein, D.; Markus, M. L.; Rosen, M.; and Swanson, E. B. Use of quantitative methods in MIS research. Proceedings of ICIS (December 1986), pp. 338-339. 26. Gordon, M. E.; Slade, L. A.; and Schmitt. N. The "science of the sophomore" revisited: iiomconiectmetoempkicism. Academy of Management Review, 11,1 (1986), 191-207. 27. Hackman, J. R., and Kaplan, R. E. Interventions into group process: an approach to improving the effectiveness of groups. Decision Sciences, 5 (1974), 459-480. 28. Hall, J., and Williams, M. S. A comparison of decision-making perfonnance in established and ad hoc groups. Journal of Personality and Social Psychology, 3 (1966), 214-222. 29. Hare, A. P. Group size. American Behavioral Scientist, 24, 5 (1981), 695-708. 30. Ho, T. H.; Raman, K. S.; and Watson, R. T. Group decision support systems: the cultural factor./C/5, 1989,119-129. 31. Hoffer, J. A.; Anson, R,; Bostrom, R. P.; and Michaele, S. J. Identifying the root causes of data and systems planning problems: an application of the PLEXSYS electronic meeting support system. Proceedings of the Twenty-Third Hawaii International Conference on Systems Sciences (1990), m : 30 39. 32. Huber, G. P. The nature of organizational decision making arxl the design of decision support systems. MIS Quarterly, 5, 2 (1981), 1-10. 33. Huber, G. P. Cognitive style as a basis for Mis and DSS designs: much ado about nothing'} Management Science, 29, 5 (1983), 567-579. 34. Jarvenpaa, S. L. The importance of laboratory experimentation in IS research. Technical Correspondence, Coffimurticanon5o///ie.4CA/, 31,12(1988), 1502-1504. 35. Jarvenpaa. S. L ; Dickson, G. W.; and DeSanctis, G. Methodological issues in experimental IS research: experiences and recommendations. MIS Quarterly, 9, 2 (1985), 141-156. 36. Jarvenpaa, S. L.; Rao, V. S.; and Huber, G. P. Computer support for meetings of me-

134

DENNIS. NUNAMAKER. AND V0C5EL

dium-sized groups working on unstructured problems: a field experiment. MIS Quarterly, 12,4 (1988), 645-666. 37. Jessup, L. M.; Connolly, T.; and Galegher, J. The effects of anonymity on group process in an idea-generating task. MIS Quarterly, forthcoming. 38. Jessup, L. M., and Tansik, D. A. Decision making in an automated environment: the effects of anonymity and proximity on group process and outcome with a group decision support system. Decision Sciences, forthcoming. 39. Kaplan, B., and Duchon, D. Combining qualitative and quantitative methods in information systems research: diczsc study. MIS Quarterly, 12,4 (1988), 571-586. 40. Kerlinger, F. N. Foundations of Behavioral Research, third edition. New York: Holt, Rinehart and Winston, 1986. 41. Kraemer, K. L., and King. J. L. Computer-based systems for cooperative work. Computing Surveys, 20 (19^1 115-146. 42. Lee, A. S. A scientific methodology for MIS case studies. MIS Quarterly, 13, 1 (1989), 33-50. 43. Lim, L. H.; Raman, K. S.; and Wei, K. K. Does GDSS promote more democratic decision-making?the Singapore experiment. Proceedings of the Twenty-Third Hawaii Intemational Conference on Systems Sciences (1990), HI: 59-68. 44. Loy, S. L.; Pracht, W. E.; and Courtney, J. F., Jr. Effects of graphical problem structuring aid on small group decision making. Proceedings of the Twentieth Hawaii International Conference on System Sciences (1987), 566-574. 45. Markus, M. L. Case selection in a disconiimiatory case study. Presented at the New Approaches in MIS Research Symposium, National Academy of Management Meeting, August 9,1988, Anaheim, CA. 46. Mason, R. O., and Mitroff, 1.1. Challenging Strategic Planning Assumptions. New York: John Wiley and Sons, 1981. 47. McCartt. A. T., and Rohrbaugh. J. Evaluating group decision support system effectiveness: a performance study of decision confeiencmg. Decision Support Systems, 5 (1989), 243-254. 48. McGrath, J. E. Groups: Interaction and Performance. Englewood Cliffs, NJ: PrenticeHall, 1984. 49. McGrath. J. E. Studying groups at work: ten critical needs for theory and practice. In P. S. Goodman and Associates (eds.). Designing Effective Work Groups. San Francisco: JosseyBass. 1986. 50. Nunamaker. J. F., Jr.; Applegate, L. M.; and Konsynski, B. R. Facilitating group cteativity vj'iih GDSS. Journal of Management Information Systems, 3 (1987). 5-19. 51. Nunamaker, J. F., Jr.; Applegate, L. M.; and Konsynski, B. R. Computer-aided deliberation: model management and group decision support. Journal of Operations Research (1988), 826-848. 52. Nunamaker., J. F., Jr; Vogel, D.; Heminger, A.; Martz, B.; Grohowski, R.; and McGoff. C. Experiences at IBM with group support systems: a field study. Decision Support Systems, 5,2(1989), 183-196. 53. Pinsonneault, A., and Kraemar, K. L. The impact of technological support on groups: an assessment o(ti\e.empirical research. Decision SImport Systems, 5, 2(1989), 197-216. 54. Poole, M. S.; Siebold, D. R.; and McPhee, R. D. Group decision-making as a structurational process. Quarterly Journal of Speech, 71 (1985), 74-102. 55. Remus, W. E. An empirical test of the use of graduate students as surrogates for managers in experiments on business decision making. Journal of Business Research, 14 (1986), 20-30. 56. Remus, W. Using students as subjects in experiments on decision support systems. Proceedings of the Twenty-Second Hawaii International Conference on Systems Sciences (1989),IV:176-180. 57. Rittel, H. W. J., and Webber, M. M. Dilemmas in a general theory of planning. 58. Sambamurthy. V., and DeSanctis, G. An experimental evaluation of GDSS effects on group performance during stakeholder analysis. Proceedings of the Twenty-Third Hawaii International Cor^erence on Systems Sciences (1990), 111:79-88.

LABORATORY VS, FIELD RESEARCH ON EMS

135

59. Sharda, R.; Barr, S. H,; and McDonnell, Decision suppon system effectiveness: a review and an empirical test. Management Science, 34,2 (1988). 139-159. 60. Shaw, M. Group Dynamics: The Psychology of Small Group Behavior, third edition. New York: McGraw HiU, 1981. 61. Siegel, J. S,; Dubrovsky, V.; Kiesler, S,; and McGture, T, Group processes in computermediated communication. Organizational Behavior and Human Decision Processes, 37 (1986), 157-187, 62. Smith, V. L. Micro economic systems as an experimental science. American Economic Review, 72 (1982), 923-955, 63. Steeb, R,, and Johnston, S. C. A computer-based interactive system for group decision making, IEEE Transactions on Systems, Man, and Cybernetics, SMC-11 (1981), 544 552. 64. Valacich, I S.; Dennis, A. R,; and Nunamaker, J. F., Jr. Electronic meeting support: the GroupSystems concept. International Journal of Man-Machine Studies, forthcoming. 65. Van Shaik, F. D,, and Sol, H. G, Effectiveness of decision support systems. Proceedings of the Twenty-Third Hawaii Intemational Cor^erence on Systems Sciences (1990), UI: 50-58. 66. Vogel, D. R,; Martz, W, B,; Nunamaker, J. F., Jr.; Grohowski, R. B.; and McGoff, C, Electronic meeting system experience at IBM, Journal of Management Information Systems (1990). 67. Vogel D, R,, and Niinamakcr, J, F., Jr, Health service group use of automated planning suppoTL Administrative Radiology (Septonber 1989), 68. Watson, R. T.; DeSanctis, G,; and Poole, M, S, Using a GDSS to facilitate group consensus: some intended and tinintended consequences, MIS Quarterly, 12, 3 (1988), 463-478. 69. Zigurs, I.; Poole, M, S,; and DeSanctis, G, A study of influence in computer-mediated commtmication. MIS Quarterly, 12,4 (1988), 625-644. 70. Zmud, R, W., and Hauser, R. Field experimentation in MIS research. Working Paper, College of Business, Florida State University, June 1988.